Indicator system
The indicator system addresses the challenge of anatomical state indication in surgical robots by analyzing endoscopic video data to generate real-time health indicators, improving surgical precision and reducing operation times through enhanced site representation.
Patent Information
- Application Number
- CN201980065492.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-10-03
- Filing Date
- 2019-10-03
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2039-10-03
AI Technical Summary
Surgical robots face challenges in accurately and efficiently indicating the state of a patient's anatomy during minimally invasive procedures, leading to potential disorientation and prolonged operation times for surgeons.
An indicator system that analyzes endoscopic video data to detect spatio-temporal variations in patient anatomy, generating health indicators such as visual, audio, or tactile feedback to enhance the representation of the surgical site, aiding surgeons in maintaining orientation and improving procedural efficiency.
Enhances surgical precision and reduces operation time by providing real-time health indicators and improved human-computer interaction, allowing surgeons to navigate and perform procedures more accurately and quickly.
Smart Images

Figure CN112804958B_ABST
Abstract
Description
Background Art
[0001] The use of robots to assist and perform surgery is known. Figure 1 A typical surgical robot 100 is shown, consisting of a base 108, an arm 102, and an instrument 105. The base supports the robot and is itself rigidly attached, for example, to an operating room floor, an operating room ceiling, or a cart. The arm extends between the base and the instrument. The arm is articulated by means of a plurality of flexible joints 103 along its length, which are used to position the surgical instrument in a desired position relative to the patient. The surgical instrument is attached to the distal end 104 of the robot arm. The surgical instrument penetrates the body of the patient 101 at a port 107 in order to access the surgical site. The instrument includes at its distal end an end effector 106 for performing a medical procedure.
[0002] Figure 2 A typical surgical instrument 200 for performing robotic laparoscopic surgery is shown. The surgical instrument includes a base 201 by which the surgical instrument is connected to a robotic arm. A shaft 202 extends between the base 201 and an articulation 203. The articulation 203 terminates in an end effector 204. Figure 2 , a pair of serrated jaws are shown as end effector 204. Articulation 203 allows end effector 204 to move relative to shaft 202. It is desirable to provide at least two degrees of freedom to the movement of end effector 204 through the articulation.
[0003] The imaging device may be located at the surgical site along with the surgical instrument. The imaging device may image the surgical site. The image of the surgical site provided by the imaging device may be displayed on a display for viewing by the surgeon performing the procedure. Thus, laparoscopic (or minimally invasive) surgery may be performed without the surgeon being able to directly see the surgical site. Summary of the invention
[0004] According to one aspect of the present invention, there is provided an indicator system for a surgical robot system, the indicator system being used to indicate the state of at least a portion of a patient's anatomy, the surgical robot system comprising a robot having a base and an arm extending from the base, the arm holding an endoscope at an end of the arm away from the base, the endoscope being configured to be inserted into a body cavity of a patient for observing a surgical site inside the patient's body, the indicator system comprising:
[0005] a receiver configured to receive video data of at least a portion of a patient's anatomy at the surgical site from the endoscope; and
[0006] A processor configured to:
[0007] detecting spatio-temporal variations in a pixel region of received video data;
[0008] identifying a parameter of the patient's anatomy in response to the detected spatio-temporal changes;
[0009] generating a health indicator indicative of the identified parameter or a curve indicative of the identified parameter; and
[0010] Outputs the generated health indicator.
[0011] The processor may be configured to compare the identified parameters with reference parameters of the patient anatomy and generate the health indicator based on the results of the comparison. The spatio-temporal variation may include one or more of: a positional variation of at least a portion of the patient anatomy in the pixel region; an intensity variation of at least a portion of the patient anatomy in the pixel region; and a spectral variation of at least a portion of the patient anatomy in the pixel region.
[0012] The processor may be configured to identify relative movement between the endoscope and a body of the patient anatomy and / or relative movement between the body of the patient anatomy and a remainder of the surgical site and to detect the spatio-temporal changes based on the identified relative movement.
[0013] The processor may be configured to amplify the space-time variation and generate the health indicator using the amplified space-time variation. When the space-time variation includes a variation between a first value and a second value, the processor may be configured to amplify the space-time variation by multiplying the first value by a first amplification factor and multiplying the second value by a second amplification factor. One or both of the first amplification factor and the second amplification factor may be greater than 1. One or both of the first amplification factor and the second amplification factor may be selected based on one or more of: a number of detected space-time variations in the pixel region; a surgeon preference; a procedure or a stage of the procedure being performed using the surgical robotic system; the patient anatomy; the identified parameters; and a comparison of the identified parameters with the reference parameters.
[0014] The pixel region may include at least a portion of the field of view of the endoscope. The processor may be configured to select the pixel region in response to one or more of a received user selection signal and an anatomical structure identification signal. The surgical robot system may include a controller having an input device for controlling an arm of the surgical robot system, the controller being configured to output the user selection signal in response to a user input related to the surgical site. The processor may be configured to identify one or more anatomical structures in the received video data, and output the anatomical structure identification signal in response to identifying the one or more anatomical structures. The pixel region may include at least a portion of an anatomical structure. The space-time variation may include peristalsis of at least a portion of the anatomical structure.
[0015] The pixel region may include one or more of: a representation of at least a portion of a ureter, and the space-time variation includes peristalsis of the ureter; and a representation of at least a portion of an intestinal tract, and the space-time variation includes peristalsis of the intestinal tract. The space-time variation may represent a measure of tissue perfusion or blood flow.
[0016] The processor may be configured to calibrate the identified parameter using another parameter. The parameter and the another parameter may relate to one or both of a patient heart rate and a patient breathing rate. The receiver may be configured to receive further video data of an external portion of the patient, and the processor may be configured to: detect another spatio-temporal variation in a pixel region of the received further video data; and identify the further parameter in response to the detected further spatio-temporal variation.
[0017] The generated health indicator may include one or more of the following: a visual indicator, and the processor is configured to output the generated health indicator for display on a display; an audio indicator, and the processor is configured to output the generated health indicator for driving a speaker; and a tactile indicator, and the processor is configured to output the generated health indicator for driving a motor to provide tactile feedback.
[0018] According to another aspect of the present invention, there is provided a method of indicating a status of at least a portion of a patient's anatomy, the method comprising:
[0019] receiving, from an endoscope at a surgical site within a body cavity of a patient, video data of at least a portion of a patient's anatomy at the surgical site;
[0020] detecting spatio-temporal variations in a pixel region of received video data;
[0021] identifying a parameter of the patient's anatomy in response to the detected spatio-temporal changes;
[0022] generating a health indicator indicative of the identified parameter or a curve indicative of the identified parameter; and
[0023] Outputs the generated health indicator.
[0024] According to another aspect of the present invention, an indicator system for a surgical robot system is provided, the indicator system being configured to perform the method as described herein.
[0025] According to another aspect of the present invention, there is provided a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, which when executed at a computer system cause the computer system to perform the method as described herein.
[0026] According to another aspect of the present invention, a surgical robot system for enhancing the representation of a surgical site where a surgical procedure can be performed is provided, the system comprising: a display configured to display part or all of a representation of at least a portion of the surgical site; a controller comprising an input device whereby the input device can control an indicator on the display, the controller being configured to output an enhancement signal in response to user input related to the representation; and a processor configured to receive the enhancement signal and enhance the representation of the surgical site in response to the received enhancement signal.
[0027] The system may further comprise an imaging device, whereby the imaging device is configured to image the surgical site and generate an image feed of the surgical site. A representation of the surgical site may be obtained from the generated image feed. The representation of the surgical site may be a 3D representation.
[0028] The system may include a robotic arm having a surgical instrument having an end effector, whereby the representation of the surgical site may include a representation of the end effector and the indicator includes a portion of the representation of the end effector.
[0029] The enhancement signal may indicate a feature displayed on a display. The system may be configured to associate the enhancement with the representation of the surgical site based on an indicator position. The processor may be configured to enhance the representation of the surgical site based on joint data associated with the robotic system. The processor may be configured to associate the enhancement with a location in the representation of the surgical site determined based on the joint data.
[0030] The processor may be configured to enhance the representation of the surgical site based on image processing of the representation.
[0031] The system may include a second input device, the processor being configured to enhance the representation of the surgical site based on user input at the second input device. At least one of the input device and the second input device may be responsive to one or more of a voice command, a gesture, and a touch interacting with the respective input device.
[0032] The processor may be configured to determine that an enhancement criterion is satisfied and enhance the representation of the surgical site based on the determination. The processor may be configured to determine that an enhancement criterion is satisfied based on at least one of:
[0033] attachment, removal, manipulation and / or status change of surgical instruments;
[0034] an image recognition signal indicating an image recognition match in the representation of the surgical site;
[0035] an error signal indicating an error associated with the surgical robotic system;
[0036] time signal; and
[0037] A signal indicating a change to a user of the surgical robotic system.
[0038] The processor may be configured to characterize features associated with an enhancement in the representation of the surgical site based on at least one of a user input and an image recognition signal. The system may include a memory coupled to the processor, the memory configured to store at least one of the representation of the surgical site and the enhancement.
[0039] According to another aspect of the present invention, a method for enhancing a representation of a surgical site where a surgical procedure can be performed is provided, the method comprising: displaying part or all of a representation of at least a portion of the surgical site; controlling an indicator on the display; receiving user input related to the representation and outputting an enhancement signal in response to the received user input; and enhancing the representation of the surgical site in response to the enhancement signal.
[0040] The method may include at least one of: generating a representation of the surgical site based on a stored representation of the surgical site; and generating an image feed of the surgical site and generating the representation of the surgical site based on the generated image feed. The indicator may include a portion of an imaging end effector of the surgical robotic system, and the method may include controlling the indicator on the display by controlling the end effector. The method may include enhancing the representation of the surgical site based on at least one of joint data and image processing of the representation. The method may include determining that an enhancement criterion is met, and enhancing the representation of the surgical site based on the determination. The method may include associating the enhancement with at least one set of enhancements. The method may include displaying the enhancement based on the set of enhancements to which the enhancement belongs.
[0041] The method may include, when the representation of the surgical site includes a plurality of enhancements, determining at least one of:
[0042] Number of enhancements;
[0043] the distance between the plurality of enhancements;
[0044] an area surrounded by the plurality of enhancements; and
[0045] A volume enclosed by the plurality of enhancements.
[0046] The method may include displaying one or more of the following on at least one of the display and the secondary display:
[0047] Number of enhancements;
[0048] Connect multiple enhanced lines;
[0049] An area surrounded by multiple enhancements; and
[0050] A volume surrounded by multiple enhancements.
[0051] Any one or more features of any aspect above can be combined with any one or more features of any other aspect above. Any device feature can be written as a method feature where possible, and vice versa. Only for the sake of brevity, these contents are not written out in full here.
[0052] This summary is provided to introduce some concepts in a simplified form, which are further described in the detailed description below. The reference to features in this summary does not indicate that they are key features or essential features of the invention or the claimed subject matter, nor should they be considered to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] The present invention will now be described by way of example with reference to the accompanying drawings.
[0054] In the attached picture:
[0055] Figure 1 A surgical robot is shown performing a surgical procedure;
[0056] Figure 2 A known surgical instrument is shown;
[0057] Figure 3 A surgical robot is shown;
[0058] Figure 4 A surgeon console is shown;
[0059] Figure 5 The configuration of the controller is schematically shown;
[0060] Figure 6 A method for enhancing the representation of a surgical site is shown;
[0061] Figure 7 An example of enhancement is shown;
[0062] Figure 8 An example of a display portion of a representation is shown;
[0063] Fig. 9 A process for centering a portion of a representation is shown;
[0064] Fig.10 Another example of enhancement is shown;
[0065] Fig.11 An exemplary endoscope is shown;
[0066] Fig.12 An exemplary indicator system is shown;
[0067] Fig.13 An example of amplification of parameters is schematically shown;
[0068] Fig.14 schematically illustrates an example of a display of a health indicator; and
[0069] Fig.15 An exemplary method of indicating a status of an anatomical structure of a patient is shown. DETAILED DESCRIPTION
[0070] The following description describes techniques in the context of surgical robotic systems, although not all features or techniques described herein are limited to such systems, and at least some of the features or techniques may be more generally applicable to robotic systems. At least some of the techniques may be applied to remotely operated robotic systems. At least some of the techniques may be applied at locations where a user of a robotic system may become disoriented while operating the robotic system. Some examples of situations where at least some of the techniques may be useful include situations where a "snake" robot is utilized for exploration, investigation, or repair.
[0071] Robotic systems can include manufacturing systems, such as vehicle manufacturing systems, parts handling systems, laboratory systems, and manipulators, such as manipulators for hazardous materials or surgical manipulators.
[0072] The present technology allows information, such as one or more parameters about at least a portion of a patient's anatomy (e.g., an anatomical structure), to be extracted from surgical videos captured at an internal surgical site during a procedure such as a laparoscopic procedure. Analyzing the video to identify these extracted parameters allows for enhanced use of a surgical robotic system when performing a surgical procedure.
[0073] For example, a surgical robotic system can detect peristalsis of an anatomical structure within the body, such as peristalsis of an anatomical structure visible in a pixel region of video data captured by an endoscope at a surgical site inside the body. The pixel region may include one or more pixels. Identification of such peristalsis can allow identification of the anatomical structure itself. A particular anatomical structure may have a characteristic peristalsis, such as a characteristic frequency and / or amplitude of movement. Thus, the identified peristalsis can be compared to one or more known peristalsis that are characteristic of the anatomical structure. When the identified peristalsis is within the range of known peristalsis, the structure experiencing the identified peristalsis can be identified.
[0074] In addition, it can be identified whether the peristalsis of the anatomical structure is within the normal range for such movement of the structure. If the identified movement is within the normal range, then the anatomical structure can be determined to be healthy. If the identified movement is outside the normal range, then the anatomical structure can be determined to be unhealthy. For a given structure, the range that is considered to be "normal range" can depend on the physiological characteristics of the patient, the procedure being performed, the stage of the procedure, etc.
[0075] Identification of peristalsis may aid in identifying the structure from surrounding structures or more generally from the surgical site.
[0076] The technology described herein includes a system for indicating the state of at least a portion of a patient's anatomy (such as an anatomical structure of the patient). The state of the anatomy or anatomical structure may be or may represent the health status of the patient. Endoscopic video captured by an endoscope at an internal surgical site may be analyzed to detect spatial and / or temporal changes in the video data (referred to as space-time changes). These changes may represent the movement of the anatomical structure and / or the blood flow through the anatomical structure. The space-time changes may be used to identify one or more parameters of the anatomical structure. For example, the parameter may relate to the movement or movement rate of the structure. The parameter may relate to the flow or flow rate of blood passing through or through the structure. The parameter may be related to a measure of tissue perfusion. A health indicator indicating a parameter or a curve (e.g., a time curve) of the parameter may be generated. The health indicator may be output. The health indicator may be output so as to notify a user of the system, such as a surgeon, of the health indicator. For example, the health indicator may be a visual signal output to a display, and / or an audio signal output to a speaker, and / or a tactile signal output to a motor to provide tactile feedback (e.g., the tactile feedback may be provided via a tactile-enabled input device).
[0077] Health indicators may help a surgeon assess the status of the anatomy (and / or the patient) during a surgical procedure.
[0078] Figure 3 A surgical robot is shown having an arm 300 extending from a base 301. The arm comprises a plurality of rigid branches 302. The branches are connected by revolute joints 303. The proximal branch 302a is connected to the base by a proximal joint 303a. It is connected in series with the other branches by further joints 303. Suitably, a wrist 304 consists of four separate revolute joints. The wrist 304 connects one branch (302b) of the arm to the distal-most branch (302c). The distal-most branch 302c carries an attachment 305 for a surgical instrument 306. Each joint 303 of the arm has one or more motors 307 which can be operated to cause rotational movement at the respective joint, and one or more position and / or torque sensors 308 which provide information about the current configuration and / or load at the joint. Suitably, the motors are arranged proximal to the joints whose movement they drive in order to improve weight distribution. For clarity, the figures in the figure are not shown in FIG. Figure 3 Only some of the motors and sensors are shown. The arm may generally be as described in our co-pending patent application PCT / GB2014 / 053523.
[0079] The arm terminates in an attachment 305 for interfacing with an instrument 306. Suitably, the instrument 306 takes the form of a Figure 2The form described. The diameter of the instrument is less than 8mm. Suitably, the diameter of the instrument is 5mm. The instrument can have a diameter less than 5mm. The instrument diameter can be the diameter of the shaft. The instrument diameter can be the diameter of the profile of the hinge. Suitably, the diameter of the profile of the hinge matches the diameter of the shaft or is narrower than the diameter of the shaft. The attachment 305 includes a drive assembly for driving the hinge of the instrument. The movable interface element of the drive assembly interface mechanically engages the corresponding movable interface element of the instrument interface so as to transfer the driving force from the robot arm to the instrument. During a typical operation, one instrument is replaced with another instrument several times. Therefore, during the operation, the instrument can be attached to the robot arm and can be removed from the robot arm. The features of the drive assembly interface and the instrument interface facilitate their alignment when they are engaged with each other, thereby reducing the accuracy with which they need to be aligned by the user.
[0080] Instrument 306 includes an end effector for performing the procedure. The end effector may take any suitable form. For example, the end effector may be a smooth jaw, a serrated jaw, a clamp, a pair of shears, a needle for suturing, a camera, a laser, a knife, a stapler, a cauterizer, or a suction device. Figure 2 The instrument comprises an articulated portion between the instrument shaft and the end effector. The articulated portion comprises a number of joints that allow the end effector to move relative to the axis of the instrument. The joints in the articulated portion are actuated by drive elements, such as cables. These drive elements are fixed to interface elements of the instrument interface at the other end of the instrument shaft. Therefore, the robot arm transmits the driving force to the end effector in the following manner: the movement of the drive assembly interface element causes the instrument interface element to move, the movement of the instrument interface element causes the drive element to move, the movement of the drive element causes the joints of the articulated portion to move, and the movement of the joints causes the end effector to move.
[0081] Controllers for motors, torque sensors, and encoders are distributed within the robot arm. The controllers are connected to a control unit 309 via a communication bus. The control unit 309 includes a processor 310 and a memory 311. The memory 311 stores software in a non-transient manner, which can be executed by the processor to control the operation of the motor 307 so that the arm 300 operates in the manner described herein. Specifically, the software can control the processor 310 to drive the motor (e.g., via a distributed controller) according to inputs from the sensor 308 and inputs from the surgeon command interface 312. The control unit 309 is connected to the motor 307 to drive them according to the outputs generated by the execution of the software. The control unit 309 is connected to the sensor 308 to receive sensing inputs from the sensor, and is connected to the command interface 312 to receive inputs from it. For example, the corresponding connections can each be a cable or an optical cable, and / or can be provided by a wireless connection. The command interface 312 includes one or more input devices, whereby the user can request the movement of the end effector in a desired manner. The input device can be, for example, a manually operable mechanical input device, such as a control handle or joystick, or a contactless input device, such as an optical gesture sensor. The software stored in the memory 311 is configured to respond to these inputs according to a predetermined control strategy and cause the joints of the arm and the instruments to move accordingly. The control strategy can include safety features that regulate the movement of the arm and the instruments in response to the command inputs. Thus, in general, the surgeon at the command interface 312 can control the movement of the instrument 306 to perform the desired surgical procedure. The control unit 309 and / or the command interface 312 can be remote from the arm 300.
[0082] Some surgical procedures may require several surgical robot arms, each carrying instruments or other instruments used simultaneously with other instruments at the surgical site. Such surgical robots are commonly used in endoscopic surgery (e.g., laparoscopic surgery), which can also be referred to as minimally invasive surgery. As known to those skilled in the art, during an endoscopic procedure, a surgeon inserts an endoscope through a small incision or natural opening of the body (e.g., but not limited to the mouth or nostril). An endoscope is a rigid or flexible tube with a camera attached thereto that transmits real-time images to a video monitor (e.g., a display of the system or a display connected to the system), which the surgeon uses to help guide his tools through the same incision / opening or through different incisions / openings. An endoscope allows a surgeon to view relevant areas of the body in detail without having to cut and expose the relevant areas. This technology allows surgeons to see inside the patient's body and operate through much smaller incisions than are required for traditional open surgery. Therefore, in a typical robotic endoscopic surgery, there is an endoscope attached to a surgical robot arm and one or more other surgical instruments, for example, a pair of forceps and / or scissors attached to one or more other surgical robot arms.
[0083] Fig.11 An exemplary endoscope 1100 is shown that can be attached to the end of a robotic arm for use in minimally invasive surgery. The endoscope 1100 has a distal end 1102 for insertion into a surgical site in a patient, and a proximal end 1104. The distal end 1102 is connected to the proximal end 1104 by an elongated shaft 1106. The proximal end 1104 includes an interface 1108 for engaging the end of the robotic arm. The endoscope 1100 has a power source and a light source for illuminating the surgical site. The endoscope 1100 also has a data line for extracting image data from the surgical site. These can all be independent of the robotic arm and attached to the proximal end 1104 of the endoscope 1100 on the outside of the robotic arm, as shown in FIG. Fig.11 As shown. Fig.11 , power is applied through rod 1112, image data is extracted through rod 1112, and light is applied through optical rod 1110. In alternative embodiments, any one or more of light input, power input, and data output may be applied / extracted to the endoscope through a robotic arm. Endoscope 1100 is mounted to the end of a robotic arm. Endoscope interface 1108 engages a complementary interface of the robotic arm. Endoscope 1100 may be attached to and detached from the robotic arm via the robotic arm and the endoscope interface. In some cases, endoscope 1100 may be operated independently of the robotic arm in its detached state. In other words, in these cases, when detached from the robotic arm, endoscope 1100 may be manually operated by a member of the operating room staff.
[0084] In addition to using images captured by an endoscope during surgery (which may be collectively referred to herein as endoscopic video), images captured by an endoscope may be recorded and subsequently used for a variety of purposes, such as, but not limited to, learning and / or teaching surgical procedures, and evaluating and / or reviewing the surgeon's performance (either by a third party or by the surgeon himself).
[0085] Suitably, an imaging device is provided. An imaging device may be provided at a surgical site. The imaging device may include an endoscope. A robotic system may include an imaging device. For example, a robotic system may include an arm to which the imaging device may be attached, and the arm is configured to move the imaging device around the surgical site. The imaging device is configured to output an image signal or image feed representing an image of the surgical site at which the imaging device is located and / or including an image of the surgical site. The image signal may include a video signal. Suitably, the imaging device is configured to capture video data.
[0086] Although the above description refers to a single screen as a display device, in some examples, the robotic surgical system includes multiple display devices or screens. The screens are suitably configured to display images as two-dimensional images and / or as three-dimensional images. The screens may be provided on a single user console, or two or more consoles may each include at least one screen. This allows for additional viewing screens that may be used to allow persons other than the console user to view the surgical site, such as for training, and / or for viewing by others in the operating room.
[0087] Indication of surgical site
[0088] A representation of the surgical site can be displayed on a display to allow the surgeon to see the site and enable them to perform the surgical procedure. The representation can be or can include an image feed from an imaging device, such as an endoscope located at the surgical site. The representation of the surgical site can include a 2D or 3D representation. A 3D representation can be generated from a 2D original representation by suitable processing. In some examples, the 3D representation can include a 3D model, such as a 3D model of a body or a part of a body. For example, when a surgical procedure is to be performed in the abdominal cavity, the 3D model can represent such an abdominal cavity. In some examples, the 3D model can be derived at least in part from data such as physical data related to the patient. For example, the data can include data from a scan such as an MRI scan. The 3D model can be modified or selected based on an understanding of the patient to undergo the surgical procedure, for example, based on an understanding of the physiology or pathophysiology of the person. In some examples, the representation is based on both the captured image feed and a model (e.g., a 3D model) of the site.
[0089] Suitably, the 3D model is a 3D anatomical model. The model may include simulation or simulation data. The model may include model data that has been established based on data obtained in relation to an earlier procedure. The earlier procedure may be for a single patient, which may be the same patient. The earlier procedure may be for more than one patient. The earlier procedure may be of the same or similar type to the procedure being planned or performed.
[0090] The representation of the surgical site may change during the surgical procedure. For example, the representation may change when the patient moves (e.g., when the patient's orientation changes, such as when the patient table is tilted) and / or when the imaging device changes its position (or location) and / or orientation relative to the surgical site. The imaging device may translate through the surgical site, and / or zoom in or out. Such changes may also change the representation of the site displayed on the display. For example, the portion of the model displayed on the display may be changed during the surgical procedure. The portion of the model displayed may change based on the determined position of the end effector. For example, the position of the end effector may be determined based on control signals sent to the end effector and / or kinematic data of the end effector (or more generally, the system). The portion of the model displayed on the display may be changed by changing the digital scaling of the representation, i.e., without changing the scaling of the imaging device itself; the change may be achieved by processing the representation of the surgical site.
[0091] Typically, the displayed representation allows the surgeon to see the position of the end effector and control the end effector accordingly in order to perform the surgical procedure. Because the surgeon sees the displayed portion of the surgical site rather than the entire body, the surgeon may become disoriented during the operation. In other words, the surgeon may lose track of which part of the surgical site is being viewed, and / or lose track of what orientation the portion of the surgical site is being viewed in. This may have effects, including extending the time it takes to perform the procedure because the surgeon needs additional time to properly orient himself before continuing. The inventors of the present application have recognized that additional information can be provided to the surgeon (and / or other members of the operating room staff), preferably during the surgical procedure. Such additional information can be provided by enhancing the representation of the surgical site, and / or by enhancing the display of the representation of the surgical site.
[0092] Additionally or alternatively, it may be desirable to display on a display information related to the state of at least a portion of the patient's anatomy (e.g., anatomical structures at the surgical site), and / or information related to the health of the patient undergoing the procedure. For example, this information may be displayed on the same display as the representation of the surgical site. This information may be used to enhance the display of the representation of the surgical site. Such information need not be provided on the same display as the representation of the surgical site. This information may be provided on or as part of the representation of the surgical site, and / or provided separately from the representation of the surgical site.
[0093] As will be explained in more detail below, this information may include, for example, information regarding the patient's heart rate, the patient's breathing rate, movement of at least a portion of the patient's anatomy, a measure of tissue perfusion of at least a portion of the anatomical structure, a measure of blood flow and / or oxygen delivery to at least a portion of the patient's anatomy or at least a portion of the anatomical structure, etc. Suitably, this information includes an indication of the patient's health status, such as whether an identified parameter associated with the patient's anatomy is within an acceptable range.
[0094] Enhancement
[0095] Enhancing the representation of the surgical site can allow for the inclusion of a visual aid to a user, such as a surgeon, which can, for example, help orient the displayed representation of the surgical site. In the examples discussed herein, "orientation" appropriately refers to determining or understanding the orientation of the displayed representation. "Orientation" can include understanding what orientation the displayed representation is in. The methods of the technology of the present invention can enable surgical procedures to be completed more quickly and / or more accurately. The enhancement can provide enhanced human-computer interaction, for example, between a surgeon using the system to control an end effector and the operation of the end effector or a robotic system in general. As will be explained herein, such enhancements can enable users of the system to perform technical tasks more repeatedly, more reliably, and / or more quickly, among other things.
[0096] Augmenting the representation of the surgical site may allow one or more health indicators to be displayed to a user, such as a surgeon. Such augmentation may provide enhanced human-computer interaction. Augmentation may enable the surgeon to more quickly determine parameters of anatomical structures associated with the procedure being performed, and to perform the procedure more quickly, more accurately, more safely, etc.
[0097] The enhancements to the representation of the surgical site may be performed before, during, and / or after the surgical procedure. Optionally, at least one enhancement is displayed on or as part of the representation during the surgical procedure. In some examples, this may enable a user, such as a surgeon, to more accurately orient the representation in real time while performing the procedure.
[0098] The enhancement can involve one or more of the path taken by the surgeon (e.g., one or more end effectors controlled by the surgeon) through the site, actions taken at one or more points at the site, features present (or absent) at particular locations at the site, movement of parts of the site, etc.
[0099] Enhancements can be added automatically or in response to user input. In the case of adding enhancements in response to user input, the user can specify a portion of the enhancement, and another portion of the enhancement can appear automatically. An example of this is feature recognition. A processor such as an image processor can monitor the representation of the part and determine which features are or may be present in the representation or in the portion of the representation displayed. The image processor or another processor that operates in dependence on the output from the image processor can automatically annotate or mark one or more determined features. Alternatively, the user can indicate a feature, and the system can automatically select a label or tag to be applied to the feature, for example, based on automatic feature determination, for example, by image recognition or image matching. The label or tag may include a confidence measure (e.g., a probability measure), and / or a confidence statement about the feature.
[0100] The above example illustrates one use of the present technology. A user may mark or identify features, such as anatomical features, in a displayed image feed or model of a surgical site. Enhancements may be added to the representation based on user input. For example, the user input may indicate a feature or location of a feature to which an enhancement is to be added. The user may also indicate or specify the nature of the enhancement to be added, such as a name or label for the enhancement, and / or the type of enhancement.
[0101] In a surgical robot system, the positions of the various parts of the surgical robot in 3D space are generally known. For example, the position of the end effector in 3D space is known or can be determined based on kinematic control information. Such kinematic data already exists in the system, so it may not be necessary to calculate additional kinematic data. Therefore, the present technology can advantageously utilize existing information in the system to provide additional benefits to users of the system. The enhancement can be an enhancement in 3D space related to the representation of the surgical site. When the representation moves in the display, the associated movement of the enhancement can take into account depth, rotation and / or lateral translation, etc. This approach can increase the accuracy of the position of the enhancement, and therefore increase the accuracy of the interaction with the enhancement-based system.
[0102] In one example, a user such as a surgeon may wish to indicate a location on an anatomical feature to make an incision or insert a stitch. This may be useful in the case where the surgeon finds a suitable location for an incision or stitch, but may wish to perform another task before making the incision or inserting the stitch. Enabling the surgeon to add such an enhancement enables the surgeon to quickly and accurately return to the location indicated by the enhancement. The location of such an enhancement relative to the current location can assist in navigating through the surgical site and / or orienting the representation of the surgical site. For example, viewing such an enhancement on the displayed portion of the representation can enable the user to quickly and accurately determine the portion of the site being displayed, thereby facilitating more efficient human-computer interaction. In the context of a surgical procedure, this can reduce surgical time by allowing the surgeon to minimize the time required to reposition the identified location. Reducing surgical time is beneficial to patients because it can reduce the risk of complications and assist recovery time. Reducing surgical time may be beneficial to patients and hospitals because it can lead to an increase in the number of surgeries that can be performed, which in turn may lead to a reduction in the cost of each surgery.
[0103] The enhancement need not be displayed on the representation or as part of the representation. For example, an enhancement may be added to a portion of a representation and then the displayed image may be changed (e.g., by zooming or translating the imaging device and / or the representation) so that the enhancement(s) are no longer visible on the display. In this case, the system may be configured to indicate the presence and / or location of the enhancement(s) by adding another enhancement, which may be of a different type, to the representation or the displayed portion of the representation. Such an enhancement may indicate a direction and / or distance to or toward a feature of the representation, such as another enhancement and / or an anatomical feature in the representation. This situation will be described in more detail below.
[0104] In some examples, actions can be performed automatically or in an assisted manner. For example, a common action such as tying a knot can be started by a surgeon, and the remainder of the action can be automatically completed to assist the surgeon. Such auxiliary actions are useful in repetitive movements or actions. Enhancements can be provided at locations where automatic actions are to be performed or where assistance is needed to perform the action. For example, a surgeon can indicate one or more locations where knots are to be tied, and / or indicate to the system that assistance is needed to tie a knot midway through tying a knot. Once an auxiliary action such as tying a knot has been performed, a pre-existing enhancement can be modified to indicate that an auxiliary action has been performed. Additionally or alternatively, a new enhancement can be provided to indicate that an auxiliary action has been performed. The newly enhanced location can provide an indication of where to perform the auxiliary action.
[0105] The surgeon appropriately identifies a suitable position (the most suitable position, orientation, spacing to each other), for example, a position where the surgeon can perform a task and / or a position where the system can perform an auxiliary action. In some examples, the identification of a suitable position can be performed automatically. Such automatic identification can be performed based on previously identified positions, such as positions contained in a model of the surgical site. The model can be constructed by considering one or more previous procedures. Therefore, automatic identification can benefit from a combined understanding of earlier procedures, which may have been performed by the same or another or more surgeons compared to the procedure to be performed. In this way, a more junior surgeon can benefit from the knowledge of more experienced colleagues without the need for those more experienced colleagues to be present during the operation or to be directly consulted before the operation. The automatic identification of the position can be based on the confirmation of the user of the system. For example, the system can suggest an optional position. The suggested optional position can be associated with a confidence factor, which can indicate the confidence that the suggested optional position is suitable for the current procedure. The confidence factor can be determined in any appropriate manner. In some examples, the confidence factor can be determined by determining the similarity between a previous procedure and the current procedure and / or the similarity between a model used in a previous procedure and the model used in the current procedure. The confidence factor can be determined based on the user who performed one or more of the previous procedures. For example, where the confidence factor is based at least in part on a procedure performed by a relatively more experienced surgeon, the confidence factor may be higher, indicating a higher degree of confidence.
[0106] The surgeon may indicate the location, for example by identifying the location and / or by confirming a suggested location, so that the system may enhance the representation of the surgical site accordingly. In the case of autonomously or partially autonomously performing actions, the surgeon may then indicate to the system that such at least partially autonomous actions may be performed.
[0107] Markings or enhancements may be added manually or automatically. Automatic enhancement may occur based on image recognition. This will be discussed in more detail below. Enhancements may be added based on an indicator. The indicator may be displayed on the display. The indicator may include a cursor or other pointer. The indicator may indicate a point or area of the display. For example, the indicator may be in the form of a shape such as a circle, the inner portion of which is indicated by the indicator. The indicator may include an outer edge that appears solid and / or colored on the display, for example, the circumference of a circular indicator may be a solid black or white line. The outer edge may flash to highlight the edge. The indicator may have a different color, contrast, 3D appearance, and / or focus compared to the rest of the display. For example, the interior of the indicator shape may be colored, and the exterior of the indicator may be black and white. In some examples, the interior of the indicator may be emphasized by focusing, while the exterior of the indicator may be out of focus. Any other suitable way of highlighting the indicator may be used.
[0108] Enhancements may be assigned tags. Enhanced annotation may be performed manually, automatically, or some combination of manual and automatic. For example, once an enhancement is added, a user may specify a tag. This may be accomplished by a user inputting data, such as by inputting the tag on a keyboard, via a voice interface, via a gesture interface, via a pointer interface such as a mouse, and / or in any other suitable manner. A combination of these methods of inputting data may be used. In some examples, a voice response input device such as a microphone may be used to input tags. A user may speak aloud tags that may be applied to an enhancement. The tag may be added to an enhancement by selecting the tag from a set of possible tags. The selection of the tag may be performed via a menu system. The menu system may include all possible tags. The menu system may include or provide a set of possible tags. The set of possible tags may be pre-selected. The set of possible tags may be selected based on at least one of a user profile of the user, a surgical procedure being performed, a location of a surgical site, and the like. In some examples, a user may pre-define tags for use. User-defined tags may be in addition to system-defined tags. Tags may be associated with one or more surgical procedures so that a subset of tags may be used for related surgical procedures. For example, the label "artery" may be appropriately used for a wide range of procedures. In the case where the kidney is not visible at the surgical site for a given procedure, the label "kidney" need not be provided.
[0109] Image recognition can be used to assist in annotation enhancement. The image recognition algorithm can select a set of possible labels to be applied to the enhancement. The image recognition algorithm can select the set based on the model (e.g., 3D model) and / or the location of the surgical site.
[0110] Labels may include text labels and label emphasis. The text label may, for example, provide the name of the enhanced feature, such as "kidney" or "artery". Any desired level of detail may be provided in the text label. Label emphasis may include a visual indication of a point or area of the representation. For example, the edge of a feature may be highlighted, such as the edge of an artery or the edge of an organ such as a kidney. Additionally or alternatively, an internal region of a feature may be highlighted, such as an area defined by the edge of the feature. For example, an internal structure such as a blood vessel may be highlighted in outline. In another example, the edge of a tumor (with or without a resection margin) may be highlighted. The highlighting may be determined depending on one or more other imaging modalities such as a preoperative CT scan. The highlighting may take the form of one or more of an outline, a shadow, a coloring, a 2D / 3D appearance change, a different contrast, and the like. In one example, an organ such as a kidney (or that portion of an organ visible in the representation) may be shaded. This may assist the user by providing a clear indication of the whereabouts of the feature (in this example, the kidney) in the representation. It may be desirable to shade the feature when the surgical procedure does not envision interacting with the feature. When the surgical procedure envisions interacting with the features to be highlighted, alternative forms of highlighting, such as forms of highlighting that do not obscure the features, may be used. The edges of the features to be highlighted may be determined manually and / or automatically. In one example, the surgeon may direct the indicator to a point on the edge and indicate to the system that the point represents a point on the edge of the feature. The surgeon may delineate the edge, or provide one or more points along the edge, based on which the remainder of the edge may be interpolated, extrapolated, and / or otherwise determined by the system, for example, through image analysis. For example, the system may be configured to determine the difference in one or more image characteristics (e.g., one or more of color, contrast, luminosity, depth, etc.) on either side of the edge, and depict a line through the image that follows the changes in the one or more characteristics. Once the feature has been annotated, the system may perform image analysis and / or tracking to consistently annotate the feature as the representation changes.
[0111] Enhancements can be added or selected by a user of the system. For example, a controller that may include an input device can be configured to output an enhancement signal. The enhancement signal can be associated with the position of an indicator on a display. The enhancement signal can indicate the position on the display of the indicator. For example, when a user controls a controller to output an enhancement signal, the system can be configured to add an enhancement at the position of the indicator. In some examples, where a menu system is to be navigated, the position of an indicator in the menu system (i.e., a menu system value or label) can be selected by activating a controller so that an enhancement signal is output.
[0112] The controller may be configured to output an enhanced signal in response to activating a user control at the controller. The user control may include a button or a switch. The user control may include a keyboard. The user control may include a resistive sensor, a capacitive sensor, a trackball, a joystick or a thumb stick, a voice sensor, a gesture sensor, and / or any combination of these and other user input devices. In some examples, the controller is configured to output an enhanced signal in response to receiving a user input at an input device. The input device may include a user control. The input device appropriately enables the user to control an indicator on the display. For example, the movement of a joystick or a thumb stick on the input device may result in a corresponding movement of the indicator on the display. In some examples, the movement of the input device may result in a corresponding movement of the indicator on the display. For example, a manual controller may be moved in three dimensions via, for example, a manual controller arm link and a universal joint. The movement of the indicator may be based on at least one dimension of the movement of the manual controller. For example, a two-dimensional (e.g., a dimension defining an xy plane or a yz plane) movement may control the two-dimensional movement of the indicator on the display. This method allows the indicator to be moved by the user on the display in an easy and intuitive manner. In one example, three-dimensional movement of a hand controller may be used to control the position of an end effector, and a thumbstick on one (or in some examples, two) input devices may be used to control the position of a pointer in the representation of the surgical site.
[0113] The controller may include or may be part of a user console of the surgical robot.
[0114] exist Figure 4 An exemplary illustration of a user console, such as a surgeon console 400, is shown in FIG. A user, such as a surgeon, can control the robotic arm 302 and the instrument 320 coupled to the robotic arm 302 via the input device 304 at the surgeon console 400 and can manipulate the robotic arm and / or the instrument as desired. Figure 4 As shown in , the surgeon console 400 includes a non-contact input device 410, which includes at least one of a gesture sensor such as an optical gesture sensor and a voice sensor. The surgeon console 400 includes a touch screen input device 420. Additionally or alternatively, the display 306 may include a touch screen input. The surgeon console 400 includes a foot-operated input device 430, such as a foot pedal. Figure 4 In the figure, one of each of the devices 410, 420, 430 is shown, but it should be understood that any combination of any number of these devices may be provided in other examples. It is not necessary to provide all input devices or all types of input devices in all examples.
[0115] exist Figure 5, a schematic diagram of the configuration of a portion of a controller such as user console 400 is shown in . The controller 500 includes a surgeon command interface 510. The system also includes a command processor unit 520 and a display 530. The command processor unit 520 is coupled to both the surgeon command interface 510 and the display 530. The surgeon command interface 510 is configured to be operable by a user such as a surgeon. The surgeon command interface 510 allows a user to input commands to a surgical robotic system. The user may use the command interface to control the operation of the surgical robotic system, for example by controlling one or more robotic arms and / or end effectors coupled to the robotic arms. The command interface 510 includes an input device 512. The input device 512 may be, for example, Figure 4 The input device shown at 304. Figure 5 Only one input device is shown in FIG. 5 , but more than one input device may be provided. Typically, two input devices 512 are provided, one for use by each of the user's hands.
[0116] In some examples, the input device may be a handheld controller for manipulation by a surgeon controlling a surgical robot. For example, the input device may be communicatively coupled to a robotic arm and instrument, whereby the position and operation of an end effector of the instrument may be controlled by the surgeon, for example, at a surgical site.
[0117] A second input device 514 is provided at the command interface 510. In some examples, the second input device is of the same type as the input device 512. In other examples, the second input device 514 is a different type of device than the input device 512. For example, the second input device 514 may include one or more of a voice interface, a gesture interface, and a touch interface. Thus, the second input device 514 may respond to voice commands, gestures, and / or touches received at the second input device. For example, the input device 514 may be a Figure 4 The device shown at 306, 410, 420 or 430.
[0118] This arrangement allows the surgeon to use the second input device to enhance the representation of the surgical site. For example, during a surgical procedure, the surgeon may use the input device to perform a portion of the surgical procedure. At a time point selected by the surgeon, it may be desirable to enhance the portion of the representation of the surgical site where the indicator controlled by the input device is located. For example, in the case where the surgeon has just completed a stitch, the surgeon may wish to enhance the representation of the surgical site at or near the stitch. This may allow the surgeon to record the location of the stitch on the representation of the surgical site. This may enable the surgeon (and / or another person) to find the location, i.e., the stitch, at a later time (e.g., later in the procedure or during a post-procedural review). As the surgical procedure proceeds, the portion of the surgical site displayed on the display is likely to change. Enhancing the representation of the surgical site so as to record the location of the stitch allows the surgeon to determine the orientation or direction and / or distance of the location from the current location. This may help to help the surgeon appropriately orient the displayed representation, such as when the stitch is no longer displayed on the display. The provision of a second input device allows the addition of enhancements without requiring the surgeon to change the way the end effector is controlled using the input device. That is, the end effector does not need to be moved to add enhancements. For example, the surgeon can say "stitch", and the second input device can detect the surgeon's voice input, determine the command (here "stitch") and cause the generation of a signal to cause the representation of the surgical site to be enhanced accordingly. Additionally or alternatively, the surgeon can perform a gesture to be detected by a gesture-sensitive input device (e.g., a camera). The surgeon can touch a touch-responsive input device, such as a display screen on which a representation of the surgical site is displayed. The second input device does not need to be controlled by the same person who controls the input device. In some examples, the surgeon will control the input device in order to perform the surgical procedure. The second input device can be used by a surgical assistant or other member of the operating room staff. In some examples, the second input device can take a form similar to the input device. In some examples, one or both of the input device and the second input device can be used to navigate a menu of a surgical robot system, such as a menu displayed on a display. The menu options can be preconfigured. The menu options can be preconfigured according to one or more of the following: user preferences, the type of surgical procedure, the stage in the surgical procedure, the type and / or number of end effectors connected to the system, etc.
[0119] In all examples, the second input device 514 need not be provided at the same command interface 510 as the input device 512. For example, the input device may be at or associated with one user console, and the second input device may be at or associated with the same or a different user console. The second input device may be configured to generate a signal. The processor may be configured to receive the generated signal from the second input device and enhance the representation of the surgical site or modify the enhancement accordingly.
[0120] For example, two surgeon consoles may be provided, each for controlling a separate robotic arm (or separate set of robotic arms). For example, one surgeon console may be used to control a robotic arm operating in the abdominal cavity, while another surgeon console may be used to control a robotic arm operating transanally. It is useful for displays associated with one surgeon console to be enhanced with enhancements that are dependent on or derived from another surgeon console.
[0121] Providing a different type of second input device 514 to the input device 512 advantageously allows a user of the command interface 510 to more easily achieve control of the robotic system. For example, when a user is controlling two manually operable input devices 512, the user may need to let go of one of the input devices in order to be able to control another manually operable input device. The user can advantageously achieve control of the second input device 514 without having to give up control of either of the two input devices 512. For example, where the second input device 514 includes a voice interface, the user can speak the command out loud. This can be done while still holding the input device(s) 512.
[0122] like Figure 3 As shown in , the command interface is connected to the control unit 309 for controlling the robot arm and the end effector of the surgical robot system. Figure 5 , the command interface 510 is communicatively coupled to a command processor unit 520. The command processor unit 520 includes a processor 521. The processor 521 is configured to communicate with the command interface or controller 510 and is capable of controlling enhancements of the representation of the surgical site. The processor 521 may be configured to perform image processing, such as image recognition. Additionally or alternatively, an image processor 522 may be provided. The image processor 522 may be configured to perform image processing, such as image recognition. The processor 521 and / or the image processor 522 may be configured to perform edge detection, spectral analysis, etc.
[0123] The processor 521 and the optional image processor 522 can access the memory 523. Figure 5In the example shown, memory is provided at the command processor unit 520. In some examples, memory may be provided elsewhere, and / or additional memory may be provided elsewhere. Providing processor 523 locally at command processor unit 520 may improve memory access times. Providing memory 523 at least partially remote from command processor unit 520 may enable the use of larger memory without requiring command processor unit 520 to have a larger physical size. Where at least a portion of memory 523 is provided remote from command processor unit 520, the remote portion of memory 523 may be coupled to command processor unit 520 via one or more of a wired connection and a wireless connection.
[0124] The memory may store programs executed by the processor 521 and / or the image processor 522. The memory 523 may be used to store the results of the processing, and optionally to store intermediate processing results. The memory 523 may store a representation of the surgical site or at least a portion of the surgical site. The memory 523 may store enhancements regarding the representation of the surgical site. The enhancements may be stored as part of the representation of the surgical site, or stored separately therefrom. In some examples, (multiple) enhancements may be stored at the memory 523 at the command processor unit, and the representation of the surgical site on which the stored (multiple) enhancements are based may be stored at a remote memory. In some examples, one or more enhancements may be stored at the memory 523 at the command processor unit, and one or more enhancements and the representation of the surgical site on which the enhancements are based may be stored at a remote memory.
[0125] The command processor unit 520 may include calculation logic 525. The calculation logic may include distance calculation logic 526, area calculation logic 527, volume calculation logic 528, and / or user-defined calculation logic 529. The calculation logic may include time-varying change calculation logic. The calculation logic is suitably configured to calculate one or more metrics based on at least one enhancement, as described in more detail elsewhere herein.
[0126] Specify Enhancement
[0127] An example of how a user may specify enhancements is described below.
[0128] As discussed, the indicator on the display is controlled by the controller. The indicator may include a pointer such as a mouse pointer. The indicator such as a mouse pointer is typically a virtual indicator in that it does not exist at the surgical site, but is added to the representation of the site, or overlaid on the representation when the representation is displayed on the display.
[0129] In some examples, the indicator may include a physical indicator. The physical indicator may be provided as part of the surgical robot. For example, the physical indicator may be provided as part of the end effector. Since the end effector may be viewed on a display, the end effector itself may be used as a physical indicator.
[0130] The physical indicator may include an indicator portion of the end effector. For example, where the end effector is a gripper tool having a pair of jaws, the indicator may include the tip of the jaws in a closed position. The end effector may have an indicator mode in which the end effector may act as an indicator. The indicator mode may be entered when the jaws are fully closed or closed beyond a predetermined point. Additionally or alternatively, the user may be able to select between an indicator mode of the end effector and a non-indicator mode of the end effector via, for example, a control at a controller (e.g., a control at one or the other of the input device and the second input device). Suitably, regardless of whether the indicator mode or the non-indicator mode is selected, the end effector remains controllable in a conventional manner so that there is no need for any interruption in the program being executed. The jaws of the gripper (or more generally, any other configuration or operation of the end effector) do not need to be in a specific configuration for the end effector to act as an indicator.
[0131] In some examples, the indicator may include a specific portion of the end effector. For example, the tip of a jaw, the tip of a needle, etc. For example, where the end effector is a gripper, the tip of the leftmost jaw may serve as an indicator. Here, the "leftmost jaw" may be any jaw on the left side of the display as observed by the user, or may be a given jaw of the end effector, regardless of the orientation of the observation. The indicator portion of the end effector may be indicated as follows: for example, by a mark on the end effector itself (which may be a difference in color, shape, and / or configuration from another part of the end effector) and / or virtually indicated on the display. Virtually indicating an indicator portion of the end effector has the advantage that such indication may be changed according to one or more of user preferences, operating conditions, an ongoing surgical procedure, etc.
[0132] An augmentation may be added to the representation of the surgical site at a location indicated by the indicator. For example, an augmentation may be added at a location where the indicator is located. The indicator location may be a location on a display (i.e., a location in the two-dimensional screen space of the display). The indicator location may be a projection of the indicator on the display onto a feature in the three-dimensional representation.
[0133] The enhancement is appropriately added in response to the controller outputting the enhancement signal.The enhancement signal may be output by the input device.
[0134] For example, enhancement can be added to the feature indicated by the indicator, or to the position on the feature indicated by the indicator. The user can control the indicator to make it a desired feature in the displayed representation, such as above an organ. The user can cause the controller to output an enhancement signal. In response, the processor 521 can enhance the representation of the surgical site, for example, by adding enhancement at the indicator position. The user can open a menu, or the menu can be automatically opened when the enhancement is added. The user can use a controller (e.g., an input device or a second input device) to navigate the menu in order to select the desired label for enhancement. In other examples, the processor 521 or the image processor 522 can determine the feature to which the enhancement has been added. This determination can be made, for example, by image recognition and / or image matching or other image analysis based on a 3D model, which can be derived at least in part from data derived from a scan such as an MRI scan. Then, appropriate labels for the determined features can be added to the representation or otherwise associated with the enhancement. In some examples, the label can be the name of the feature, for example, "artery", "kidney", etc.
[0135] Enhancements can be added automatically by the system, for example by the processor 521. The user does not need to take any action to cause the controller to output an enhancement signal. In some examples, the controller may include a clock or have access to a clock, and the enhancement signal may be output by the controller according to the clock. For example, the enhancement signal may be output at a preselected time or at a preselected frequency. This may allow data to be obtained about the time taken by different surgeons to reach a given point in the procedure, or allow data to be obtained about the point in the procedure reached by the surgeon at a given time after starting the procedure. Such data may enhance post-procedural analysis or off-line analysis of a performed procedure. Such enhancements may be done automatically without user input. This method enables data to be obtained repeatedly and efficiently.
[0136] Enhancements can be added automatically or manually at any desired stage in the procedure. In some examples, the system can monitor the procedure and compare the procedure being performed with one or more previous procedures. Such comparisons can be performed continuously or periodically. Periodic comparisons can use less processing power than continuous comparisons and can therefore be preferred. The comparison can be completed by image processing of a representation of the surgical site relative to the procedure being performed and a representation of the corresponding site relative to one or more previous procedures. The representation of the corresponding site can be based on an average value of a model associated with one or more previous procedures. When an action occurs in the procedure, the comparison can be preferably performed at a relatively greater rate than when no action occurs. Therefore, when performing the comparison, for example, the movement of the end effector at the site can be considered. In the case of a large movement rate, the comparison rate can be increased. A higher comparison rate during the active period enables accurate comparisons while saving processing power during relative inactivity. The comparison rate can be determined based on robot kinematic data. For example, the comparison rate can be determined based on the speed of a portion of the end effector and / or the operation of the end effector.
[0137] An enhancement may be added when it is determined that the current procedure deviates from one or more previous procedures. The deviation may involve the time or relative time at which the various stages of the procedure are performed. The deviation may involve the location at which the various stages of the procedure are performed. When the current procedure differs from the expected behavior (e.g., the behavior expected based on one or more previous procedures or based on the procedure plan (based on a standard plan / prescription card or a plan / prescription card written just for the current patient)) by more than a threshold amount, it may be determined that a deviation has occurred. The threshold amount may be a time period over which an action is performed. For example, when an action is performed more than 5 seconds, 10 seconds, 30 seconds, 1 minute, 2 minutes, or 5 minutes earlier or later than the expected time, it may be determined that a deviation has occurred. The threshold amount may be a distance from a predetermined location in a representation of the surgical site. For example, when an action is performed more than 1 mm, 2 mm, 5 mm, 10 mm, or 20 mm from the expected location, it may be determined that a deviation has occurred. This approach allows for useful comparisons between procedures while allowing for differences such as physiological differences to be taken into account.
[0138] In some examples, the representation may be enhanced if an error condition is determined to have occurred. The error condition may include a robot arm and / or instrument failure. The threshold for determining the occurrence of an error condition may be user-definable and / or preset.
[0139] The enhancement signal may be generated from any desired source, such as a telematics data source. For example, the enhancement signal may be generated from one or more of determining a change in instrument, a change in manual controller-arm association, a change in electrosurgical mode, movement of the endoscope, re-indexing at least one manual controller, etc. The enhancement signal may be generated from a specific combination of any actions described herein occurring.
[0140] The enhanced signal may indicate a feature displayed on the display. The enhanced signal may include position data indicating the position of the indicator relative to the representation of the surgical site. For example, the enhanced signal may include data related to the position of the indicator relative to the 3D model. The controller (e.g., an input device) may be configured to provide the position data. The position data may be obtained based on the displayed indicator.
[0141] The position data may include joint data, such as data associated with one or more joints of a robot arm and / or an end effector. The joint data may include joint position data. For example, the joint position data may include data related to the position, orientation and / or configuration of a joint of a robot arm supporting an instrument, and / or data related to the position, orientation and / or configuration of a joint of an instrument and / or an end effector of the instrument. The joint data may include kinematic data. For example, the kinematic data may include data related to a change in the position, orientation and / or configuration of one of the multiple joints of the robot arm and the instrument. The kinematic data may include data related to the rate of change of the position, orientation and / or configuration of one of the multiple joints of the robot arm and the instrument. The kinematic data may include initial position data of one or more joints, from which the position change occurs. Providing such position data is particularly useful in the case where the end effector acts as an indicator. In this case, the 3D position of the indicator (i.e., the end effector) will be known. Therefore, enhancements can be added to the representation in a highly accurate manner. This method can also provide savings in terms of required processing because the position data already exists in the system and does not need to be recalculated.
[0142] In the case where it is necessary to more accurately identify the features in the displayed representation, the features can be marked at more than one position. The marks can be entered at intervals on the features, such as at positions spaced laterally. In some examples, it may be desirable to change the orientation or scaling of the representation between one mark and another. In some examples, it may be desirable to change the viewing conditions between one mark and another. The viewing conditions may include whether the view is a 2D view or a 3D view, image contrast, image coloring and / or shading, and / or whether image enhancement exists. In some examples, when the patient is in one position, the feature can be indicated by one or more marks, and the patient then moves (such movement may include moving one or more limbs of the patient, and / or moving the patient's orientation against the operating table, etc.). In some cases, patient movement may cause parts of the surgical site to move relative to each other. For example, a change in the patient's orientation may cause an organ to move due to different gravitational effects. When the patient is in a new position, the same feature can be marked with one or more marks. Marking features in this way can help the system to be robust against patient movement or other similar effects. For example, marking features at different patient positions can help to track the feature during a procedure, during which the patient may change position. Image processing can be performed to identify whether two (or more) marked locations are part of the same feature, such as points spaced apart along an artery or points at different locations on the kidney. When one point is on one feature (such as on an artery) and another point is on a different feature (not on an artery), the system can prompt the user to mark one or more additional points. The feature can be selected as a feature to be enhanced based on the relative number of marks at the feature compared to marks not at the feature. In this way, there is no need for the user to remove or adjust accidentally added marks (which can be time consuming), but one or more additional marks can be made (which can be faster for the user) to indicate the features to be enhanced.
[0143] Once the representation has been enhanced, the enhancement may be moved, for example, automatically or by a user. The enhancement may be moved by dragging and dropping it on the display. The controller may be configured to allow such dragging and dropping, for example, via a computer mouse cursor, an input device, and / or a second input device.
[0144] As mentioned, enhancement can be marked by the user using free text and / or selecting from a set of labels. Enhanced annotation can occur automatically, or at least partially automatically. For example, the representation can be processed by processor 521 or image processor 522, and image recognition technology for specifying what a particular feature is. In some cases, it may be difficult to use such image recognition technology alone. This may be because the surgical site being imaged may not have a high enough contrast, and / or it may be poorly illuminated. Advantageously, the technology of the present invention allows for enhancement in image processing. A comparison can be made between the representation of the site of the current procedure and one or more representations of previous procedures and / or models. The representation of the previous procedure and / or model appropriately includes at least one marked feature or known feature. When it is determined that the feature in the current representation is the same or similar to the feature in the previous representation or model, the label of the feature in the previous representation or model can be selected by the user, or automatically applied to the enhancement of the feature in the current representation. It can be determined whether the label is available for user selection or automatically applied based on the confidence factor associated with the label. This method allows the system to "learn" the identity of different features by establishing a database of similar types of procedures and / or models of procedures. Therefore, the automatic annotations or suggestions of possible labels can be made more accurate. This can save the user time in correctly annotating features. Annotating features more accurately can improve the accuracy of tasks based on those labels.
[0145] The processor may be configured to monitor or track enhancements and / or features of the representation, such as anatomical features, such as organs or vessels. In some examples, features such as anatomical features may be determined automatically. For example, the system may determine the features by image processing. Such monitoring or tracking of enhancements and / or features is useful when the representation of the surgical site changes. For example, the viewing position of the representation may change. This may be due to a lateral movement of the viewing position or an angular change in the viewing position, for example, as may occur when the position and / or orientation of the imaging device changes. Usefully, the system is configured so that as the portion of the representation displayed on the display changes, the enhancement maintains its position relative to the representation of the surgical site. For example, in the case where the imaging device moves to the right, thereby causing the representation based on the image output of the imaging device to move to the left on the display, the enhancement will also move to the left. The system is appropriately configured so that the enhancement moves in alignment with the representation of the surgical site, that is, the system is configured so that the enhancement moves with the representation. For example, in the case where the enhancement is added to a specific feature in the representation, the enhancement appropriately moves with the movement of the feature. Such movement of the feature may occur when the translation and / or zoom change relative to the representation of the surgical site. Movement of features may occur in other ways. For example, a surgical procedure may involve moving features. Features such as organs may move due to one or more of breathing, heartbeat, and gravity (e.g., when adjusting a patient table).
[0146] As described above, enhancements may be added manually and / or automatically. Enhancements may be added based on determining that enhancement criteria are met. Processor 521 may be configured to determine whether enhancement criteria are met.
[0147] The enhancement criteria may include determining whether a surgical instrument is attached to the system, detached from the system, operated by the system, and / or whether there is a change in the state of the surgical instrument. The enhancement criteria may include determining that there is an image recognition signal indicating an image recognition match in the representation of the surgical site. For example, when a feature in the representation is determined by image recognition to be a specific feature of, for example, a kidney, it is useful for such an image recognition match to occur and trigger an enhancement of the representation accordingly.
[0148] The enhanced criteria may include determining that there is an error signal indicating an error associated with the surgical robotic system. The enhanced criteria may include determining that a specific time has been reached, or a specific time has passed. More generally, it may be determined, for example by the processor 521, based on the time signal whether the enhanced criteria are met. For example, the enhanced criteria may include a specific action occurring at a specific time or within a specific time range. This may include making a stitch or a series of stitches within a given time period.
[0149] The time signal may include a signal indicating the time of day. The time signal may include a signal indicating the time that has elapsed since the start of a procedure and / or a predefined point in the procedure. The predefined point may be, for example, the start or end of a cutting procedure, the start or end of a suturing procedure, and / or the start or end of an electrocautery procedure. The time signal may include an indication of the duration of a procedure (e.g., one or more of a cutting, suturing, and electrocautery procedure).
[0150] The enhancement criteria may include determining that there is a change of user of the surgical robotic system. The signal indicating the change of user may instruct the surgeon to control the input device to pause the procedure (e.g., by clutching out the input device to disengage the input device from the active controls of the end effector) and restart the procedure by another surgeon. The signal indicating the change of user may instruct a surgeon at one console to take over from a surgeon at another console. This may occur during the surgical procedure or at an interruption in the surgical procedure.
[0151] The presence of a user change can be determined based on a signal such as a user change signal. The user of the system can be determined to have changed based on a login or registration associated with the user of the system. The user of the system can be determined to have changed based on an identification signal associated with the user. The identification signal can be output from an imaging device or a visual processor, which can be configured to perform facial recognition to, for example, identify a user from a group of users, and / or perform pattern recognition on, for example, a 2D code (e.g., a QR code). The identification signal can be output from a wireless receiving device configured to detect a wireless signal. The wireless receiving device can be configured to detect WiFi (TM) signals, Bluetooth (TM) signals, and / or radio frequency signals, such as RFID signals. A device that transmits at least one of these types of signals carried by a user can be used to distinguish between users, and optionally to identify a specific user.
[0152] "Offline" enhanced representation
[0153] As described in the examples above, enhancements can be added during the procedure. Enhancements can be added additionally or alternatively before or after the procedure. Enhancements can be added before the procedure begins, such as in the planning phase. Before the procedure begins, there will be no "live" image feed from an imaging device at the surgical site. Therefore, preferably, enhancements are added to a model (e.g., a 3D model) of the site before the procedure begins. Such a 3D model can be generated in one of several ways. For example, a 3D model can be derived from a scan such as an MRI scan. The 3D model can be derived from a stereo mode, which can be selected based on one or more patient-related parameters. The 3D model can be derived from more than one stereo mode. For example, a 3D model can be derived from a weighted combination of different stereo modes.
[0154] The controller is suitably configured to be able to navigate through the model in order to visualize the intended representation during the procedure. This may be done, for example, by the input device and / or the second input device.
[0155] In the planning stage, it is useful to enhance the representation of the surgical site so that possible regions of interest and / or the location of the expected surgical intervention can be identified. In such a planning stage, enhancements can be added by a surgeon or other physician. For example, enhancements can be added by a trainee surgeon or nurse. Such enhancements added in the planning stage do not have to be so accurate. It is sufficient to indicate a general area or feature. Such enhancements can, for example, indicate the approximate position in the entire surgical site by identifying key features (e.g., blood vessels, organs, tumors, safety / cut edges and / or bone structures) and / or the direction of key features. Such indications can reduce the time required by the surgeon during the procedure. For example, by helping the surgeon to locate himself in the surgical site, enhancements can save the surgeon's time and / or energy. This can help reduce the total time required for surgery, which can have advantages for both patients and hospitals, as discussed elsewhere in this article.
[0156] During a procedure, enhancements can be added to highlight areas of interest. These can include points that the surgeon may wish to return to during a procedure. This may be because the surgeon has discovered something unexpected that requires more detailed examination. Such enhancements can indicate a single point of interest (e.g., an organ) or multiple points of interest (e.g., multiple organs or multiple points on the same organ).
[0157] The surgeon may add enhancements to indicate areas of higher risk or danger. For example, it may be desirable to highlight the location of blood vessels (e.g., arteries) with the aid of enhancements. This may help the surgeon avoid blood vessels, thus reducing the risk of causing unexpected bleeding during the procedure.
[0158] Enhancements can be added to indicate waypoints in a procedure. Such waypoints can be used to guide a user (e.g., an intern or less experienced surgeon). This method can enable a user to more quickly redraw a traversed path, which in turn allows a reduction in the time required to complete a procedure. Waypoints can be used to guide an intern or less experienced surgeon during a live procedure or during a simulated live procedure.
[0159] Augmentations can be added at locations of suturing, incisions, electrocautery operations, and / or grasping points of tissue. Typically, augmentations can be added at points where the end effector is or becomes operable. Augmenting the representation of the surgical site in this way allows tracking of active sites. The 3D locations of these active sites can be determined based on the augmentations. This can allow for later analysis of a procedure or specific activities during a procedure.
[0160] Enhancements can be added after the procedure is completed. Such enhancements can be added on the recorded feed of the procedure and / or on a model constructed based on data obtained from or during the procedure. Enhancements can indicate possible areas that need improvement. Adding such enhancements after the procedure is completed means that the review of the procedure can be performed in an environment that is less stressful than during the procedure itself. Therefore, a higher level of analysis can be performed compared to the analysis that may be performed during the procedure. In some examples, enhancements can indicate the optimal location and / or spacing of the suture site. Such enhancements can increase the user's awareness of potential problems that may occur in subsequent procedures that are the same or similar to the procedure being reviewed. Increasing awareness in this way can reduce the number of undesirable accidents in subsequent procedures, which can, for example, improve the efficiency of these subsequent procedures and / or can reduce the number of complications during the procedure.
[0161] Augmentation added to the representation of the surgical site can be used in several different ways. One example of a way in which augmentation can be used is to help orient the surgeon during a procedure. Augmentation allows indications to be added to the representation of where the surgeon is looking in the site. For example, by labeling an organ, such as a kidney, the surgeon will have a better understanding of the site and can therefore more easily control the end effector and / or move around the site. This will be discussed in more detail below.
[0162] Grouping enhancements
[0163] Enhancements can be added to one or more groups of enhancements. Multiple enhancements can be added to a particular group of enhancements. Enhancements can be grouped based on features common to those enhancements. For example, enhancements can be grouped based on one or more of the following:
[0164] · Add enhanced points to the system's users,
[0165] Add enhanced users,
[0166] The program being executed,
[0167] The type of program being executed,
[0168] the type of feature being enhanced (e.g., organ, vessel, tissue and / or bone, damaged and / or diseased area),
[0169] the feature being enhanced (e.g., a specific organ or blood vessel),
[0170] points where action needs to be taken (i.e. a "to be done" list, which may include entry points),
[0171] Time (e.g., all enhancements added in the last hour or last 30 minutes),
[0172] etc.
[0173] Enhancements in different groups may be distinguished on the display. For example, the system (e.g., processor 521) may be configured to highlight enhancements in different groups in different ways. Enhancements may be highlighted in one or more of the following ways: different colors, labels with different fonts and / or different sizes, different outlines, and flashing or flashing at different frequencies.
[0174] The system is suitably configured to display or hide enhancements depending on the group of enhancements that the particular enhancement belongs to. This increases the ease with which a user can identify enhancements of a particular type, and therefore can take action based on those enhancements.
[0175] The system (e.g., processor 521) can be configured to perform calculations based on one or more enhancements. Where appropriate, the calculations can be performed automatically. For example, the system can automatically count the number of enhancements added or the number of enhancements in a particular group of enhancements. For example, a group of enhancements may involve tendrils extending along a feature, such as a stomach. The surgeon may need to move around the site to see all of the tendrils, as some tendrils may be located on opposite sides of the stomach from each other. The surgeon need not add enhancements for all of the tendrils at once. Enhancements may be added at different stages of a procedure, and indeed even in more than one procedure. In such a case, it may be difficult to remember the number of tendrils correctly. More than one user may have added enhancements. Therefore, it is useful if the system provides a count of the number of enhancements in, for example, a "tendril" group of enhancements.
[0176] The system can be configured to determine the distance between multiple enhancements or multiple enhancements in a group of enhancements. The distance between multiple enhancements can include the maximum distance between enhancements, such as the distance between two enhancements that are farthest apart from each other (in the 2D or 3D space of the representation of the surgical site). The distance between multiple enhancements can include the minimum distance between enhancements, such as the distance between two enhancements that are closest to each other (in the 2D or 3D space of the representation of the surgical site). The distance between multiple enhancements can include the average value (e.g., one or more of the mean, mode, or median) of the distance between multiple enhancements or multiple enhanced subsets. The distance between multiple enhancements can include the distance between subsequent enhancements (i.e., for 3 enhancements, the total distance can be the sum of the distance between the first enhancement and the second enhancement and the distance between the second enhancement and the third enhancement). This method allows the user to add enhancements along the length of the feature, and allows the system to then determine the length of the feature.
[0177] The system can be configured to determine the orientation of the feature. For example, the system can be configured to determine the orientation of the feature based on one or more lines connecting two or more enhancements associated with the feature. The orientation can be determined relative to a convenient reference frame, such as an operating table, a body cavity, another identified or selected feature at the surgical site, etc.
[0178] The system can be configured to indicate a line between two or more enhancements or between multiple enhancements in a group of enhancements. The line can be visible on the display and / or on another display. Indicating lines in this way may be useful for telestration. For example, during a program or during training before performing a program, providing telestration capabilities may be useful for teaching and / or guidance. The system can be configured to indicate lines based on one or more of a point-to-point line connecting all selected enhancements, a best fit line, and a spline based on enhancements. The system can be configured to indicate lines in 2D or 3D.
[0179] The system can be configured to determine an area surrounded by multiple enhancements or multiple enhancements in a group of enhancements. The area surrounded by multiple enhancements can include the area of a polygon with an enhancement at each corner of the polygon. The area surrounded by multiple enhancements can include the area surrounded by multiple enhancements projected onto a feature in a 3D model. For example, multiple enhancements can be located on a curved feature such as an organ. The area surrounded by multiple enhancements can include the surface area of the curved feature onto which the enhancements are projected. This method enables determination of the surface of a feature within a representation of a surgical site, which surface may be more useful than the area where the feature is projected onto a 2D display.
[0180] The system can be configured to determine a volume surrounded by multiple enhancements or multiple enhancements in a group of enhancements. The volume surrounded by multiple enhancements can include a volume surrounded by the location of the enhancements in 3D space, such as in a 3D model of the surgical site. Determining the volume surrounded by multiple enhancements can allow the volume of a feature at the surgical site to be determined. For example, enhancements can be added across the surface of a tumor. The system is suitably configured to determine the size or approximate size of a tumor based on the location of the enhancements associated with the tumor in 3D space.
[0181] The system (e.g., processor 521) can be configured to restrict the end effector from entering a specified point, line, area, or volume, or restrict the end effector from moving through or near it. The specified point, line, area, or volume can be defined by one or more enhancements in one or more groups of enhancements. For example, enhancements can be added on or along a feature such as an artery (this can be done in the planning stage). Enhancements related to arteries can be in the "artery" group enhancements. More enhancements can be added for veins (or other features). Enhancements related to veins can be in the "vein" group enhancements. The system can be configured to restrict the end effector from entering a volume around one or more selected features. For example, the system can restrict the end effector from entering a volume around one or more selected features of the "artery" and "vein" group enhancements (or the "vessel" group enhancements that include the "artery" and "vein" group enhancements). This can reduce the risk of the surgeon accidentally damaging a feature (here, an artery or vein) during surgery. The system can be configured to restrict or prevent the end effector from moving within a preset distance of such a point, line, area, or volume. The system can be configured to provide feedback (e.g., visual and / or tactile feedback, such as an audible sound or vibration (e.g., of an input device)) based on the proximity of a portion of the end effector to a constrained point, line, area, or volume (or generally, to any specified enhancement or set of enhancements). This approach can allow the end effector to move faster than it would otherwise while reducing the likelihood of tissue damage. Thus, this approach can help speed up procedures, which can improve patient recovery time.
[0182] Reference now Figure 6 , a method for enhancing a representation of a surgical site is shown. Optionally, at least a portion of a surgical site may be imaged using an imaging device 602. Optionally, a model (e.g., a 2D or 3D model) of the surgical site may be obtained 604. A representation of the surgical site is obtained at 606. The representation of the surgical site may be obtained based on one or both of the image of the surgical site captured at 602 and the model of the surgical site obtained at 604. At 608, the representation is displayed. For example, the representation may be displayed on a display. An indicator on the display is controlled 610. The indicator is appropriately controlled by an input device at a controller. At 612, an enhancement signal is received. The enhancement signal may be output by the controller, such as by the input device. In response to receiving the enhancement signal, the representation of the surgical site is enhanced 614. The representation of the surgical site may be enhanced based on the position of the indicator on the display.
[0183] A model of the surgical site may be obtained from a stored representation of the site 604. The end effector visible on the display may include an indicator. The method may include controlling the indicator by controlling the end effector.
[0184] Orientation Enhancement
[0185] As mentioned, one or more enhancements may be provided on or as part of a representation to help orient the representation. Such navigation aids may enhance understanding of the displayed representation and improve the speed and / or accuracy with which a user of the robotic system may perform a procedure using the robotic system.
[0186] In some examples, the processor 521 is configured to receive an imaging device signal. The imaging device may be configured to output an imaging device signal. The imaging device signal indicates at least one of the position and orientation of the imaging device, for example relative to the surgical site or some other suitable reference system (e.g., a patient table (which may be movable) or an operating room). This enables the processor to determine the position and orientation of a portion of the surgical site that can be viewed using the imaging device. That is, the processor can determine the portion of the surgical site to which the image feed of the imaging device relates and which can be displayed. As discussed above, the representation of the surgical site can be based on the image feed and / or the 3D model. Therefore, the imaging device signal enables the processor to determine the portion of the surgical site viewed by the imaging device. The processor may be configured to match the imaged portion of the surgical site with the 3D model. In some examples, the imaged portion of the surgical site may be the same or closely approximated with the 3D model. This may be the case when the organs and blood vessels in the image are in the same or approximately the same relative position as in the 3D model. However, the organs and blood vessels do not necessarily move together with each other in the same manner. For example, when the patient tilts, different organs and / or blood vessels (or indeed any other internal structure) may behave differently. Different behaviors may be due at least in part to the relative "softness" (or elasticity) of the tissue. Suitably, the 3D model and / or processor may access data or assumptions about the softness of the tissue. When there is a change that may cause the tissue to move, such as a change in the patient's angle, the new position of the tissue (e.g., organs and blood vessels) can be estimated based on the softness value (or relative softness value) associated with each tissue. An updated 3D model can be generated based on the estimated position. In this way, the processor can be configured to match the imaged portion of the surgical site with the 3D model and / or the updated 3D model. This method facilitates the generation of representations based on both image feeds and 3D models, because it can enable the imaged view to be correctly aligned with the 3D model. Representations can be enhanced based on imaging device signals. The processor can output a display signal to cause at least a portion of the enhanced representation to be displayed. The system may include a display configured to display at least a portion of the enhanced representation.
[0187] During the procedure, the imaging device will typically be moved. That is, at least one of the position and orientation of the imaging device will change. The scaling of the imaging device may change. Suitably, the processor is configured to receive another imaging device signal. The other imaging device signal suitably indicates an updated position and / or orientation of the imaging device, and / or indicates an updated scaling or translation state of the imaging device. The processor is suitably configured to determine a change in at least one of the position of the imaging device, the orientation of the imaging device, the scaling of the imaging device, and the translation state of the imaging device based on the imaging device signal and the other imaging device signal. The processor may be configured to update the enhanced representation based on the determined change. The processor may be configured to update the enhanced representation by updating the enhancement applied on or to the representation.
[0188] The representation can be enhanced based on features of the representation, such as an organ, a vessel, or more generally one or more anatomical features, and / or the location of one or more of a stitch, an incision, an electrocautery action, etc. The enhancement can indicate the orientation of the representation of the surgical site. The representation can be enhanced based on another enhancement, such as an enhancement added earlier. Other enhancements can be based on one or more anatomical features. For example, the system can be configured to detect the presence of anatomical features in the representation. This detection can be performed by image processing such as image matching. In an illustrative example, the system can recognize that there are kidneys in the representation (whether or not it can be viewed on a display—for example, the portion of the representation being viewed may not include the portion of the representation that includes the kidney). The kidney can be labeled in this way, for example, according to one or more techniques described herein. Such a label can be added as an enhancement (again, whether or not it can be viewed on a display). The representation can be further enhanced based on this enhancement (i.e., based on the "kidney" label in this example). In another example, the representation can be enhanced based on determining that the kidney is present in the representation, without first enhancing the representation for the kidney.
[0189] The enhancement indicating the orientation of the representation can take one of several forms. The enhancement can indicate the orientation by indicating the "overall" orientation of the representation, such as by indicating one or more of an upward direction (e.g., against gravity or perpendicular to the patient table) and a direction toward a portion of the surgical site as a whole or a portion of the body (e.g., toward a trocar insertion site or toward the patient's head). The enhancement can indicate the orientation by indicating the "overall" orientation of the representation, such as by indicating the position of an anatomical feature (e.g., an anatomical feature that can move relative to the surgical site as a whole). For example, an organ (or other anatomical feature) can move during a procedure. An organ can move due to gravity when the patient moves. An organ can move due to breathing. An organ can be moved by a surgeon. In some examples, the enhancement can point to or otherwise indicate a feature that is "off-screen," i.e., a feature that is not part of the displayed representation.
[0190] Enhancements (e.g., to indicate "overall" orientation) may include artificial horizon lines. An example of this is Fig.10 . For example, a line or plane 1002 (e.g., on a 3D enhancement) may indicate a horizontal line or plane. The enhancement may include an indication 1004 of a direction and / or distance toward a portion of the surgical site (e.g., in the above example, the trocar insertion point) or more generally, toward a feature of interest. For example, an arrow 1004 pointing to the feature of interest may be provided. The arrow may be provided in 2D or 3D. In some examples, both a line or plane and an arrow may be provided. For example, the enhancement may include a sphere 1000, a plane 1002 may be rotated within the sphere to indicate a gravity horizontal plane, and one or more arrows 1004 (e.g., radial arrows extending between the center of the sphere and the outer surface of the sphere) may be provided within the sphere to indicate the direction toward the feature of interest. The sphere may be shown in any convenient manner. The sphere may be transparent. The sphere may be indicated by one or more longitude and / or latitude lines.
[0191] The processor may be configured to determine at least one of a direction and a distance to a feature of interest. The distance to the feature of interest may be determined based on determining a boundary of the screen projection onto the representation and / or a 3D model on which the representation is based. The distance to the feature of interest may be determined based on an anatomical model (e.g., a 3D model). The distance to the feature of interest may be determined based on the position of one or more enhancements in the model. For example, where an enhancement associated with a given feature (e.g., a blood vessel) has been added, the position of the enhancement may be considered to indicate the position of the associated feature (e.g., the blood vessel). Therefore, the distance from some other point (e.g., a boundary of the screen projection onto the representation) to the position of such an enhancement may be considered to be the distance to the associated feature.
[0192] The enhancement may include an indication of the distance toward the feature of interest. The distance may be measured from a point on the edge of the screen (i.e., a point at the edge of the displayed portion of the representation), from the center of the screen (or the displayed portion of the representation), or from any other point as desired. The distance to the nearest edge of the feature of interest, to the center point of the feature of interest, to a point midway between the nearest edge and the farthest edge of the feature of interest, or to any other point as desired may be measured. Preferably, the distance is measured as the distance from the point at the edge of the displayed portion of the representation closest to the feature of interest to the edge of the displayed portion of the representation where the feature of interest is closest to. That is, the distance may be determined as the shortest distance between the displayed portion of the representation and the feature of interest. The portion of the feature of interest closest to the displayed portion of the representation may change with changes in the position, orientation, scaling, and / or translation of the imaging device. It may be determined that the processor automatically updates the distance determination accordingly. The enhancement may include a text label providing, for example, the distance to the feature of interest in millimeters. Any other units may be used. The unit of measurement and / or the precision of the measurement may be selectable, for example, user selectable. The units of measurement to be used and the precision with which the measurements are displayed can be preset in the user profile.
[0193] Thus, in the event that a feature of interest is not viewable in the displayed portion of the representation, the augmentation may provide an indication of where the feature of interest is located. This indication may include the direction and distance to the feature of interest. This may enable a user, such as a surgeon, to more quickly get their bearings when presented with the representation, and / or reduce the risk of them losing their bearings when navigating through a surgical site (where such navigation may include translational and / or rotational motion within the 3D site).
[0194] When a feature is not visible on a display showing a portion of a representation, the feature may be considered not to be present in the displayed portion of the representation. When a feature is occluded by another feature in the representation and / or is outside the field of view of the displayed representation, the feature may be considered not to be present in the displayed portion of the representation.
[0195] More than one orientation enhancement may be provided. Each such enhancement may point to or otherwise indicate a corresponding feature of interest. More than one such enhancement may point to or otherwise indicate the same feature of interest. For example, where two such enhancements are provided (one at each of the upper right and upper left corners of the display), the enhancements together may provide information in addition to that provided by a single enhancement. For example, by comparing the relative distance between each enhancement and the feature of interest, it may be readily determined whether the feature of interest (relative to the display) is centrally located or located to one side or the other. Thus, such combinations of enhancements may be used to obtain a more accurate orientation of a representation.
[0196] The enhancement may include an indication of the distance toward the feature, such as the distance between the edge of the screen and the feature (e.g., the portion of the feature closest to the edge of the screen). The indication of the distance toward the feature may be provided in any convenient manner: flicker, flicker frequency, color, size, shape, outline, transparency, translucency, and / or any one or more of these and / or other visual characteristics of the enhancement. The enhancement may be selected or displayed based on the determined distance. Figure 7 Some examples of enhancements that can indicate the distance to a feature of interest are shown in FIG.
[0197] Figure 7 a shows an arrow 701 which may point to a feature of interest. Figure 7 The arrows in a have shafts of width w1. Figure 7 b shows another arrow 702 that may point to the feature of interest. Figure 7 The arrows in b have shafts with a width of w2. w1 is greater than w2, which can indicate that the distance from arrow 701 to the feature of interest is greater than the distance from arrow 702 to the feature of interest. In some examples, a relatively large arrow width can indicate a relatively short distance to the feature of interest.
[0198] By comparison Figure 7 b and the arrows in 7c to see another feature that is enhanced that can instead indicate the distance to the feature of interest. Figure 7 The arrow 702 in b has an arrow head with a width of w3. Figure 7 Arrow 703 in c has an arrow head with a width of w4. w3 is greater than w4, which can indicate that the distance from arrow 702 to the feature of interest is greater than the distance from arrow 703 to the feature of interest. In some examples, a relatively large arrow head width can indicate a relatively short distance to the feature of interest.
[0199] The width of the arrow shaft and / or the width of the arrow head can be based on the distance to the feature of interest. For example, the width of the arrow shaft and / or the width of the arrow head can be proportional or inversely proportional to the distance to the feature of interest. In some examples, the width of the arrow shaft and / or the width of the arrow head can be proportional to the distance to the feature of interest linearly, inversely linearly, logarithmically, or inversely logarithmically.
[0200] A decrease in the width of the arrow shaft and / or the width of the arrow head may indicate that the feature of interest is getting closer to the edge or border of the display. A "zero" width of the shaft and / or arrow head may indicate that the feature of interest is at the border. When the feature of interest appears on the display, the enhancement (here, the arrow) may be removed from the display. In some examples, an enhancement or marker indicating the feature of interest may replace the arrow (or other "off-screen" enhancement) when the feature of interest appears on the screen. In some examples, when the feature of interest moves off the screen, the marker associated with the feature may be replaced by an "off-screen" enhancement indicating the direction and / or distance to the feature.
[0201] Suitably, the enhancement indicating the distance and / or direction of the off-screen feature of interest is dynamically updated, for example in response to movement of the imaging device and / or patient tissue.
[0202] Other aspects of the size of the enhancement may instead indicate distance. For example, where the determined distance is greater, the size of the enhancement as a whole may increase. In other examples, where the determined distance is greater, the size of the enhancement may decrease. The processor may be configured to modify the size of the enhancement as a function of distance. The processor may be configured to modify the size of the enhancement within one or both of a minimum size and a maximum size. The minimum size and / or the maximum size may be selectable, such as user selectable.
[0203] In some examples, the processor is configured to remove the enhancement when it is determined that the feature of interest is visible on the screen, such as when the distance from the edge of the screen to the feature of interest becomes zero. In some examples, the processor is configured to remove the enhancement when the feature of interest extends a certain distance within the screen boundary. Taking this approach may mean that before removing the enhancement, the feature of interest is visible and potentially recognizable by the user. The distance inside the screen boundary at which the processor removes the enhancement of the feature of interest can be preset and / or selectable, such as user selectable. The distance inside the screen boundary at which the processor removes the enhancement of the feature of interest can be based on the nature, type and / or classification of the feature of interest (for example, when the feature cannot be immediately recognized, it may be desirable to ensure that more of the feature is visible on the screen before removing the enhancement), and / or the program being executed (for example, different actions can be selected for different features, for example, depending on how critical each feature is to the particular program). In some examples, data from at least one previous program (optionally including data related to one or more results from the program) can be used to determine the feature of interest, when to enhance the representation of the determined feature of interest, and / or when to remove the enhancement of the determined feature of interest. For example, where such data indicates that it is unclear what a particular feature is (e.g., a given feature is mistaken for another feature), that feature may be selected for enhancement and / or enhancement for that feature may be maintained longer than enhancement for another feature.
[0204] Figure 7 d and 7e show another visual attribute of an enhancement that can change with distance to a feature of interest. In the example shown, the enhancement is an arrow. Other shapes can be provided as enhancements or parts of enhancements. Figure 7 d and 7e schematically illustrate enhancements that flash at different frequencies. For example, the processor can cause the enhancement to flash at a frequency or relative frequency that indicates the distance to the feature of interest. For example, a relatively high flashing frequency can indicate a relatively closer (or farther in other examples) feature of interest.
[0205] The enhancement may include a text label. For example, where the enhancement includes an arrow, the text label may be provided along the length of the arrow, overlapping the arrow, adjacent to the arrow, etc. Figure 7 7f and 7g show examples of arrows with text labels showing the distance to the feature of interest. Where the enhancement includes other shapes, text labels may be similarly provided. The text label may include a description of the feature, such as "kidney".
[0206] Where the processor is configured to remove an enhancement indicating a direction and / or distance to a feature, the processor may be configured to retain a label, such as a descriptive label, about the feature. For example, in the case where a kidney is not on the screen, an enhancement may be applied to a representation indicating the distance and direction to the kidney. Once the kidney appears on the screen, the enhancement may include only the label "kidney" with the remainder of the enhancement removed.
[0207] Suitably, the processor is configured to determine the enhancement in dependence on a feature not present in the displayed portion of the representation. For example, the enhancement is determined in dependence on a feature that is off-screen.
[0208] In some examples, the processor is configured to determine the enhancement based on the enhancement data. The processor may be configured to receive the enhancement data. The enhancement data may include data associated with the representation. Such data associated with the representation may be stored with the representation or as part of the representation. Such data may include features of markings (or annotations). Such data may be based on or include data generated during the planning stage and / or one or more previous procedures. For example, the data associated with the representation may include data obtained based on one or more previously performed procedures, and the one or more previously performed procedures are of the same or similar type as the procedure being performed or planned to be performed. In some examples, the data associated with the representation may include data obtained based on one or more previously performed procedures performed by the same surgeon or team.
[0209] In some examples, the enhancement data includes data indicating features in the representation. For example, the enhancement data may include feature labels and / or data related to edges of one or more of a blood vessel and an organ.
[0210] In some examples, the data indicating features in the representation indicate one or more feature groups in a set of feature groups. The feature groups may relate to one or more general organs, blood vessels, sites of surgical procedures to be performed / performed, sites of interest, and sites or features that depend on the procedures to be performed / performed. This grouping of features may enable the orientation of the representation to be based on features that are relevant to the surgeon at / at this time in the procedure. This approach may enable faster orientation determination—helping surgeons perform procedures more quickly and efficiently, i.e., facilitating better human-computer interaction.
[0211] In some examples, the received imaging device signal and / or the received additional imaging device signal include kinematic data related to the imaging device. The kinematic data may come from a kinematic controller, such as a kinematic controller at the command processor unit 520. The kinematic controller may be part of the processor 521, or it may be provided separately. Using kinematic data in this way enables the use of existing information within the system; kinematic data does not need to be calculated separately for enhanced representation. This may, for example, improve the efficiency of the system in terms of at least one of speed, processing power, and data bandwidth. Kinematic data is highly accurate because kinematic data enables the surgical robot to move. Therefore, using this same kinematic data enables the orientation of the representation of the surgical site to be determined quickly and accurately. This may improve human-machine interaction, thereby enabling users of the system (e.g., surgeons) to quickly orient themselves during surgical procedures. This enables surgeons to perform surgery more quickly and / or more accurately. This facilitates improvements in surgical procedures, such as shortening procedures (which is beneficial to patients and hospitals).
[0212] The processor may be configured to track the position of a feature of interest, such as an anatomical feature. The processor may be configured to track the position relative to the displayed portion of the representation. The processor may be configured to update the enhanced representation based on a change in the tracked position and at least one of the imaging device orientation, imaging device position, field of view of the imaging device, and field of view of the displayed portion of the representation. In some examples, the position may be tracked by image processing of an image feed generated by the imaging device. In some examples, the position may be tracked by processing the 3D coordinates of the position in a 3D model and tracking how the model moves relative to the imaging device. Combinations of these methods and other methods are possible.
[0213] Data sets can be established from one or more programs, and such data sets can be used to improve orientation determination. In some examples, the processor can be configured to determine the orientation based on features within the representation of the surgical site (e.g., features in the image feed generated by the imaging device). The processor can be configured to determine the orientation based on more than one feature in the representation. Establishing a data set through many different programs (which can be programs of the same type) enables more accurate determination of the orientation based on a greater number of reference features and reference combinations of features. For example, in the image feed, one or more features in the representation of the surgical site can be compared with one or more features in the data set. For example, the orientation can be predicted or estimated based on one or both of the position and orientation of the representation and / or imaging device. Elements of the data set corresponding to the orientation within the expected range around the predicted orientation can be selected to be compared with one or more features in the image feed. The elements of the data set that best match one or more features in the image feed can be used to determine the orientation. For example, the orientation can be determined as the previously determined orientation about the best match. In some examples, the predicted or estimated orientation can be combined with the kinematic data of the imaging device to improve accuracy.
[0214] In some examples, the processor may be configured to determine (or identify) features in the representation based on the position and / or orientation of the imaging device. For example, the processor may determine that there are features in the generated image feed. The processor may determine the portion of the representation being imaged by the imaging device from the position and / or orientation of the imaging device. The processor may determine what the feature is, for example, based on an anatomical model, and may enhance the representation accordingly. The processor may determine what the feature is based on a data set including data obtained from one or more previous programs. The data set may include one or more results from at least one previous program. For example, where the feature has been previously identified and / or at least satisfactory results have been achieved, the identification may be used by the processor to identify the feature in the current representation. This method enables the use of previously identified features, so that the results of the program in which those features are identified can be considered. It can be expected that better results will be positively correlated with more accurate feature identification. Therefore, focusing on features identified when the results are above a threshold level (e.g., a qualitative level) can be expected to improve the accuracy of later feature identification, and therefore improve the accuracy of enhancement.
[0215] At least some anatomical features will typically move within the surgical site. For example, an organ may move due to one or more of gravity, the patient breathing or being artificially ventilated, being moved by a surgeon, etc. Therefore, the position of the organ in the surgical site will not remain constant. The position can change in a predictable manner. For example, an organ (or more generally, a feature of interest) may be in a first position at a first time and may be in a second position at a second time. The processor is suitably configured to determine whether the current time is the first time or the second time. The processor can be configured to identify the feature based on the determination of whether the current time is the first time or the second time and / or the determined position of the feature.
[0216] Conversely, in some examples, the processor is configured to determine a position and / or orientation based on the identified features in the representation. For example, when a feature such as an organ can move under the effect of gravity, if the processor determines that the feature (e.g., the organ) is in a particular position (e.g., a position to which the feature would typically move when gravity acts in a particular direction relative to the surgical site), the orientation (e.g., the direction of gravity) can be determined based on the position.
[0217] The user can mark a feature, such as an organ, move the patient in a known manner, and re-mark the same feature. This can give an indication of the elasticity of the feature, which can allow the position of the feature to be estimated when the patient undergoes other movements.
[0218] As discussed above, features can be tagged at various times before, during, and / or after a procedure. Features can be tagged at different locations relative to the surgical site. Tagging features in this manner can help improve the accuracy of later determinations based on such tags. This approach enables the creation of a data set of a range of possible locations where a given feature may exist. It can be found that the location of a feature may depend on one or more of the procedure being performed, the stage of the procedure being performed, the time since the start of the procedure, the action being taken, etc. Tagging features multiple times enables better characterization of the movement of features, which can enable more accurate determinations based on those characterizations. This approach can also help improve predictor efficiency, for example by narrowing the range of possible locations of a feature, or by narrowing the list of potential features at a given location.
[0219] In some examples, the processor is configured to automatically center the field of view of the displayed representation at a feature. For example, for a given program or a given portion of a particular program, the feature may be predetermined. The feature may be selectable, such as user selectable. The initiation of the automatic centering (or re-centering) function may depend on an automatic centering signal. The automatic centering signal may be generated by a controller, such as an input device at the controller. In response to receiving the automatic centering signal, the processor may be configured to automatically re-center the displayed portion of the representation so that the displayed representation is centered on the identified feature. The processor may be configured to re-center the displayed portion of the representation without moving the imaging device. In other words, the imaging device does not need to move to center the field of view of the displayed representation on a specific feature. For example, the processor may be configured to change the scaling of the displayed representation so that the rescaled representation can be properly centered without moving the imaging device. This method facilitates digital re-centering of features, which may be faster than re-centering by moving the imaging device. Furthermore, it enables to subsequently return to the original viewpoint after re-centering, in this way avoiding the introduction of position and / or orientation errors when returning to the original viewpoint.
[0220] This type of re-centering Figure 8 Shown in. Figure 8 802. The displayed portion of the representation includes (in this highly simplified and schematic example) features such as an organ 804 and a blood vessel 806. The junction between the organ 804 and the blood vessel 806 is toward the right of the displayed portion of the representation 802. This junction may be designated as a feature of interest, and a re-centering signal may be generated for this junction. In response, the processor may scale the representation so that the scaled representation is centered on this feature of interest. The scaled portion of the representation is at Figure 8 In a, it is shown as a dotted line at 808. Figure 8 b shows a re-centered representation, which again includes the organ 804 and the vessel 806, but now the junction between the organ 804 and the vessel 806 is located in the middle of the display 802. Such centering of the feature of interest may enable a more convenient interaction between the user of the system and the system itself. This may be because centering of the feature of interest allows the feature of interest to be more clearly seen in the displayed portion of the representation.
[0221] exist Fig. 9, a process for re-centering a representation is shown in . A feature of interest (901) is identified on which the displayed portion of the representation can be centered. The identification of the feature of interest can be performed using one or a combination of the methods described herein. In some examples, the feature of interest can be manually identified and / or annotated. In some examples, the feature of interest can be automatically identified and / or annotated. An automatic centering signal (902) can be generated. The generation of the automatic centering signal can be automatic, for example, in response to satisfying one or more criteria. The generation of the automatic centering signal can be manual, for example, by a user interacting with an input device at a controller. Thus, the initiation of the automatic centering function can be performed in response to input from a user and / or in response to (for example) a predetermined portion of a procedure being reached, or some other standard or combination of standards. The standard appropriately depends on user preferences and / or the procedure being performed. For example, the standard can depend on a user profile. Thus, the system can be configured so that when a surgeon reaches a predetermined point in a procedure where, for example, a clearer view of the feature of interest is desired (for example, when an incision is to be performed or a stitch is to be inserted), an automatic centering signal can be generated.
[0222] In response to the auto-centering signal, the processor may be configured to change the scaling of the displayed representation (903). The change in scaling may depend on the identified feature. Thus, the scaling may be changed in a manner that allows the identified feature to be centered in the scaled representation. The representation may then be re-centered (904). Suitably, the representation is re-centered based on the identified feature. The re-centered representation may be displayed (905). For example, the processor may be configured to output a display signal to cause the representation to be displayed.
[0223] It is also useful to re-center an instrument end effector (or a center point between multiple instrument end effectors). One or more instrument end effectors can be moved into a volume around a feature on the representation, such as a marker. The representation can then be re-centered around the feature (e.g., by adjusting the zoom of the representation).
[0224] In some examples, the representation can be reoriented on the display. Figure 8 b and Figure 8 An illustration of this situation can be seen in c. Figure 8 b shows an organ 804 and a blood vessel 806. As shown on the display 802, the right edge of the organ 804 is substantially vertical. The blood vessel 806 extends upward to the right. Figure 8 c also shows an organ 804 and a blood vessel 806, but Figure 8 In c, the orientation of the representation has been changed so that the vessel extends approximately horizontally. Figure 8 The representation shown in c has been relative to Figure 8b is rotated clockwise. This change in the orientation of the representation on the display can be used to achieve a consistent orientation of a given feature (e.g., blood vessel 806). This can provide a user, such as a surgeon, with a consistent view of a given feature between several different procedures, which can enhance the repeatability and / or accuracy of the procedure performed by the surgeon. For example, in the case where a surgeon is adding a stitch, the surgeon may prefer to view the feature being sutured in a horizontal orientation.
[0225] The particular orientation to which the representation may be rotated may be selected based on user input, a user profile, a preset orientation, etc. Selecting the orientation based on the user profile may enable each surgeon to view the representation in the orientation that best suits that surgeon.
[0226] The representation can be reoriented, for example, at the processor by rotating the imaging device and / or by digitally rotating the representation. Reorienting the representation at the processor has the advantage of not requiring movement of the imaging device. When the representation is reoriented at the processor, it is generally appropriate to remap the controller to the surgical instrument so that movement of the controller by the surgeon (e.g., to the left) causes the corresponding surgical instrument to move in the correct direction (in a left direction in the reoriented reference frame of the displayed representation of the surgical site). The controller can be remapped to the surgical instrument based on the displayed representation. Thus, the reorientation of the representation can be taken into account in the remapping.
[0227] Health Indicator
[0228] As introduced above, pixel data, such as data corresponding to a pixel region of video data captured by an endoscope at a surgical site inside a patient, can be analyzed to derive data that can provide an additional source of feedback to a user of the surgical robotic system. Analysis of the endoscopic video can be used to identify or characterize movement (e.g., peristalsis) of anatomical structures and / or tissue perfusion. The analysis suitably allows for the derivation of one or more parameters related to the anatomical structures and / or tissue perfusion. The derived parameters can be used to provide feedback to a user of the system, thereby enhancing the usability of the system.
[0229] Video magnification techniques, such as the Euler video magnification method, take a video sequence as input. Spatial decomposition can be performed, followed by temporal filtering. The resulting signal can be amplified to derive information. The algorithm used to analyze the video data may differ depending on the information derived from the video data. For example, a color change specific algorithm may be used to perform identification of color changes and derive parameters related to the identified color changes. In general, "color changes" may involve changes in the spectrum, which are not necessarily limited to the visible part of the spectrum. A movement specific algorithm may be used to perform identification of movement and derive parameters related to the identified movement. Identification of color changes and identification of movement may be performed together or separately. The video magnification technique may include a tuning stage. The tuning stage may include identifying a specific range of values on which to focus. The tuning stage may include identifying a specific area on which to focus.
[0230] The video data captured by the endoscope can be displayed for viewing by the surgeon or other members of the operating room staff during the procedure. The video data typically shows an image of the surgical site, including the patient's anatomy, and the end effectors of the surgical instruments provided at the surgical site. It is desirable for the surgical robotic system to parameterize at least some aspects of the surgical site.
[0231] Such parameterization of the surgical site, such as one or more anatomical structures at the surgical site, can provide additional information to the surgeon. Spatio-temporal variations in the endoscopic video that may not be visible in the displayed video data can be identified and parameters derived therefrom. Such parameters can be used to enhance or enhance a display, such as a display at a surgeon's console. Enhancement of the display can provide information to the surgeon, thereby enabling a better user experience.
[0232] Thus, the present technology allows a user of the system to enhance understanding of small space-time differences in endoscopic video. For example, small space-time differences can be magnified and the magnified differences can be visualized. The nature of the space-time differences that can be magnified varies depending on the selection of the input filter, and can include subtle movements of structures (e.g., anatomical structures) in space, and subtle changes in pixel color amplitude and / or color or spectrum. Changes in the spectrum are not necessarily visible changes. Changes in the spectrum can include non-visible wavelengths. The term "color" can be considered to cover a range of wavelengths of light that are not necessarily visible wavelengths, but appropriately, the range of wavelengths includes visible wavelengths.
[0233] Examples of features of the surgical site that can be identified by analyzing spatial-temporal variations include:
[0234] Blood flows through and perfuses at least a portion of the patient's anatomy or a specific anatomical structure;
[0235] The presence of blood vessels (e.g., arteries, arterioles, and veins); and
[0236] Movement of anatomical structures, such as peristalsis.
[0237] The above may be used alone or in combination to identify patient anatomy, such as anatomical structures.
[0238] In more detail, analysis of the video data may be used to detect tissue perfusion (and therefore blood flow). This may be done during or after a surgical procedure. The analysis may be performed with respect to at least a portion of the patient's anatomy or a specific anatomical structure (e.g., a specific organ) in the field of view of the endoscope, or another specified area of the field of view. The analysis may be performed for the entire field of view of the endoscope. In general, the analysis may be performed for a region of pixels in the video data (which may represent an anatomical structure or a portion thereof, the field of view or a portion thereof, etc., as will be described in more detail elsewhere). Such analysis may be performed on a region of pixels in the video data to determine:
[0239] whether the anatomy represented by the pixels at the pixel region has been devascularised, for example during a procedure to separate and resect the anatomy;
[0240] whether the anatomy represented by the pixels at the pixel region has sufficient tissue perfusion (and therefore blood flow) to enable assessment of potential damage to the anatomy, such as mishandling of the anatomy during a surgical procedure; and
[0241] Whether the anatomy represented by the pixels at the pixel region has adequate tissue perfusion (and therefore blood flow) during and after a surgical step or procedure in order to judge the success or likely success of the step or procedure, or likely future outcomes following the step or procedure. For example, it is contemplated that it would be useful to be able to assess tissue perfusion during and after rejoining separate intestinal ends during intestinal anastomosis (or more generally, surgical anastomosis), vaginal stump closure, reconstructive plastic surgery, and the like. Assessing the adequacy of tissue perfusion in such circumstances may enable more accurate monitoring of tissue health, and therefore allow detection of situations that may lead to tissue death or other complications, and enable remedial action to be taken before tissue death or other complications occur. An example of a serious complication is failure of an intestinal anastomosis, which may result in leakage of feces into the peritoneal space, which may lead to life-threatening septic shock. Thus, the ability to detect such failures is a potentially life-saving feature.
[0242] Analysis of the video data can be used to detect and identify blood vessels in the field of view, such as arteries, arterioles, and veins. This can enable enhanced representation of the surgical site to identify such detected blood vessels, as described elsewhere herein.
[0243] Analysis of video data can be used to detect movement of anatomical structures that might not otherwise be visible. Examples of anatomical structures whose movement can be detected include:
[0244] The ureter or part thereof; and
[0245] The intestine or part of it.
[0246] The ureter is a small retroperitoneal structure that drains urine produced by the kidneys into the bladder. Healthy ureters show subtle peristalsis. These movements can be used to identify the ureters in endoscopic video data. The movements can be used to distinguish the ureters from surrounding tissue. The ability to detect (and amplify) such small peristalsis from video data permits identification of the ureters during surgical procedures, and thus can help prevent accidental damage to the ureters.
[0247] Peristalsis can be used as a surrogate marker for intestinal health. Detected peristalsis can provide reassurance that the blood supply to the intestine is sufficient and the intestine is not damaged during surgery, which is caused by, for example, misalignment or twisting of the intestine, application of external pressure, insufficient blood supply or improper handling by the surgeon. The detection of peristalsis of the intestine enables (optionally by amplifying the movement) to visualize the movement, and the movement is measured to generate one or more parameters related to the intestine. Intestinal parameters can be used to assess intestinal health during the procedure, at the end of the procedure and / or after the completion of the procedure. Detection of the absence of peristalsis in the intestine can indicate that there is a problem. If this situation is detected during the procedure or at the end of the procedure, potential problems can be investigated before the completion of the procedure and remedial actions can be taken. The measurement of peristalsis, or the absence of peristalsis can be used as a higher risk predictor of postoperative complications. This can provide a measurement that can be used to determine where monitoring resources should be directed (for example, for indicating areas where complications are more likely to occur) and / or for determining postoperative actions such as care paths.
[0248] refer to Fig.12, an indicator system 1200 for a surgical robotic system is shown. The indicator system can be used to indicate the state of at least a portion of a patient's anatomy (e.g., an anatomical structure of the patient). The indicator system includes a receiver 1210 configured to receive video data 1212 of at least a portion of a patient's anatomy (e.g., an anatomical structure) at a surgical site from an endoscope. The indicator system includes a processor 1220 configured to detect space-time changes in a pixel region of the received video data. The processor shown includes a space-time change detector 1222 configured to detect space-time changes. The space-time change detector may include a decomposition module 1223 and a filtering module 1224, which are configured to decompose the video data into component parts and filter the decomposed data. The decomposition and filtering are appropriately dependent on the space-time changes to be detected, for example, expected space-time changes and / or space-time changes typical of a particular anatomical structure. The processor is configured to identify a parameter of the anatomical structure in response to the detected space-time changes. The processor shown includes a parameter identifier 1224 configured to identify the parameter. The processor shown also includes an amplifier 1226 configured to amplify the output of the space-time variation detector and pass the amplified output to the parameter identifier. In all examples, the amplifier need not be provided as a separate module. The processor is configured to generate a health indicator 1228 indicating the identified parameter or a curve indicating the identified parameter and output the generated health indicator. The processor shown includes a health indicator generator 1230 configured to generate the health indicator.
[0249] This method enables an indicator system to provide an indication of the health of the anatomical structure, which may be indicative of the health of the patient. The health indicator may be output to indicate the health of the anatomical structure to a surgeon or another member of the surgical team.
[0250] The spatial-temporal change may involve a change in one or both of the position of the anatomical feature and the color of the anatomical feature. The change in position may represent movement of the anatomical structure relative to the surgical site or movement of the endoscope relative to the surgical site, or both. Suitably, the nature of the movement may be characterized as described elsewhere herein. The movement may be determined as movement of at least a portion of the anatomical structure due to, for example, peristalsis. The processor may then identify a parameter characterizing the movement. The parameter may involve one or more of the amplitude of the movement, the frequency of the movement, the duration of the movement, the anatomical structure (or portion thereof) being moved. The parameter may include a measure of the movement. The measure of the movement may include a rate of movement. The measure of the movement may include the amplitude and / or orientation of the movement.
[0251] The color change may represent tissue perfusion and blood flow. The processor may identify a parameter characterizing the color change. The parameter may relate to one or more of a relative change in color, a rate of color change, a duration of color change, an anatomical structure (or a portion thereof) experiencing the color change, etc. The parameter may relate to a change in one or more color channels of the video data. The parameter may include a measure of blood flow. The measure of blood flow may include a measure of heart rate or pulse. The measure of blood flow may include a volumetric measurement or estimate of blood flow through or through at least a portion of a patient's anatomy (such as an anatomical structure).
[0252] The health indicator may be a binary indicator, such as indicating that the anatomical structure is healthy or unhealthy. The health indicator may take the form of a visual signal or include a visual signal for display on a display. The health indicator may include a visual signal for turning a light on or off. The health indicator may take the form of an audio signal or include an audio signal for driving a speaker. The audio signal may cause the speaker to turn off (or remain off) when the anatomical structure is healthy, and turn on (or remain on) when the anatomical structure is unhealthy. In this way, the audio signal may act as a warning signal. The health indicator may take the form of a tactile signal or include a tactile signal for causing the generation of tactile feedback by a motor. The tactile signal may cause the motor to turn off (or remain off) when the anatomical structure is healthy, and turn on (or remain on) when the anatomical structure is unhealthy.
[0253] The health indicator may include a multi-value indicator. The health indicator may indicate a parameter, for example, by obtaining one of a plurality of values of the parameter corresponding to the anatomical structure. The health indicator may cause a color indicator to be displayed, the color of the color indicator indicating that the value of the parameter is within a parameter value range. The health indicator may cause a sound to be emitted, the pitch and / or volume of the sound indicating that the value of the parameter is within the parameter value range. The health indicator may cause a representation of an audio waveform to be displayed. The health indicator may cause an audio output to represent a tone of blood flow, such as a periodically varying tone. The health indicator may cause tactile feedback to be generated, the amplitude and / or pattern of the tactile feedback may represent the value of the parameter.
[0254] The health indicator may include a digital health indicator. For example, where the health indicator includes a visual signal, the visual signal may cause the display to display a number representing the patient's heart rate and / or breathing rate, for example derived from video data.
[0255] The processor is suitably configured to store the parameter value to generate a curve of the parameter value over time. The health indicator may indicate a curve of the parameter. For example, where the parameter is a measure of the patient's heart rate, the curve of the parameter may include a graph of the heart rate over time.
[0256] The processor can compare the identified parameter with the reference parameter of the patient's anatomy or anatomical structure. The processor can generate a health indicator based on the result of the comparison. The reference parameter is appropriately a parameter value that characterizes whether the patient's anatomy, anatomical structure, or patient is healthy. For example, in the case where the parameter is heart rate and it is known that a heart rate exceeding X beats per minute or less than Y beats per minute indicates poor health, the processor can determine whether the heart rate identified from the video data is between X beats per minute and Y beats per minute. If within this range, the processor can generate a health indicator to indicate that the parameter is within a healthy range. If it exceeds this range, the processor can generate a health indicator to indicate that the parameter is outside the healthy range. Suitably, values X and Y can be selected based on the procedure being performed, the stage of the procedure, the patient's medical history and / or physiology, etc. Suitably, values X and Y are dynamically variable. In general, the reference parameter to be compared with the identified parameter can be determined based on one or more of the procedure being performed, the stage of the procedure, the anatomical structure identified in the video data, the patient's medical history and / or physiological condition, etc.
[0257] The processor is suitably configured to determine whether the identified parameter is within a predefined range of the reference parameter, and the health indicator may include a warning that the identified parameter is outside the predefined range. Taking the above-mentioned heart rate as an example, the health indicator may indicate that the heart rate is in an unhealthy range when the heart rate drops below Y beats / minute. However, if the heart rate drops by another predefined amount, such as 10 beats less than Y per minute, the indicator system may be configured to generate a health indicator including a warning. Such warnings can usefully indicate where the value of the identified parameter reaches a critical level. This can enable remedial action to be taken at the appropriate time.
[0258] The spatio-temporal variation suitably comprises one or more of: a change in position of at least a portion of the patient's anatomy or anatomical structure in the pixel region; a change in intensity of at least a portion of the patient's anatomy or anatomical structure in the pixel region; and a change in color of at least a portion of the patient's anatomy or anatomical structure in the pixel region. The variation may be a variation with respect to at least one pixel of the pixel region.
[0259] Suitably, the processor is configured to identify relative movement between the endoscope and the body of the anatomy and / or relative movement between the body of the anatomy and the remainder of the surgical site, and detect spatio-temporal changes based on the identified relative movement. In this way, the indicator system may consider movement that is not indicative of anatomical peristalsis or other movement that may be expected to be detected, and may filter out such movement in order to more accurately detect the desired movement. The desired movement may be smaller than the movement to be filtered out. For example, when a portion of the intestine is visible in the video data and the endoscope is moved through the surgical site, the peristalsis of the intestine will likely be much smaller than the overall movement of the intestine within the field of view of the endoscope.
[0260] Therefore, suitably, the processor is configured to identify relative movement between the surgical site and the endoscope. This can be performed by detecting movement of the main body of the surgical site in the field of view. For example, when at least a predetermined proportion of the surgical site moves in a given direction, it can be determined that the endoscope is moving relative to the surgical site. Before the movement in the pixel area is analyzed, the overall movement can be quantified and filtered out. The relative movement between the surgical site and the endoscope can be determined from the kinematic data of the robotic system. Therefore, the processor is suitably configured to access the kinematic data and determine a measure of movement between the surgical site and the endoscope from the kinematic data. Before the movement in the pixel area is analyzed, the determined measure of movement can be filtered out.
[0261] In a similar manner, movement of a portion of the surgical site may be considered. For example, a kidney may move at the surgical site, such as by a surgeon. Movement of the entire kidney may be detected and filtered out before analyzing video data of movement of a portion of the kidney or a structure extending therefrom (e.g., a ureter).
[0262] Motions of interest in the video data can be identified by using filters such as frequency filters. For example, peristalsis can be expected to occur at a characteristic frequency or at frequencies within a characteristic frequency range. A large amount of movement of anatomical structures can occur at frequencies outside this range. Therefore, the video data can be filtered to enable analysis only within the frequency range of interest.
[0263] The pixel region may move between frames of the video data. The anatomical structure may move relative to the endoscope. Such movement may be caused by movement of the endoscope relative to the surgical site and / or by movement of the anatomical structure relative to the surgical site. Suitably, the pixel region relates to a given anatomical structure or portion thereof. Thus, when the anatomical structure moves in the field of view, the pixel region moves in a corresponding manner. That is, the movement of the anatomical structure may be tracked, and the tracking used to adjust the position of the pixel region in the video data. Thus, the processor may be configured to detect spatial-temporal changes based on pixels in frames of video data corresponding to the same portion of the anatomical structure.
[0264] The processor can be configured to use image stabilization techniques to stabilize the video data captured by the endoscope. Using such image stabilization techniques can help filter out unwanted movement in the video data. This can improve the accuracy with which the parameters can be identified. The processor can include an image stabilizer 1232 configured to stabilize the received video data.
[0265] The pixel region may form part or all of the field of view of the endoscope. The pixel region may include a region of interest within the field of view. The region of interest may be defined by one or more of the following:
[0266] A bounding box on the display, which may be the center of the display, or offset from the center by a desired amount in a desired direction. The offset direction and / or distance may be preset. The offset direction and / or distance may be user definable.
[0267] A movable and / or resizable bounding box on the display.
[0268] - An abstract area as drawn by an operator, for example by manipulating an input device.
[0269] Single focus or multiple focus.
[0270] • Anatomical structures or other features that can be automatically determined (eg, through video processing such as image matching, image recognition, etc.) The anatomical structures or other features can be tracked across frames of video data.
[0271] A fixed or variable size zone around one or more instrument tips. The size of the zone can be selected by the user. The size of the zone can be selected based on the procedure being performed, the stage of the procedure, surgeon preference, etc.
[0272] The bounding box may be a polygonal bounding box.
[0273] The pixel regions may be selectable, for example user selectable. The pixel regions may be selected in response to a user signal. The processor is suitably configured to receive the user signal and select the pixel regions in response to the received user signal. The user signal may be output by a controller having an input device. The controller is suitably configured to output the user signal in response to a user input relating to video data captured by the endoscope.
[0274] The pixel region may be automatically selected, for example, based on an anatomical structure identification signal. The anatomical structure identification signal may be generated based on image analysis of the video data, for example, using image recognition techniques to identify anatomical structures within the field of view of the endoscope. The anatomical structure identification signal may be generated based on information about the surgical site available to the system. For example, when a model of the surgical site is present and this can be compared to data indicating the position and orientation of the endoscope (which may be derived from, for example, kinematic data), then the expected anatomical structure may be identified within the field of view of the endoscope.
[0275] Suitably, the processor is configured to amplify the space-time variation and generate a health indicator using the amplified space-time variation. The space-time variation may be amplified by amplifier 1226. The space-time variation may include a variation between a first value and a second value. The processor may amplify the space-time variation by multiplying the first value by a first amplification factor and multiplying the second value by a second amplification factor. In some cases, only one of the first value and the second value is multiplied by its corresponding multiplication factor. This may be achieved by setting the other multiplication factor to 1.
[0276] The space-time variation may be amplified about a midpoint of a range between the first value and the second value. The space-time variation may be amplified symmetrically about a midpoint of a range between the first value and the second value. The first amplification factor and the second amplification factor may be the same, but they need not be the same. The space-time variation may be amplified about a selected value between the first value and the second value. The selected point need not be halfway between the first value and the second value.
[0277] One or both of the first upscaling factor and the second upscaling factor may be selected depending on one or more of the following:
[0278] the number of detected spatio-temporal variations in the pixel region;
[0279] This can reduce the likelihood that the amplified spatio-temporal variation obscures another variation, including another detected spatio-temporal variation. Where multiple spatio-temporal variations are detected, the amplification factor can be appropriately selected. This can include selecting a smaller amplification factor. This can include selecting an amplification factor that causes the amplified variation to deviate from another detected spatio-temporal variation.
[0280] Surgeon preference;
[0281] the procedure or stage of the procedure being performed using the surgical robotic system;
[0282] Patient anatomy or anatomical structure;
[0283] the identified parameters; and
[0284] • Comparison of the identified parameters with the reference parameters.
[0285] For example, in the event that the health indicator indicates that the anatomical structure is in poor health, the amplification may be increased. In the event that the health indicator includes a warning, the amplification may be increased. Suitably, in the event that the health indicator includes a warning, the amplification is further increased.
[0286] exist Fig.13 A schematic example of the amplification of the parameters is shown in . Fig.13The time-varying values of the identified parameter 1302 are shown. The identified parameter as shown varies symmetrically around a value s, with a maximum amplitude s1. The amplified parameter varies symmetrically around s, but has a larger maximum amplitude s2. In the case where the identified parameter is a position, the amplified position can make it easier to (e.g., visually) perceive the change.
[0287] As discussed, the indicator system can generate a health indicator that indicates a parameter of the anatomical structure, such as the patient's heart rate or breathing rate. It is known to determine the heart rate and breathing rate in other ways, such as using standard operating room equipment. Suitably, the indicator system is configured to have access to another parameter. The other parameter may include the heart rate and / or breathing rate measured using such standard equipment. The other parameter can be received by the indicator system. The other parameter can be received by the surgical robot system. The other parameter is suitably received from a monitoring system remote from the surgical robot system. The other parameter can be used to calibrate the identified parameter. This can improve the accuracy of the indicator system.
[0288] The receiver is suitably configured to receive further video data of an external portion of the patient. The further video data may include video of an exposed surgical site and / or the patient's skin. Suitably, the processor is configured to detect a further space-time variation in a pixel region of the received further video data. The processor may identify a further parameter in response to the detected further space-time variation.
[0289] The generated health indicator may include a visual indicator. The processor may output the generated health indicator for display on a display. The generated health indicator may include an audio indicator. The processor may output the generated health indicator for driving a speaker. The generated health indicator may be output to be overlaid on at least a portion of a display (e.g., a display for displaying a representation of the surgical site). The overlay may be partially transparent. The overlay may replace the portion of the representation of the surgical site that it overlays. Thus, the health indicator may include a modified portion of the video that may be overlaid on the representation of the surgical site.
[0290] exist Fig.14 A schematic example of a display of a health indicator is shown in FIG. Fig.14A display 1402 is shown. The display includes a representation of a surgical site, including two end effectors 1404. The display also includes two health indicators. A first health indicator 1406 includes a numerical value of an identified parameter (e.g., heart rate). A second health indicator 1408 includes a graph of the identified parameter. The second health indicator may include a graph of the identified parameter shown in the first health indicator, but need not. Any two or more health indicators may be displayed on the display. A health indicator or several health indicators may be selected for display based on one or more of the procedure being performed, the stage of the procedure being performed, surgeon preferences, one value of an identified parameter, or several values of an identified parameter, etc.
[0291] Now refer to Fig.15 A method of indicating a state of at least a portion of a patient's anatomy (e.g., an anatomical structure of the patient) is described. Video data is received at 1502. Video data is received from an endoscope at a surgical site inside a body cavity of a patient. The video data includes video data of the anatomical structure at the surgical site. At 1504, spatio-temporal variations in a pixel region of the received video data are detected. In response to the detected spatio-temporal variations, a parameter of the anatomical structure is identified at 1506. Then, at 1508, a health indicator is generated that indicates the identified parameter or a curve that indicates the identified parameter. At 1510, the generated health indicator is output.
[0292] Operation Mode
[0293] The space-time changes may be detected by using a space-time difference algorithm. The space-time difference algorithm may be enabled and disabled multiple times during a procedure. The indicator system may operate in one of several operating modes. Suitably, the operating modes include:
[0294] Perfusion mode
[0295] The perfusion mode can be used to amplify small changes in the video corresponding to blood flow. For example, in the perfusion mode, the indicator system can detect small changes in the red channel of the video data. Thus, in the perfusion mode, the indicator system is able to appropriately determine the presence or absence of perfusion. In this mode, the indicator system can be calibrated with filters to detect perfusion for a range of human heart rates. Thus, appropriately, the indicator system is configured to perform a temporal analysis of the video data (e.g., the red channel of the video data) to identify the heart rate. In the perfusion mode, the indicator system can apply red and / or infrared spectral filters to detect the presence of oxygenated and deoxygenated hemoglobin.
[0296] Creeping mode
[0297] Peristaltic mode can be used to amplify small movements in the video. In peristaltic mode, the indicator system can detect typical movements of interest, including colonic peristalsis and ureteric peristalsis. The indicator system can detect peristalsis by identifying movements with movement amplitude and / or movement frequency that are characteristic of movements of interest (e.g., colonic peristalsis or ureteric peristalsis).
[0298] Vascular mode
[0299] The vascular mode can be used to amplify small movements due to pulsating blood flow. In the vascular mode, the indicator system can detect movements at a frequency or frequency range of interest. This can allow the indicator system to identify blood vessels. In the vascular mode, the indicator system can apply red and / or infrared spectral filters in order to detect the presence of oxygenated and deoxygenated hemoglobin.
[0300] Hybrid mode
[0301] Hybrid modes are mixed modes that include analyses from any two or more other modes.
[0302] Output Health Indicator
[0303] The health indicator is output in one of several different ways. As discussed elsewhere herein, the health indicator may include a visual signal and / or an audio signal. The health indicator may be output to a user of the surgical robotic system to identify the detected spatio-temporal changes in different ways.
[0304] The space-time variations may be amplified, and the health indicator may include the amplified space-time variations. The health indicator may include the amplified space-time variations added to the video data. The amplified space-time variations may amplify the colors and / or movement in the video data. Thus, these changes may become more visually apparent to a user of the system.
[0305] The health indicator may include a colored region for recoloring a portion of the display. For example, the colored region may be partially transparent to enable a color map to be overlaid on a portion of the display, such as a portion of the display showing a representation of a surgical site. The recolored portion may be used to highlight an area of interest in the representation of the surgical site. The recolored portion may be used to highlight an area where a spatio-temporal change has been detected. For example, the recolored portion may indicate an area in the field of view where peristalsis has been detected.
[0306] The health indicator may include an audio signal indicating the presence or other aspects of a spatio-temporal variation, or a spatio-temporal variation of interest (e.g., a spatio-temporal variation that meets predefined criteria). The health indicator may indicate the presence or other aspects of a spatio-temporal variation of interest at a particular pixel region in the field of view, such as a pixel region surrounding the tip of the end effector. Thus, the end effector may be moved through the field of view of the endoscope, and the health indicator may indicate whether the end effector passes through an area where a spatio-temporal variation of interest exists. This arrangement allows a user of the indicator system to move the end effector across anatomical structures visible in the video data and assess whether a spatio-temporal variation exists with respect to each anatomical structure.
[0307] The audio signal may be rescaled, for example, by frequency adjustment. Such rescaled audio signal may be used to quantify color changes and / or motions. The quantized color changes and / or motions may be quantized relative to background color changes or background motions, respectively. The quantized color changes and / or motions may be quantized relative to predefined color changes or predefined motions, respectively.
[0308] The detected spatio-temporal changes may generally be changes relative to a defined value or a defined baseline value for a given parameter. The baseline may be obtained relative to a frame of video data as a whole or a sub-region of a frame. The baseline may be obtained relative to a previous frame or a sub-region of a previous frame. The previous frame may be the immediately preceding frame or an earlier frame. The previous frame may be selected during a calibration phase. The baseline may be an average value over the region of interest, such as an average value over the region.
[0309] As discussed elsewhere herein, the indicator system can output an indication of the patient's heart rate and / or respiratory rate obtained from the video data captured by the endoscope. The heart rate / respiratory rate can be correlated with the output of the patient's vital signs monitor. The result of this correlation can be used to provide confirmation of sufficient signal detection. For example, this correlation can confirm that the tissue perfusion measured at the intestinal anastomosis is a true signal representing the patient's actual heart rate. Such correlation can be used to determine when the detected heart rate or respiratory rate is not a true heart rate or respiratory rate. This can make it possible to determine more accurate data at the indicator system.
[0310] Health indicators may include measures of parameters globally (e.g., of the entire field of view of an endoscope, or within a body cavity (e.g., by aggregating measures over multiple frames of video data), or within a defined region of the field of view or body cavity (e.g., a region defined in the software, such as by a user during a planning phase), or of a specific anatomical structure. Measures of perfusion may be provided as absolute measures or as relative measures. Relative measures may be provided relative to another anatomical structure, relative to the field of view, relative to the remainder of the field of view (i.e., excluding the anatomical structure being measured), relative to a pixel region or another area at a specified time point or within a specified time range, or relative to a predefined value.
[0311] The indicator system may be configured to identify the parameter at a specified point in time or within a specified time range. The specified time range may be, for example, the duration of a surgical procedure or the duration of a phase of a surgical procedure.
[0312] The indicator system can be used to detect subclinical or subtle patient seizures, such as in neurosurgery. The indicator system can be used to detect subtle muscle movements, such as facial muscles, that can be related to shallow anesthesia levels or anesthesia depth. Thus, the indicator system can be configured to monitor the patient's depth of anesthesia during a surgical procedure. The measure of the depth of anesthesia can be correlated with other vital signs monitored by the indicator system and / or away from the indicator system. This correlation can provide additional information to the surgeon to enable him to more effectively monitor the health status of the patient undergoing the surgical procedure.
[0313] The indicator system is suitably configured to interface with the data system. The indicator system may be configured to output the health indicator to the data system. This may enable automatic reporting of perfusion detection or automatic reporting of measured perfusion index. The data system may include an electronic patient record system. The indicator system is suitably configured to interface with the data system via a network. The network may include a wired network. The network may include a wireless network. Suitably, the network is a dedicated network, such as a hospital network.
[0314] The indicator system may be configured to generate a patient risk score. The health indicator may include a patient risk score. The data system may be configured to generate a patient risk score based on the data output from the indicator system. The patient risk score is a measure of patient risk and may depend on the procedure. The patient risk score is appropriately determined based on the identified parameter values. For example, taking into account a measure of tissue perfusion determined by the indicator system, the patient risk score may provide a measure of the risk of intestinal anastomosis failure. The patient risk score may be determined based on additional surgical data, patient data, and / or surgeon data, any or all of which may be determined by the indicator system, by the surgical robotic system, or from predetermined data.
[0315] The techniques described herein can be used for non-surgical robotic purposes. For example, they can be used in robotic systems or more generally in systems where it is desirable to obtain additional information about a procedure, such as a robotic procedure.
[0316] Applicants hereby independently disclose each individual feature described herein and any combination of two or more such features, as long as these features or combinations can be implemented based on the present specification as a whole according to the common knowledge of those skilled in the art, regardless of whether these features or combinations of features solve any problem disclosed herein, and do not limit the scope of the claims. Applicants point out that various aspects of the present invention may consist of any such single feature or combination of features. In view of the above description, it is obvious to those skilled in the art that various modifications can be made within the scope of the present invention.
Claims
1. An indicator system for a surgical robot system, the indicator system for indicating a state of at least a portion of a patient's anatomy, the surgical robot system comprising a robot having a base and an arm extending from the base, the arm holding an endoscope at an end of the arm remote from the base, the endoscope being configured to be inserted into a body cavity of a patient for viewing a surgical site inside the patient's body, the indicator system comprising: a receiver configured to receive video data of at least a portion of a patient's anatomy at the surgical site from the endoscope; as well as A processor configured to: detecting spatio-temporal variations in a pixel region of received video data; identifying a parameter of the patient's anatomy in response to the detected spatio-temporal changes; generating a health indicator indicative of the identified parameter or a curve indicative of the identified parameter; and outputting the generated health indicator; The processor is further configured to: Relative movement between the endoscope and a main body of the patient anatomy and / or relative movement between the main body of the patient anatomy and a remainder of the surgical site is identified, and the spatio-temporal changes are detected based on the identified relative movement.
2. The indicator system of claim 1, wherein the processor is configured to compare the identified parameters with reference parameters of the patient's anatomy and to generate the health indicator based on a result of the comparison.
3. An indicator system according to claim 1 or claim 2, wherein the space-time variation comprises one or more of the following: a change in position of at least a portion of the patient's anatomy in the pixel region; a variation in intensity of at least a portion of the patient's anatomy in the pixel region; and A spectral variation of at least a portion of the patient's anatomy in the pixel region.
4. An indicator system according to claim 1 or claim 2, wherein the processor is configured to amplify the space-time variation and generate the health indicator using the amplified space-time variation.
5. An indicator system according to claim 4, wherein when the space-time change includes a change between a first value and a second value, the processor is configured to amplify the space-time change by multiplying the first value with a first amplification factor and multiplying the second value with a second amplification factor.
6. An indicator system according to claim 5, and claim 4 is dependent only on claim 2, wherein one or both of the first magnification factor and the second magnification factor are selected according to one or more of the following: a number of detected spatio-temporal variations in the pixel region; surgeon preference; a procedure or a stage of a procedure being performed using the surgical robotic system; said patient anatomy; the identified parameters; and Comparison of the identified parameters with the reference parameters.
7. An indicator system according to claim 1 or claim 2, wherein the processor is configured to select the pixel area in response to one or more of a received user selection signal and an anatomical structure identification signal.
8. The indicator system of claim 7, wherein the processor is configured to identify one or more anatomical structures in the received video data and to output the anatomical structure identification signal in response to identifying the one or more anatomical structures.
9. An indicator system according to claim 1 or claim 2, wherein the spatio-temporal variation comprises peristalsis of at least a portion of the anatomical structure.
10. The indicator system of claim 1 or claim 2, wherein the pixel region comprises one or more of: a representation of at least a portion of a ureter, and said spatio-temporal variations include peristalsis of said ureter; and A representation of at least a portion of an intestinal tract, and the spatio-temporal variations include peristalsis of the intestinal tract.
11. An indicator system according to claim 1 or claim 2, wherein the processor is configured to calibrate the identified parameter using another parameter.
12. An indicator system according to claim 11, wherein the parameter and the further parameter relate to one or both of a patient's heart rate and a patient's breathing rate.
13. The indicator system according to claim 11, wherein the receiver is configured to receive additional video data of an external part of the patient, and the processor is configured to: detecting another spatio-temporal variation in a pixel region of the received further video data; and The further parameter is identified in response to the detected further spatio-temporal variation.
14. The indicator system of claim 1 or claim 2, wherein the generated health indicators include one or more of the following: a visual indicator, and the processor is configured to output the generated health indicator for display on a display; an audio indicator, and the processor is configured to output the generated health indicator for driving a speaker; as well as A tactile indicator is provided, and the processor is configured to output the generated health indicator for driving the motor to provide tactile feedback.
15. An apparatus configured to indicate a status of at least a portion of a patient's anatomy, the apparatus comprising: a receiver configured to receive, from an endoscope at a surgical site within a body cavity of a patient, video data of at least a portion of a patient's anatomy at the surgical site; and A processor configured to: identifying relative movement between the endoscope and a main body of the patient anatomy and / or relative movement between the main body of the patient anatomy and a remainder of the surgical site; The device comprises: a spatio-temporal variation detector configured to detect spatio-temporal variations in a pixel region of the received video data based on the identified relative movement; a parameter identifier configured to identify a parameter of the patient's anatomy in response to the detected spatio-temporal variation; a health indicator generator configured to generate a health indicator indicative of the identified parameter or a curve indicative of the identified parameter; and Outputs the generated health indicator.
16. A surgical robotic system for enhancing a representation of a surgical site capable of performing a surgical procedure, the system comprising: a display configured to display part or all of a representation of at least a portion of a surgical site; a controller including an input device, whereby the input device is capable of controlling an indicator on the display, the controller being configured to output an enhancement signal in response to a user input relating to the representation; as well as An indicator system according to claim 1 or claim 2; Wherein the processor is configured to receive the enhancement signal and, in response to the received enhancement signal and the generated health indicator, enhance the representation of the surgical site.
17. The system of claim 16, comprising a robotic arm having a surgical instrument having an end effector, whereby the representation of the surgical site comprises a representation of the end effector and the indicator comprises a portion of the representation of the end effector.
18. An apparatus for enhancing a representation of a surgical site where a surgical procedure can be performed, the apparatus comprising: a receiver configured to receive, from an endoscope at a surgical site inside a body cavity of a patient, video data of an anatomical structure at the surgical site; and A processor configured to: identifying relative movement between the endoscope and a main body of the patient anatomy and / or relative movement between the main body of the patient anatomy and a remainder of the surgical site; The device comprises: a spatio-temporal variation detector configured to detect spatio-temporal variations in a pixel region of the received video data in response to the identified relative movement; a parameter identifier configured to identify a parameter of the anatomical structure in response to the detected spatio-temporal variation; a health indicator generator configured to generate a health indicator indicative of the identified parameter or a curve indicative of the identified parameter; a display configured to display part or all of a representation of at least a portion of the surgical site; A controller, the controller being configured to: controlling an indicator on the display; and receiving user input related to the representation, and outputting an enhancement signal in response to the received user input; and The apparatus is configured to enhance the representation of the surgical site in response to the enhancement signal and the generated health indicator.
19. The apparatus according to claim 18, wherein the apparatus is configured to: generating a representation of the surgical site based on the stored representation of the surgical site; and / or A representation of the surgical site is generated based on the received video data.
20. An apparatus according to claim 18 or claim 19, wherein the indicator comprises a portion of an imaging end effector of the surgical robotic system, and the controller is configured to control the indicator on the display in response to control of the end effector.
21. An apparatus according to claim 18 or claim 19, configured to enhance the representation of the surgical site based on at least one of joint data and image processing of the representation.
22. An apparatus according to claim 18 or claim 19, configured to determine that enhancement criteria are met, and to enhance the representation of the surgical site in dependence upon the determination.
23. An apparatus according to claim 18 or claim 19, configured to associate the enhancement with at least one group of enhancements.
24. The apparatus of claim 23, the apparatus being configured to display the enhancement according to the group of enhancements to which the enhancement belongs.
25. The apparatus of claim 18 or claim 19, the apparatus being configured to, when the representation of the surgical site comprises a plurality of enhancements, determine at least one of: Number of enhancements; the distance between the plurality of enhancements; an area surrounded by the plurality of enhancements; and A volume enclosed by the plurality of enhancements.
26. An apparatus according to claim 18 or claim 19, the apparatus being configured to display one or more of the following on at least one of the display and a secondary display: Number of enhancements; Connect multiple enhanced lines; An area surrounded by multiple enhancements; and A volume surrounded by multiple enhancements.
Citation Information
Patent Citations
Augmented surgical reality environment
CN106663318A
