System and method for facilitating optimization of imaging device viewpoint during an operation session of a computer-assisted operating system

Abstract

A viewpoint optimization system identifies a condition associated with an operation session during which a computer-assisted operation system performs a plurality of operations on a body while an imaging device included in the computer-assisted operation system provides an image of the body from a first viewpoint for display on a display device during the operation session. The viewpoint optimization system defines a second viewpoint of the imaging device based on the identified condition, the second viewpoint being more optimized than a first viewpoint of an operation included in the plurality of operations. In addition, the viewpoint optimization system directs the display device to display an indication of the second viewpoint. Corresponding systems and methods are also disclosed.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 804,688, filed on February 12, 2019, entitled “SYSTEMS AND METHODS FOR FACILITATING OPTIMIZATION OF AN IMAGING DEVICE VIEWPOINT DURING AN OPERATINGSESSION OF A COMPUTER-ASSISTED OPERATION SYSTEM,” the contents of which are incorporated herein by reference in their entirety. Background Art

[0003] Various technologies, including computing, robotics, medical, and extended reality (e.g., augmented reality, virtual reality, etc.), enable users, such as surgeons, to perform and be trained to perform various types of operations and procedures. For example, users can perform and be trained to perform minimally invasive medical procedures, such as computer-assisted surgical procedures in a clinical setting (e.g., operating on the body of a living human or animal patient), in a non-clinical setting (e.g., operating on a human or animal cadaver, a body of tissue removed from a human or animal anatomy, etc.), in a training environment (e.g., operating on the body of a physical anatomical training model, the body of a virtual anatomical model in an extended reality environment, etc.), and the like.

[0004] During an operation session in any such environment, the user may view an image of the operation area associated with the body (e.g., an area inside the body) as the user directs the instrument of the computer-assisted operation system to perform an operation on the body in the operation area. The image may be provided by an imaging device such as an endoscope included in the computer-assisted operation system. As various operations are performed in this manner, the viewpoint of the imaging device may significantly affect the efficiency and effectiveness with which the user is able to perform the operation. Summary of the invention

[0005] Systems and methods for facilitating optimization of an imaging device viewpoint during an operation session of a computer-assisted operating system are described herein. For example, one embodiment is implemented as a system comprising a memory storing instructions and a processor communicatively coupled to the memory and configured to execute the instructions. For example, the instructions may direct the processor to identify a condition associated with an operation session during which the computer-assisted operating system performs a plurality of operations on a body while an imaging device included within the computer-assisted operating system provides an image of the body from a first viewpoint for display on a display device during the operation session. The instructions may also direct the processor to define a second viewpoint of the imaging device based on the identified condition, the second viewpoint being more optimized than a first viewpoint of an operation included in the plurality of operations. In addition, the instructions may cause the processor to direct the display device to display an indication of the second viewpoint.

[0006] Another exemplary embodiment is also implemented as a system, which includes a memory storing instructions and a processor communicatively coupled to the memory and configured to execute the instructions. In this embodiment, the instructions may direct the processor to determine during an operation session that a user uses a first wrist gesture associated with a first viewpoint to direct a computer-assisted operating system to perform an operation included in a plurality of operations, during which the computer-assisted operating system performs the plurality of operations on a body while an imaging device included in the computer-assisted operating system provides an image of the body displayed to the user through a display device from the first viewpoint. The instructions may also direct the processor to define during the operation session a second viewpoint associated with a second wrist gesture, the second wrist gesture being more optimized than the first wrist gesture for directing the execution of the operation, the second viewpoint having a horizontal orientation different from the horizontal orientation of the first viewpoint. The instructions may further direct the processor to direct the display device to integrate a reticle overlay graphic indicating the horizontal orientation of the second viewpoint with the image of the body displayed from the first viewpoint when the display device displays the image of the body from the first viewpoint. In response to the integration of the reticle overlay graphic indicating the horizontal orientation of the second viewpoint, the instructions may direct the processor to receive a user input indicating that the user selects to view the image of the body from the second viewpoint instead of viewing the image of the body from the first viewpoint. Thus, the instructions may direct the processor to direct the display device to switch from displaying an image of the body from a first viewpoint to displaying an image of the body from a second viewpoint in response to a user input.

[0007] Another exemplary embodiment is implemented as a method performed by a viewpoint optimization system. For example, the method includes identifying a condition associated with an operation session during which a computer-assisted operation system performs a plurality of operations on a body while an imaging device included in the computer-assisted operation system provides an image of the body from a first viewpoint for display on a display device during the operation session. The method also includes defining a second viewpoint of the imaging device based on the condition, the second viewpoint being more optimized than a first viewpoint of an operation included in the plurality of operations. In addition, the method includes directing the display device to display an indication of the second viewpoint.

[0008] Another exemplary embodiment is implemented by a non-transitory computer-readable medium storing instructions that, when executed, direct a processor of a computing device to perform operations described herein. For example, the instructions may direct the processor to identify a condition associated with an operation session during which a computer-assisted operating system performs a plurality of operations on a body while an imaging device included within the computer-assisted operating system provides an image of the body from a first viewpoint for display on a display device during the operation session. The instructions may also direct the processor to define a second viewpoint of the imaging device based on the identified condition, the second viewpoint being more optimized than a first viewpoint of an operation included in the plurality of operations. Additionally, the instructions may cause the processor to direct the display device to display an indication of the second viewpoint. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings illustrate various embodiments and are part of the specification. The illustrated embodiments are examples only and do not limit the scope of the present disclosure. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements.

[0010] Figure 1 An exemplary computer-assisted operating system according to the principles described herein is illustrated.

[0011] Figure 2 The present invention illustrates a method for implementing ... Figure 1 An exemplary imaging device within a computer-assisted operating system.

[0012] Figure 3 Illustrated are exemplary viewpoints from which an imaging device captures images of a body according to principles described herein.

[0013] Figure 4 An exemplary viewpoint optimization system for facilitating optimization of an imaging device viewpoint during an operational session of a computer-assisted operating system according to the principles described herein is illustrated.

[0014] Figure 5AAn exemplary operating session according to principles described herein is illustrated during which a computer-assisted operating system performs a plurality of operations on a body while an imaging device included within the computer-assisted operating system captures images of the body from different exemplary viewpoints.

[0015] Figure 5B An exemplary display device according to the principles described herein is shown on which an Figure 5A Images captured from different viewpoints during a manipulation session.

[0016] Figure 5C The diagram shows a method for performing a user-defined Figure 5A and Figure 5B Example wrist gestures used to perform operations while viewing images from different viewpoints are illustrated in FIG.

[0017] Figure 6 Illustrated are display devices showing images from exemplary viewpoints having different horizontal orientations according to principles described herein.

[0018] Figure 7 Illustrated is a display device displaying images from exemplary viewpoints having different zoom orientations according to principles described herein.

[0019] Figure 8 Illustrated are display devices showing images from exemplary viewpoints having different planar orientations according to principles described herein.

[0020] Fig. 9 A display device is illustrated that displays images from exemplary viewpoints having orientations that differ in various respects, according to principles described herein.

[0021] Fig.10 A display device showing images from different exemplary viewpoints, switching between the different exemplary viewpoints in a semi-automatic manner, is illustrated according to the principles described herein.

[0022] Fig.11 A display device showing images from different exemplary viewpoints is illustrated, switching between the different exemplary viewpoints in an automatic manner, according to the principles described herein.

[0023] Fig.12 The diagram shows a method for Figure 4 The viewpoint optimization system provides exemplary entities of input data.

[0024] Fig.13 An exemplary method for facilitating optimization of an imaging device viewpoint during an operational session of a computer-assisted operating system according to the principles described herein is illustrated.

[0025] Fig.14 An exemplary computing device according to the principles described herein is illustrated. DETAILED DESCRIPTION

[0026] This article describes systems and methods for facilitating the optimization of the viewpoint of an imaging device during an operation session of a computer-assisted operating system. As described above, the effectiveness and efficiency with which a user (e.g., a surgeon, a member of a surgical team, another user of a computer-assisted operating system, etc.) may be able to guide a computer-assisted operating system (e.g., a computer-assisted surgical system) to perform a particular operation may be significantly affected by the viewpoint of the image captured and provided by the imaging device to the user during the operation. Unfortunately, however, as users become accustomed to using the computer-assisted operating system, consistent and effective optimization of the viewpoint of the imaging device may be a skill that is difficult for users to master. In addition, in certain scenarios (e.g., training scenarios, scenarios in which operations are performed according to the preferences of certain users, operations performed using traditional laparoscopic techniques, etc.), the skill of viewpoint optimization may not work at all. For example, in these scenarios, it may be necessary to perform the optimization of the viewpoint of the imaging device in a fully automatic manner to allow the user to focus on other aspects of the operation rather than viewpoint optimization. Therefore, the systems and methods described herein can be configured to facilitate users (including novice computer-assisted operating system users who are trained to learn to use the system) to improve their viewpoint selection effectiveness and skills. As described below, this is accomplished by making it easier for a user to see, understand, and switch to a more optimal view during a clinical, non-clinical, training, or other operational session. For example, the systems and methods described herein can facilitate real-time optimization of viewpoints during an operational session by making suggestions to encourage a user to switch viewpoints, semi-automatically switching viewpoints, automatically switching viewpoints, and the like.

[0027] In one exemplary embodiment, a system for facilitating viewpoint optimization of an imaging device may include a memory storing instructions and a processor communicatively coupled to the memory and configured to execute the instructions to perform functions associated with facilitating viewpoint optimization or implemented by the memory, from which the imaging device captures and provides images during an operation session. For example, according to the instructions, the system may identify conditions associated with the operation session, during which a computer-assisted operating system performs multiple operations on a body, while an imaging device included in the computer-assisted operating system provides an image of the body from a first viewpoint for display on a display device during the operation session. For example, as will be described in more detail below, the identified conditions may relate to a user's current wrist posture, a specific operation included in the multiple operations being performed, the identity of the user who directed the computer-assisted operating system to perform the operation, known habits of the identified user (e.g., previously observed performance advantages and disadvantages, etc.), the current Cartesian position of the user's hands relative to each other, the co-location state of the user's hands relative to the controlled instrument, etc.

[0028] Based on the identified condition, the system can define a second viewpoint for the imaging device that is more optimized than the first viewpoint for the operation being performed, and can direct the display device to display an indication of the second viewpoint. For example, as will be described in more detail below, the system can direct the display device to display an indication of the second viewpoint by automatically or semi-automatically switching from displaying an image captured from the first viewpoint to an image captured from the second viewpoint. As another example, the system can direct the display device to display an indication of the second viewpoint by directing the display device to continue to display the image captured from the first viewpoint while also introducing a graphical overlay or other indicator that is presented with (e.g., integrated with) the image presented from the first viewpoint.

[0029] As used herein, "optimization" of a viewpoint may refer to changing one or more features of a viewpoint (e.g., one or more aspects or parameters defining an orientation) to improve the viewpoint for a particular operation. Thus, a viewpoint that is "more optimized" than another viewpoint for a particular operation will be understood to be an improvement in some way (e.g., making the operation easier to perform effectively and / or efficiently), but it will also be understood that an "optimized" viewpoint or "better" viewpoint is not necessarily the best possible viewpoint for the operation. The determination that one viewpoint is better than another may be subjective (e.g., based on the opinions of experienced users, etc.) or objective (e.g., based on a viewpoint selection algorithm, etc.).

[0030] Embodiments of the systems and methods described herein generally relate to or employ computer-assisted operating systems, such as computer-assisted medical systems (e.g., minimally invasive robotic surgical systems, conventional laparoscopic surgical systems employing robotic endoscopes, or other computer-assisted vision systems, etc.). However, as will be described in more detail below, it should be understood that the inventive aspects disclosed herein may be embodied and implemented in various ways, including by employing robotic and non-robotic embodiments and implementations. Embodiments related to surgical or other medical systems are merely exemplary and should not be construed as limiting the scope of the inventive aspects disclosed herein. Because the instruments, systems, and methods described herein may be used for medical treatment or diagnosis, cosmetic enhancement, imaging of human or animal anatomical structures, collecting data from human or animal anatomical structures, setting up or removing systems, training medical or non-medical personnel, etc. (any of which may or may not involve surgical aspects), any reference to, for example, surgical instruments, surgical techniques, and / or other such details related to the surgical context will be understood as non-limiting. In other examples, the apparatus, systems, and methods described herein may also be used for procedures performed on or with animals, human cadavers, animal cadavers, portions of human or animal anatomy, tissue removed from human or animal anatomy (which may or may not be re-implanted into human or animal anatomy), non-tissue artifacts, training models, etc. In other examples, the apparatus, systems, and methods described herein may be applied for non-medical purposes, including use in industrial systems, general purpose robotics, telesurgery systems, and / or sensing or manipulating non-tissue artifacts.

[0031] The systems and methods described herein can provide various benefits to facilitate optimization of the imaging device viewpoint. In a non-computer-assisted operation session (e.g., a standard surgical procedure that does not employ a robot or other computer-assisted operation technique), it may be intuitive and natural for the surgeon to move his or her head and body to obtain the optimal viewpoint of the body being operated on and to find a suitable angle to perform various operations. For example, if the surgeon needs to see more details, he or she may naturally move the head closer to the operating area to see better. As another example, a certain wrist posture may provide maximum comfort and control for performing operations such as suturing an incision to close it, and therefore, the surgeon may naturally position him or herself relative to the body to be able to use the wrist posture when he or she performs the suturing operation.

[0032] When guiding the computer-assisted operating system to perform similar operations, the same principles (e.g., perspective and detail, wrist posture, etc.) can be applied, but it may not be intuitive for the user to successfully achieve the optimal viewpoint (e.g., the viewpoint that provides the optimal view of the operating area, the viewpoint associated with the optimal wrist posture, etc.), especially for users who are not familiar with the computer-assisted operating system. For example, a surgeon who wants to see a more detailed view of the surgical area may not be able to simply move his or her head closer to the patient to obtain a better view while performing an operation using the computer-assisted operating system. Instead, in order to obtain a more optimal viewpoint, the surgeon may have to perform a more deliberate series of actions. For example, the surgeon can step on a foot pedal to switch the system from an operating mode in which the robotic instrument follows or imitates the surgeon's hand movements to an imaging adjustment mode in which the surgeon uses gestures to modify the orientation of the imaging device (e.g., zoom, translate, rotate and / or articulate the imaging device, etc.). The surgeon can make imaging device orientation adjustments in the imaging adjustment mode, and then can reposition his or her hand and perform certain additional actions (e.g., pinching actions, etc.) to switch the computer-assisted operating system back to the operating mode.

[0033] While expert users may become so adept at this process that they are comfortable making imaging adjustments at a frequency of every second or every few seconds during an operating session, less experienced users may be less comfortable with these imaging adjustment procedures. As a result, these users may be less likely to switch from one viewpoint to another, even if the new viewpoint would be more optimal. Furthermore, less experienced users may not fully appreciate the extent to which the selected viewpoint determines not only what can be seen, but also the wrist postures that can be used, the sensitivity of hand movements that can be used, etc. As a result, these users may inadvertently or unknowingly use suboptimal wrist postures or fail to fully utilize the benefits associated with an optimal viewpoint. As a result, various operations performed by these users may be more difficult and / or more time consuming than would be the case using a more optimized viewpoint.

[0034] To address these challenges, the systems and methods described herein help train and guide users (e.g., surgeons, etc.) to find more optimal viewpoints, use viewpoints more successfully to improve the efficiency and / or effectiveness of the operations being performed, and the like. As will be described in more detail below, the systems and methods described herein can be used during clinical operation sessions and help provide training and practice during non-clinical or training operation sessions. In addition, the systems and methods described herein can more easily switch from less optimal viewpoints to more optimal viewpoints, so that even experienced users of computer-assisted operation systems (e.g., expert users who are already good at finding optimal viewpoints) can benefit from convenient viewpoint switching as they continue to continuously update their viewpoints to maintain optimal viewpoints during operation sessions. Therefore, for novice and expert users, the systems and methods described herein can ultimately help lead to easier, more effective, and more efficient operation execution; reduce the learning curve of complex computer-assisted operation systems; and, in the case of medical-related systems, improve patient outcomes.

[0035] Various embodiments will now be described in more detail with reference to the accompanying drawings.The systems and methods described herein may provide one or more of the benefits mentioned above as well as various additional and / or alternative benefits that will become apparent from the following description.

[0036] The viewpoint optimization systems and methods described herein can be operated as part of or in conjunction with a computer-assisted operating system (e.g., a computer-assisted medical system such as a robotic surgical system). Therefore, in order to facilitate understanding of the viewpoint optimization systems and methods described herein, an exemplary computer-assisted operating system will now be described. The exemplary computer-assisted operating system described is illustrative and not restrictive. The viewpoint optimization systems and methods described herein can be integrated with (e.g., built into) or otherwise operated as part of or in conjunction with the computer-assisted operating system described herein and / or other suitable computer-assisted operating systems.

[0037] Figure 1An exemplary computer-assisted operating system 100 ("operating system 100") is illustrated. Although, as described above, a computer-assisted operating system may be used to perform various types of operations in various types of applications, operating system 100 will be understood as a computer-assisted medical system configured to perform operations associated with surgical and / or non-surgical medical procedures. As shown, operating system 100 may include a manipulation system 102, a user control system 104, and an auxiliary system 106 that are communicatively coupled to one another. A medical team may utilize operating system 100 to perform computer-assisted medical procedures or other such procedures on the body of a patient 108 or any other body that may be used for a particular embodiment. As shown, a medical team may include a first clinician 110-1 (e.g., a surgeon or other physician), an assistant 110-2, a nurse 110-3, and a second clinician 110-4 (e.g., an anesthesiologist or other physician), all of whom may be collectively referred to as "team members 110," and each of whom may control, interact with, or otherwise be a user of operating system 100. Additional, fewer, or alternative team members that may be used for a particular embodiment may be present during a medical procedure. For example, for some medical procedures, "clinician 110-1" may not be a physician. Additionally, the team composition for non-medical procedures is often different and includes other combinations of members serving in non-medical roles.

[0038] Although Figure 1 An ongoing medical procedure, such as a minimally invasive surgical procedure, is illustrated, but it should be understood that the operating system 100 may similarly be used to perform open surgical procedures or other types of operations that may similarly benefit from the accuracy and convenience of the operating system 100. For example, the operating system 100 may also be used to perform operations such as exploratory imaging operations, simulated medical procedures for training purposes, and / or other operations. Furthermore, it should be understood that any medical procedure or other operation employing the operating system 100 may include not only an operation phase, but may also include pre-operative, post-operative, and / or other such operation phases.

[0039] like Figure 1As shown, the manipulation system 102 may include a plurality of manipulator arms 112 (e.g., manipulator arms 112-1 to 112-4), to which a plurality of instruments (e.g., surgical instruments, other medical instruments, or other instruments) may be coupled. Each instrument may be any suitable surgical tool (e.g., a tool having tissue interaction functionality), a medical tool, an imaging device (e.g., an endoscope), a sensing instrument (e.g., a force sensing instrument), a diagnostic instrument, or an instrument that can be used for a computer-assisted medical procedure such as performing a surgical procedure on a patient 108 (e.g., by being at least partially inserted into the patient 108 and being manipulated to perform a computer-assisted medical procedure on the patient 108). Although the manipulation system 102 is depicted and described herein as including four manipulator arms 112, it should be appreciated that the manipulation system 102 may include only a single manipulator arm 112 or any other number of manipulator arms that may be used for a particular embodiment. In addition, it will be understood that in some exemplary systems, certain instruments may not be coupled to or controlled by a manipulator arm, but may be hand-held and manually controlled (e.g., by a surgeon, other clinician, or other medical personnel). For example, certain handheld devices of this type may be used in conjunction with or as an alternative to a computer-assisted instrument coupled to a Figure 1 The manipulator arm 112 is shown in FIG. 1 and described in various examples herein.

[0040] The manipulator arm 112 and / or an instrument attached to the manipulator arm 112 may include one or more displacement sensors, orientation sensors, and / or position sensors for generating raw (i.e., uncorrected) kinematic information. One or more components of the operating system 100 may be configured to use the kinematic information to track (e.g., determine its position) and / or control the instrument.

[0041] The user control system 104 can be configured to facilitate the clinician 110-1 to control the manipulator arm 112 and the instruments attached to the manipulator arm 112. For example, for a surgical procedure, the clinician 110-1 can be a surgeon. In this example, the clinician 110-1 can interact with the user control system 104 to remotely move or manipulate the manipulator arm 112 and the instruments to perform a plurality of operations included in a surgical operation or other medical procedure. To this end, the user control system 104 can provide the clinician 110-1 with an image (e.g., a high-definition 3D image) of the body of the patient 108 captured from a specific viewpoint by an imaging device. In some examples, the user control system 104 can include a stereoscopic viewer with two displays, wherein the clinician 110-1 can view a stereoscopic image of the body captured from a viewpoint by a stereoscopic imaging device. The clinician 110-1 can utilize the image to perform one or more procedures, wherein one or more instruments are attached to the manipulator arm 112.

[0042] In order to facilitate the control of the instrument, the user control system 104 may include a set of master controls. These master controls can be manipulated by the clinician 110-1 to control the movement of the instrument (e.g., by utilizing a robot and / or remote operation technology). The master controls can be configured to detect various hand, wrist and finger movements of the clinician 110-1. In this way, the clinician 110-1 can intuitively perform a procedure using one or more instruments. As described above, the master controls and other controls such as foot pedals can allow the clinician 110-1 to not only control the manipulator arm 112 to perform the required operations of the surgical procedure, but also control at least one manipulator arm 112 associated with the imaging device, so as to set and continuously adjust the orientation of the imaging device as the operation is performed (e.g., zoom, level, plane, pitch, yaw and / or other aspects of orientation).

[0043] The auxiliary system 106 may include one or more computing devices configured to perform the primary processing operations of the operating system 100. In such a configuration, the one or more computing devices included in the auxiliary system 106 may control and / or coordinate operations performed by various other components of the operating system 100, such as the manipulating system 102 and / or the user control system 104. For example, a computing device included in the user control system 104 may transmit instructions to the manipulating system 102 via one or more computing devices included in the auxiliary system 106. As another example, the auxiliary system 106 may receive and process image data representing an image captured by an imaging device attached to one of the manipulator arms 112.

[0044] In some examples, the assistance system 106 may be configured to present visual content to the team member 110, who may not otherwise have access to the images provided to the clinician 110-1 at the user control system 104. To this end, the assistance system 106 may include a display monitor 114 configured to display one or more user interfaces, an image of the body of the patient 108 (e.g., a 2D or 3D image), information associated with the patient 108 and / or the medical procedure, and / or any other content that may serve a particular implementation. In some examples, the display monitor 114 may display the image of the body and additional content (e.g., graphical content, contextual information, etc.) displayed simultaneously with the image. The display monitor 114 may be implemented by a touch screen display with which the team member 110 may interact (e.g., via touch gestures) to provide user input to the operating system 100, or may be implemented by any other type of display screen that may be useful for a particular implementation.

[0045] As will be described in greater detail below, the viewpoint optimization system may be implemented within the operating system 100 or may operate in conjunction with the operating system 100. For example, in some embodiments, the viewpoint optimization system may be implemented by the user control system 104 (e.g., using a display device such as a stereoscopic viewer included in the user control system 104), the auxiliary system 106 (e.g., using a display device such as a display monitor 114), or by other suitable means.

[0046] The operating system 102, the user control system 104, and the auxiliary system 106 may be communicatively coupled to each other in any suitable manner. Figure 1 As shown, the operating system 102, the user control system 104, and the auxiliary system 106 can be communicatively coupled via a control line 116, which can represent any wired or wireless communication link that can be used for a particular implementation. To this end, the operating system 102, the user control system 104, and the auxiliary system 106 can each include one or more wired or wireless communication interfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, etc.

[0047] Figure 2 An exemplary imaging system 200 is illustrated, which, as will be described below, can be used in accordance with the systems and methods described herein to capture images of a body from various viewpoints characterized by various aspects of orientation. As shown, the imaging system 200 includes an imaging device 202 and a controller 204. The imaging system 200 may include additional or alternative components for particular embodiments. For example, the imaging system 200 may include various optical and / or electrical signal transmission components (e.g., wires, lenses, optical fibers, choke circuits, waveguides, etc.), cables that house wires and / or optical fibers and are configured to interconnect the imaging device 202 and the controller 204, etc.

[0048] The imaging device 202 may be implemented by an endoscope or similar such imaging tool (e.g., laparoscope, etc.) that is configured to capture images of a scene, such as any of the internal views of the body described herein. Figure 2 In the example of , the imaging device 202 is stereoscopic. However, in other examples, the imaging device 202 can be monoscopic (e.g., by including one image sensor instead of two image sensors). In addition, although imaging devices such as endoscopes, laparoscopes, etc. can be used in the present invention with respect to Figure 2 The described manner captures images of the body, but it will be understood that in other examples, other imaging techniques (e.g., ultrasound imaging, imaging other than visible light) and other types of imaging devices or combinations of devices may be used to capture images of the body.

[0049] For example, ultrasound imaging or other such techniques may be employed in some examples, wherein the imaging device includes an ultrasound probe inserted into the operating area and can be manipulated using an instrument attached to the manipulator arm, rather than being controlled by itself directly attached to the manipulator arm. As another example, hyperspectral imaging techniques and tools may be used to capture images in other regions of the electromagnetic spectrum other than the visible spectrum. This may facilitate, for example, imaging of features (e.g., blood vessels, etc.) below an outer surface that reflects visible light. Similarly, due to the known metabolism and / or decomposition patterns of imaging agents, performing infrared, ultraviolet, or other hyperspectral imaging may allow fluorescent imaging agents to be injected into tissues to highlight imaging techniques of different features at different times. Such imaging techniques may be implemented by a single imaging system (e.g., imaging system 200) or by different imaging systems (e.g., an imaging system that can be replaced with imaging system 200 if the medical team performing the operation requires it).

[0050] As shown, the imaging device 202 includes a camera 206, a shaft 208 coupled to and extending away from the camera 206, image sensors 210 at the distal end of the shaft 208 (i.e., a right image sensor 210-R and a left image sensor 210-L), and an illumination channel 212. Each of these elements will now be described in more detail.

[0051] In some examples, imaging device 202 can be controlled by a surgical team member, such as clinician 110-1, in a computer and / or robotic-assisted manner. For example, camera 206 can be coupled to a manipulator arm of a computer-assisted operating system (e.g., one of manipulator arms 112 of operating system 100) and controlled using robotic and / or teleoperation techniques.

[0052] The distal end of the shaft 208 can be positioned at an operative area to be imaged by the imaging device 202. In this configuration, the imaging device 202 can be used to capture images of anatomical structures and / or other objects that are part of or near the body. In various embodiments, the shaft 208 is rigid (e.g., Figure 2 ). Alternatively, shaft 208 may be coupled (e.g., including an articulated mechanism that allows for pitch and / or yaw orientation adjustment) and / or may be flexible. Further, while the distal end of shaft 208 is shown in this example as terminating at an angle orthogonal to the axis of shaft 208, such that imaging device 202 captures images of objects about the axis of shaft 208 (i.e., objects directly in front), in other examples, the distal end of shaft 208 may taper at an angle that is not orthogonal to the axis of shaft 208 (e.g., a 30° angle, a 45° angle, etc.). In this way, imaging device 202 may capture images of objects that are offset from the axis of shaft 208, thereby allowing for greater flexibility in the location where the field of view of imaging device 202 may be directed.

[0053] Image sensor 210 may be implemented by any suitable image sensor, such as a charge coupled device (“CCD”) image sensor, a complementary metal oxide semiconductor (“CMOS”) image sensor, etc. In some examples, Figure 2 As shown, image sensor 210 is located at the distal end of shaft 208. Alternatively, image sensor 210 may be located closer to the proximal end of shaft 208, inside camera 206, or outside imaging device 202 (e.g., inside controller 204). In these alternative configurations, optics (e.g., lenses, optical fibers, etc.) included in shaft 208 and / or camera 206 may transmit light from the scene to image sensor 210.

[0054] Image sensor 210 is configured to detect (e.g., capture, collect, sense, or otherwise acquire) light. For example, image sensor 210-R is configured to detect light from a right perspective, while image sensor 210-L is configured to detect light from a left perspective. The light detected by image sensor 210 may include, for example, visible light reflected from a body or an object within the field of view, hyperspectral (i.e., invisible) light reflected from the body, fluorescent illumination produced in vivo by a fluorescent imaging agent, or any other light of any frequency that may be used in a particular embodiment. As described in more detail below, image sensor 210 may convert the detected light into data representing one or more images.

[0055] The illumination channel 212 may be implemented by one or more optical components (eg, optical fibers, light guides, lenses, etc.) As will be described below, illumination may be provided through the illumination channel 212 to illuminate the operating area and objects included therein.

[0056] The controller 204 may be implemented by any suitable combination of hardware and software configured to control and / or interface with the imaging device 202. For example, the controller 204 may be implemented at least in part by a computing device included in the auxiliary system 106.

[0057] Controller 204 includes a camera control unit ("CCU") 214 and an illumination source 216. Controller 204 may include additional or alternative components that may be used for a particular implementation. For example, controller 204 may include circuitry configured to provide power to components included in imaging device 202. In some examples, CCU 214 and / or illumination source 216 may alternatively be included in imaging device 202 (e.g., in camera head 206).

[0058] CCU 214 is configured to control various parameters of image sensor 210 (e.g., activation time, automatic exposure, etc.). As will be described below, CCU 214 may also be configured to receive and process image data from image sensor 210. Figure 2 2 is shown as a single unit, but the CCU 214 may alternatively be implemented by a first CCU configured to control the right image sensor 210 -R and a second CCU configured to control the left image sensor 210 -L.

[0059] The illumination source 216 may be configured to generate and emit illumination 218. The illumination 218 (also referred to herein as light) may travel through the illumination channel 212 to a distal end of the shaft 208 where the illumination 218 exits to illuminate a scene.

[0060] The illumination 218 may include visible light or hyperspectral light having one or more frequency (e.g., color) components. The illumination 218 may additionally or alternatively include fluorescence excitation illumination configured to induce fluorescence illumination through a fluorescent imaging agent (e.g., by exciting a fluorescent imaging agent that has been injected into a patient's bloodstream to begin emitting fluorescent illumination). In some examples, the fluorescence excitation illumination has a wavelength in the infrared region (e.g., in the near infrared region). Although a single illumination source 216 is shown as included in the controller 204, multiple illumination sources, each configured to generate and emit illumination of different configurations, may alternatively be included in the controller 204.

[0061] To capture one or more images of a scene, the controller 204 (or any other suitable computing device) can activate the illumination source 216 and the image sensor 210. When activated, the illumination source 216 emits illumination 218, which travels to the operating area via the illumination channel 212. The image sensor 210 detects the illumination 218 reflected from one or more surfaces of the anatomical structure of the body or other objects near the body. In the case where the illumination 218 includes fluorescence excitation illumination, the image sensor 210 can additionally or alternatively detect the fluorescence illumination caused by the fluorescence excitation illumination.

[0062] The image sensor 210 (and / or other circuitry included in the imaging device 202) can convert the sensed light into image data 220 representing one or more images of the scene. For example, the image sensor 210-R outputs image data 220-R representing an image captured from a right perspective, while the image sensor 210-L outputs image data 220-L representing an image captured from a left perspective. The image data 220 can have any suitable format and can be transmitted from the image sensor 210 to the CCU 214 in any suitable manner.

[0063] The CCU 214 may process (e.g., package, format, encode, etc.) the image data 220 and output processed image data 222 (e.g., processed image data 222-R corresponding to the image data 220-R and processed image data 222-L corresponding to the image data 220-L). The processed image data 222 may be transmitted to an image processor (not shown), which may prepare the processed image data 222 for display on one or more display devices (e.g., in the form of a video stream and / or one or more still images). For example, the image processor may generate one or more full-color images, grayscale images, and / or fluorescent images based on the image data 222 for display on one or more display devices (such as a stereoscopic viewer of the user control system 104 or a display monitor 114 of the auxiliary system 106).

[0064] As the imaging system 200 captures images of the body in the manner described above, the imaging system may capture images from a particular viewpoint. Based on user preferences for which operation is being performed at any given time and various other factors, it may be desirable to adjust the viewpoint from which the image is captured by adjusting one or more aspects of the orientation of the viewpoint.

[0065] To illustrate, Figure 3 An exemplary viewpoint 300 is shown from which an imaging device 202 (within the imaging system 200) captures an image of a body 302. As described above, operations within an operation session may be performed with respect to (e.g., within) various types of bodies, including, but not limited to, the body of a living patient, the body of a corpse, the body of a non-human subject (e.g., an animal, etc.), or another such organism. In some examples, the body on which the operation is performed or within may be an anatomical portion of only one of these other types of bodies. For example, the body may be a disembodied organ or other body part taken from a complete organism, an artificial training device (e.g., an artificial organ or other body part), or a virtual body for training, experimentation, and / or other such purposes (e.g., using a real or extended reality training system). In other examples, a computer-assisted operating system similar to the operating system 100 may be used to perform inspection or overhaul operations within the body of a complex electrical or mechanical system such as an engine or other complex system. As yet another example, the computer-assisted operating system may be used in a law enforcement or surveillance environment (e.g., inspecting and disabling dangerous explosive devices, monitoring in confined spaces, etc.) and / or in any other environment or with any other technology that may be used for a particular implementation.

[0066] As used herein, a "viewpoint" of an imaging device such as viewpoint 300 (also referred to as an "imaging device viewpoint") may refer to a combination of various aspects that combine together to define the position, orientation, configuration, resolution, etc. of an image captured by the imaging device at a particular moment. Figure 3Viewpoint 300 is depicted as an arrow extending along the axis of imaging device 202 to indicate that as changes are made to the position, orientation, configuration, resolution, etc. of imaging device 202 , viewpoint 300 will be adjusted accordingly.

[0067] The viewpoint 300 may be defined by various aspects of the position, orientation, configuration, resolution, etc. of the imaging device 202. As will now be described, each of these aspects will be referred to herein as a different aspect of orientation or a different type of orientation 304 (e.g., orientations 304-1 through 304-5) of the viewpoint 300.

[0068] As shown, the zoom orientation 304-1 of the viewpoint 300 relates to the apparent position of the viewpoint 300 along the longitudinal axis of the axis of the imaging device 202. Thus, for example, the adjustment of the zoom orientation 304-1 may cause the image to appear larger (closer) or smaller (farther) than the initial zoom orientation 304-1 that has not been adjusted. In some embodiments, the adjustment of the zoom orientation 304-1 may be made by physically moving or sliding the imaging device 202 closer to the portion of the body 302 being captured or away from the portion of the body 302 being captured. Such zoom adjustments may be referred to herein as optical zoom adjustments. In other embodiments, adjustments may be made without physically moving or adjusting the physical orientation of the imaging device 202. For example, the zoom adjustment may be made optically by internally changing the lens, lens configuration, or other optical aspects of the imaging device 202, or by applying a digital zoom manipulation to the image data captured by the imaging device 202.

[0069] The horizontal orientation 304-2 of the viewpoint 300 is aligned with the longitudinal axis of the imaging device 202 along the axis of the imaging device 202 (i.e., according to Figure 3 304-1) will result in an inverted image compared to a horizontal orientation of 0°. In some embodiments, the horizontal orientation 304-1 may be adjusted by physically rotating the imaging device 202, while in other embodiments, such an adjustment may be made without physically moving or adjusting the physical orientation of the imaging device 202. For example, the horizontal adjustment may be made by digitally manipulating or processing the image data captured by the imaging device 202.

[0070] The planar orientation 304-3 of the viewpoint 300 is related to the position of the imaging device relative to the plane of the body 302 being captured. Figure 3The coordinate system shown is parallel to the xy plane) by translating the imaging device 202 left, right, up, or down. As the plane orientation 304-3 is adjusted, the image of the body scrolls so that a different portion of the body is depicted by the image data after the plane orientation 304-3 is adjusted than before.

[0071] As described above, certain embodiments of the imaging device 202 may be coupled, flexible, or may otherwise have the ability to articulate to capture images in directions away from the longitudinal axis of the imaging device 202. Furthermore, even if a particular embodiment of the imaging device 202 is rigid and straight, angled view settings (e.g., 30° angled views up or down, etc.) may be used to similarly allow the imaging device to capture images in directions other than straight ahead. Thus, for any of these embodiments of the imaging device 202, the yaw orientation 304-4 that affects the heading of the imaging device along the normal axis (i.e., the y-axis of the coordinate system shown) and the pitch orientation 304-5 that affects the tilt of the imaging device along the lateral axis (i.e., the x-axis of the coordinate system shown) may also be adjustable.

[0072] While various orientations 304 have been explicitly described, it should be understood that various other aspects of how the imaging device 202 captures images of the body 302 may similarly be included as adjustable aspects of the orientation of the imaging device 202 in certain embodiments.

[0073] Based on viewpoint 300, imaging device 202 is shown capturing a particular field of view 306 of body 302. It should be appreciated that field of view 306 may change in various ways (e.g., move left or right, become larger or smaller, etc.) as various orientations 304 of viewpoint 300 of imaging device 202 are adjusted.

[0074] Figure 4 An exemplary viewpoint optimization system 400 ("system 400") is shown for facilitating optimization of an imaging device viewpoint (e.g., viewpoint 300 of imaging device 202) during an operating session of a computer-assisted operating system (e.g., operating system 100). Figure 4 As shown, system 400 may include, but is not limited to, a storage facility 402 and a processing facility 404 selectively and communicatively coupled to each other. Facilities 402 and 404 may each include or be implemented by hardware and / or software components (e.g., processors, memories, communication interfaces, instructions stored in memories for execution by processors, etc.). In some examples, facilities 402 and 404 may be distributed among multiple devices and / or multiple locations that may serve a particular implementation.

[0075] As described above, in some embodiments, the system 400 may be implemented by, integrated with, or incorporated into the operating system 100 (e.g., through integration with the auxiliary system 106, the user control system 104, etc.). In other embodiments, the system 400 may be incorporated into a computing device that is separate from (but communicatively coupled to) the operating system 100. Each of the facilities 402 and 404 will now be described in more detail.

[0076] The storage facility 402 may maintain (e.g., store) executable data for use by the processing facility 404 to perform any of the functions described herein. For example, the storage facility 402 may store instructions 406 that may be executed by the processing facility 404 to perform any of the functions described herein. The instructions 406 may be implemented by any suitable application, software, code, and / or other executable data instances. The storage facility 402 may also maintain any data received, generated, managed, used, and / or transmitted by the processing facility 404.

[0077] The processing facility 404 can be configured to perform (e.g., execute instructions 406 stored in the storage facility 402 to perform) various processing functions associated with optimizing (or facilitating optimization) an imaging device viewpoint during an operating session of the computer-assisted operating system. As used herein, an operating session can refer to any session during which a user (e.g., clinician 110-1) directs a computer-assisted operating system (e.g., operating system 100) to perform one or more operations on any type of body described herein. For example, certain operating sessions can be clinical sessions involving surgical procedures performed on human or animal patients, imaging or exploratory procedures performed before or after such surgical procedures, and the like. In other examples, an operating session can be a non-clinical session involving a training procedure performed on a cadaver or artificial body or involving an extended reality (e.g., virtual or augmented reality) body in an extended reality environment.

[0078] The processing facility 404 can facilitate the optimization of the imaging device viewpoint in any suitable manner. For example, in one example, the processing facility 404 can identify a condition associated with an operation session during which the operating system 100 performs multiple operations relative to the body and the imaging device 202 (which can be included in the operating system 100) provides images for display on a display device during the operation session. For example, the imaging device can provide an image of the body from a first viewpoint. As will be described in more detail below, the identified condition can be any suitable condition related to the user who guides the operating system 100 to perform the multiple operations, the operation itself, etc. Based on the identified condition, the processing facility 404 can define a second viewpoint different from the first viewpoint for the imaging device. In particular, for operations included in the multiple operations, the second viewpoint can be defined as more optimized than the first viewpoint. The processing facility 404 can then direct the display device to display an indication of the second viewpoint in any manner described herein.

[0079] As another more specific example, the processing facility 404 may determine during an operation session such as described in the above example that a user is using a first wrist posture associated with a first viewpoint to guide a computer-assisted operating system to perform an operation included in a plurality of operations. For example, the processing facility 404 may determine that a user is using a suboptimal wrist posture to perform an operation such as driving a needle through tissue to perform suturing. The processing facility 404 may define a second viewpoint associated with a second wrist posture that is more optimized than the first wrist posture to guide the performance of the surgery. For example, the processing facility 404 may define a second viewpoint associated with a wrist posture that is more neutral for a particular direction in which a needle will be driven through tissue (e.g., requiring the user's wrist to bend or reach less awkwardly). The second viewpoint may be defined in any suitable manner, such as, for example, having a horizontal orientation that is different from the horizontal orientation of the first viewpoint (e.g., to achieve a more neutral wrist posture).

[0080] When the display device displays an image of the body from a first viewpoint, processing facility 404 may direct the display device to integrate a reticle overlay graphic indicating a horizontal orientation of a second viewpoint with the image of the body displayed from the first viewpoint. Then, in response to the integration of the reticle overlay graphic indicating a horizontal orientation of the second viewpoint, processing facility 404 may receive user input indicating a user selection to view the image of the body from the second viewpoint instead of viewing the image of the body from the first viewpoint. In response to the user input, processing facility 404 may direct the display device to switch from displaying the image of the body from the first viewpoint to displaying the image of the body from the second viewpoint.

[0081] To illustrate, Figure 5AAn exemplary operating session 500 is shown during which operating system 100 (or, in other examples, another computer-assisted operating system similar to operating system 100) performs a plurality of operations with respect to body 302 while imaging device 202 (which may be included within operating system 100) captures images of body 302 from different exemplary viewpoints 300 (e.g., viewpoints 300-1 and 300-2). More specifically, Figure 5A The particular portion of body 302 into which an incision has been made and the relative position of the distal end of imaging device 202 relative to the incision are depicted from a side perspective view showing the location of imaging device 202. As shown, various instruments 502, 504, and 506 are used to perform one or more operations on body 302 at the operating site. For example, instruments 502 and 504 may be used primarily to manipulate tissue and / or tools to facilitate the operation being performed, while instrument 506 may be used to hold certain portions of tissue out of the way or otherwise facilitate the performance of the operation.

[0082] exist Figure 5A , the distal end of the imaging device 202 is depicted at a first moment in time (depicted using solid lines) and at a second, later moment in time (depicted using dashed lines). As shown, the imaging device 202 has a first viewpoint 300-1 at the first moment in time and a second viewpoint 300-2 at the second moment in time. The small arrows depicted behind each viewpoint 300-1 and 300-2 indicate the horizontal orientation of the viewpoint relative to a three-dimensional ("3D") coordinate system shown as having X, Y, and Z dimensions (i.e., how the imaging device 202 is rotated along the longitudinal axis). More specifically, the horizontal orientation of viewpoint 300-1 is shown as having a positive X dimension pointing upward, while the horizontal orientation of viewpoint 300-2 is shown as having a positive Y dimension pointing upward. As viewpoints 300-1 and 300-2 differ in their respective horizontal orientations, the zoom orientation from viewpoint 300-1 to 300-2 is also shown as being adjusted because viewpoint 300-2 is closer to (i.e., optically magnified) the tissue of the body 302.

[0083] Figure 5B An exemplary display device is illustrated on which images captured from viewpoints 300-1 and 300-2 during an operational session 500 are displayed. Specifically, when the viewpoint of imaging device 202 has been adjusted (i.e., zoomed in and rotated 90 degrees), image 508-1 captured by imaging device 202 from viewpoint 300-1 is displayed on display device 510 at a first moment in time, while image 508-2 captured by imaging device 202 from viewpoint 300-2 is displayed on display device 510 at a second moment in time. To help clarify what is depicted in images 508-1 and 508-2 and how they differ from one another, it will be noted that Figure 5A The same coordinate system included in Figure 5B202. In both cases, the Z dimension is illustrated by a dotted notation to indicate that the z-axis is to be understood as straight out of the imaging device screen (i.e., parallel to the longitudinal axis of the imaging device 202 in this example). However, while the X dimension is shown as pointing upward in image 508-1, the 90° adjustment in horizontal orientation from viewpoint 300-1 to viewpoint 300-2 is shown as resulting in the Y dimension pointing upward in image 508-2.

[0084] exist Figure 5B , the display device 510 is shown as a rectangular monoscopic display screen. For example, refer again to the above Figure 1 The operating system 100 described herein, the display monitor 114 of the auxiliary system 106 may implement such a display device 510 in certain embodiments. In the same or other embodiments, it will be understood that the display device 510 may be additionally or alternatively implemented by other types of display screens. For example, the display device 510 may be implemented by a stereoscopic display screen of the user control system 104, and the stereoscopic display screen of the user control system 104 is viewed by the clinician 110-1 as the clinician 110-1 guides the manipulation system 102 to perform operations on the body 302.

[0085] As described above, switching from a less optimal viewpoint to a more optimal viewpoint may provide benefits that are even greater than the significant benefit of improving the view of the operating area where the operation is being performed. For example, as described above, when a suboptimal viewpoint is associated with a relatively unnatural, uncomfortable, or inefficient wrist posture, a more natural, comfortable, and efficient wrist posture may be achieved with a more optimal viewpoint.

[0086] To illustrate, Figure 5C Exemplary wrist gestures 512-1 and 512-2 are shown for a user (e.g., clinician 110-1, etc.) to perform operations while viewing images from viewpoints 300-1 and 300-2, respectively. For each of wrist gestures 512-1 and 512-2, the left wrist and the right wrist are posed (i.e., positioned, oriented, etc.) to mimic the gestures of instruments 502 and 504, respectively. Once operating system 100 is in normal operating mode, instrument 502 may thus be configured to follow and be guided by the user's left hand and wrist, while instrument 504 may be configured to follow and be guided by the user's right hand and wrist. However, as Figure 5C As shown, the wrist pose required to direct the instruments when they are posed in image 508-1 is significantly different than the wrist pose required to direct the instruments when they are posed in image 508-2.

[0087] Specifically, as shown, for some tasks, the wrist posture 512-1 associated with the viewpoint 300-1 and the apparatus 502 and 504 posing in the image 508-1 may be a relatively awkward, uncomfortable and inefficient wrist posture. For example, the left arm is awkwardly retracted backwards, the left wrist is bent backwards to a great extent, and the right arm is extended forward, and the right wrist is bent forward to a somewhat unnatural degree. Although this wrist posture may be acceptable or even desirable for performing certain operations, it may be suboptimal and undesirable for performing other operations. Therefore, the system 400 can define the viewpoint 300-2 and guide the display device 510 to display an indication of the viewpoint 300-2 by displaying the image 508-2.

[0088] As shown, in this way, system 400 can allow a user to see a more detailed view of the operating area shown in image 508-2, as well as adopt a more comfortable and optimized wrist posture. Specifically, as shown, for certain operations, wrist posture 512-2 associated with viewpoint 300-2 and instruments 502 and 504 posed in image 508-2 can be a more optimized (e.g., more natural, more comfortable, more efficient, etc.) wrist posture than wrist posture 512-1. Therefore, for such operations, viewpoint 300-2 may be a more optimized viewpoint 300-1.

[0089] Although FIG. 5A to FIG. 5C Viewpoint adjustments including changes to both the horizontal orientation and the zoom orientation are illustrated, but it should be understood that the system 400 can define the second viewpoint for any suitable reason, in any suitable manner and using any suitable orientation described herein (e.g., any orientation 304). As an example, the second viewpoint can be specifically defined to facilitate intuitive and natural motion of movement associated with an operation being performed or an operation expected to be performed next. For example, if the operation involves driving a needle through tissue to suture two portions of tissue together, a more optimized viewpoint can be defined to allow the needle to be driven at an angle that the user's wrist and hand are capable of high intensity and control delivery while the operation is being performed. As another example, the second viewpoint can be defined to achieve an appropriate zoom level for a particular operation, being zoomed in close enough to allow the user to utilize good depth perception of the tissue and objects being operated on while also being zoomed out far enough to allow the user to view an appropriate amount of context around the area being operated on.

[0090] As yet another example, system 400 may define a viewpoint having a horizontal orientation that allows a user to switch relatively easily between controlling one instrument and controlling another instrument. For example, if a user uses his or her right hand to alternately control instruments 504 and 506 (e.g., switching back and forth between which instrument follows the right hand), it may be inefficient or burdensome to continually make the significant wrist gesture changes required to direct each of these instruments in a distinctly different gesture. Therefore, system 400 may define a more optimal viewpoint as one that takes into account the gestures of both instruments 504 and 506 and the corresponding wrist gestures required for control.

[0091] To better illustrate these and other examples of how system 400 can facilitate the performance of operations by helping to optimize the imaging device viewpoint, Figures 6 to 11 Each illustrates that the display device 510 displays an image from a first exemplary viewpoint that is suboptimal for a particular operation, and then displays an image from a second exemplary viewpoint that is more optimal for the particular operation. Specifically, Figures 6 to 11 Each of the displays 510 on the left side of the figure shows an image from a first (suboptimal) viewpoint, while the display 510 on the right side of the figure shows an image from a second (more optimal) viewpoint. In addition, to help illustrate the adjustments to the horizontal orientation, pitch orientation, yaw orientation, etc. made between the first viewpoint and the second viewpoint in some examples, Figures 6 to 11 Each depiction in the image includes a 3D coordinate system having X, Y, and Z coordinates to be understood as being relative to the body 302 and the instrument depicted in the image, and thus remains consistent between the first viewpoint and the second viewpoint.

[0092] Figure 6 The display device 510 is illustrated as displaying an image 600-1 from a first viewpoint that would be understood as suboptimal, and subsequently displaying an image 600-2 from a second viewpoint that has a different horizontal orientation than the first viewpoint and would be understood as more optimized than the first viewpoint. As described above, the system 400 can define the second viewpoint based on an identified condition associated with the operational session, and the identified condition can relate to a specific operation being performed (e.g., driving a needle is an exemplary operation that has been described). The system 400 can determine the operation being performed or the operation to be performed in any suitable manner. For example, the system 400 can receive manual input from a user indicating an operation being performed or to be performed, or the system 400 can be configured to automatically identify an operation based on the action being performed by the user, other operations previously performed as part of a sequence, and the like.

[0093] Based on a specific operation that has been determined to be in progress or about to be performed during the operation session, the system 400 can analyze the user's wrist posture and define a second viewpoint accordingly. More specifically, the system 400 can identify a condition associated with the operation session by first determining that the first wrist posture 602-1 associated with the first viewpoint is being used to guide the operating system 100 to perform a specific operation, and, secondly, determining that the second wrist posture 602-2 associated with a viewpoint having a horizontal orientation different from the horizontal orientation of the first viewpoint will be more optimized than the wrist posture 602-1 to guide the execution of the specific operation. Based on identifying such a condition that the user is using a suboptimal wrist posture 602-1 instead of a more optimal wrist posture 602-2, the system 400 can define the second viewpoint based on the identified condition by defining the second viewpoint as a viewpoint associated with the wrist posture 602-2. In other words, the system 400 can define the second viewpoint as a viewpoint that allows the user to take a more optimal wrist posture 602-2.

[0094] In other examples, the system 400 may identify the condition associated with the operation session in other suitable ways, or the condition may correspond to other suitable factors associated with the operation session described herein. For example, in some embodiments, the system 400 may not only consider the user's current wrist posture when evaluating the condition of the operation session, but may also further consider the current spatial position of the user's hands relative to each other, the co-location state of the user's hands relative to the controlled instrument, etc. In embodiments of the computer-assisted operating system, the common orientation of the user's hands and the instrument may be required (i.e., so that the wrist posture, finger positioning, etc. of the hand are aligned with the instrument orientation before the instrument is controlled by the user). However, in at least some of these embodiments, the co-location of the hand and the instrument may not be required (i.e., even if the instruments are close to each other in 3D space, the user's hands may be far away from each other in 3D space, and vice versa). Therefore, the system 400 may not only consider the user's comfort and convenience in terms of wrist posture and hand orientation when determining a more optimized viewpoint, but also consider the comfort and convenience of each of the user's hands relative to each other in terms of spatial position and range. In other examples, as will be described in more detail below, the identified conditions may relate to the specific operation being performed, the identity or known habits of the user (e.g., previously observed performance advantages and disadvantages, etc.), or any other conditions associated with the operational session that may be useful for a particular implementation.

[0095] Once system 400 has defined the second, more optimized viewpoint, system 400 may direct display device 510 to display an indication of the second viewpoint in any manner that is useful for a particular implementation.

[0096] As an example, system 400 may direct display device 510 to display an indication of a second viewpoint by directing display device 510 to display a graphical object indicating a second viewpoint while display device 510 is displaying image 600-1 of body 302 from a first viewpoint. Specifically, the graphical object may be displayed as an overlay graphic integrated with the image of body 302 displayed from the first viewpoint. For illustration, Figure 6 A reticle object 604 integrated with the image 600-1 of the body 302 is shown. Reticle object 604 may be an example of a type of graphical object that may be used to indicate a second viewpoint, and other such examples will be described and illustrated below. As shown, reticle object 604 indicates the horizontal orientation of the second viewpoint. Specifically, in this example, a solid line representing the horizontal orientation of the first viewpoint is shown together with a dotted line representing the horizontal orientation of the second viewpoint. An arrow points from the solid line to the dotted line in reticle object 604 to indicate a counterclockwise adjustment of the horizontal orientation of the first viewpoint that will result in an adjustment of the second viewpoint. It should be understood that reticle object 604 is exemplary only, and in other examples, reticle objects of other types or styles that may be used for a particular embodiment (e.g., a semi-transparent crosshair, etc.) may be used to indicate the horizontal orientation of the second viewpoint.

[0097] In other examples, system 400 may direct display device 510 to display an indication of the second viewpoint in other ways. For example, rather than directing the display of a graphical object such as graticule object 604, system 400 may direct display device 510 to display image 600-2 together with image 600-1. For example, system 400 may direct display device 510 to display image 600-2 in a picture-in-picture manner overlaying image 600-1 in a semi-transparent manner or in any other suitable manner on a different presentation layer (e.g., covering or underlaying the display of image 600-1). In addition, as will be described in more detail below, system 400 may direct display device 510 to automatically or semi-automatically stop displaying image 600-1 and display image 600-2 in place of image 600-1, thereby automatically adjusting imaging device 202 to capture image 600-2 from the second viewpoint, rather than suggesting to the user how the user can manually adjust imaging device 202 to capture image 600-2 from the second viewpoint.

[0098] In addition to or in lieu of identifying conditions related to a particular operation being performed and conditions related to an associated wrist posture for performing the operation, the system 400 may also identify conditions related to the relative position of the imaging device 202 to the body 302 and / or the instrument 502 (and other instruments, such as those not in the Figures 6 to 11For example, the conditions based on which the second viewpoint identified and defined by the system 400 is based may be conditions related to the relative zoom orientation of the imaging device 202, the relative planar orientation of the imaging device 202, etc. Figure 7 and Figure 8 Each illustrates an example of an optimization viewpoint defined based on these types of conditions.

[0099] Specifically, Figure 7 The display device 510 is illustrated as displaying an image 700-1 from a first viewpoint that would be understood as suboptimal, and subsequently displaying an image 700-2 from a second viewpoint having a different zoom orientation than the first viewpoint and that would be understood as more optimized for a particular operation than the first viewpoint. In this example, the system 400 may identify a condition by 1) determining that the display device 510, when displaying the image 700-1, depicts an active image of the performance of an operation on the body 302 at a first level of detail, and 2) determining that displaying an image of the body from a viewpoint having a zoom orientation different from the zoom orientation of the first viewpoint will result in the active image being depicted at a second level of detail that is more optimized for performing the operation than the first level of detail. For example, the system 400 may determine that the first viewpoint displaying the image 700-1 is zoomed out too far to provide an optimized level of detail, depth perception, instrument sensitivity, etc. for the particular operation being performed, and, as a result, may determine that a more optimized viewpoint would be a viewpoint having a zoom orientation that is further zoomed in to provide a greater level of detail. As another example, system 400 may determine that a first viewpoint is zoomed in too closely to provide an appropriate level of context around the operations being performed, and as a result, may determine that a more optimal viewpoint would have a zoom orientation that is zoomed out to provide a lower level of detail.

[0100] In either case, system 400 defining the second viewpoint based on the identified condition may include defining the viewpoint at a more optimized zoom orientation (e.g., further zooming in or out may be required depending on the situation). Specifically, system 400 may define the second viewpoint as a viewpoint having a zoom orientation different from the zoom orientation of the first viewpoint, so that the active image is described at a second level of detail that is more optimized for performing the operation.

[0101] exist Figure 7In the example shown, image 700-1 depicts an active image depicting the performance of a procedure on body 302 at a relatively low first level of detail. As used herein, an "active image" depicting the performance of an operation on a body refers to an image depicting the area where the operation is being performed and the user is focused on (rather than other areas immediately adjacent to the area of ​​user focus). Thus, the active image at any given moment during the performance of the operation may include an image of the body portion where the operation is being performed, as well as images of instruments and / or other objects used to perform the operation, while not including images of other parts of the body and / or other instrument objects that are not particularly related to the operation being performed.

[0102] Image 700-1, shown at a relatively low first level of detail, may not be optimal for performing certain operations. For example, a user's depth perception of tissue and / or objects within the active image may be suboptimal from a zoom orientation so far from body 302, and the user may generally not be able to perceive the optimal amount of detail to perform the operation in the most effective and efficient manner. Therefore, system 400 can define a second viewpoint in which the zoom orientation is adjusted to provide a relatively high second level of detail, such as shown in image 700-2. In this way, the user can enjoy visibility, depth perception, and understanding of what is happening at the operating site. In addition, in some examples, the instruments used to perform the operation (e.g., instrument 502, etc.) can become more sensitive through more detailed views, which can enable the user to more easily perform complex movements and detailed work.

[0103] As mentioned above Figure 6 As described above, once the system 400 defines the second viewpoint, the system 400 can direct the display device 510 to display an indication of the second viewpoint in various ways. Figure 7 As shown, one way to guide display device 510 to display an indication of the second viewpoint is to guide display device 510 to display a graphic object indicating the second viewpoint while display device 510 displays image 700-1. Specifically, as shown in the figure, the graphic object indicating the second viewpoint may include a bounding box 702 indicating at least one of a zoom orientation and a plane orientation of the second viewpoint. As with the above-mentioned graticule object 604, bounding box 702 may be displayed as an overlay graphic integrated with image 700-1. Based on bounding box 702, a user may manually adjust the orientation parameters of imaging device 202 to move to an optimized second viewpoint and begin receiving image 700-2, or in some examples may automatically or semi-automatically adjust to the optimized viewpoint in any manner described herein.

[0104] Figure 8The display device 510 is illustrated as displaying an image 800-1 from a first viewpoint that would be understood as suboptimal, and subsequently displaying an image 800-2 from a second viewpoint that has a different planar orientation than the first viewpoint and would be understood as more optimized than the first viewpoint. In this example, the system 400 may identify a condition by 1) determining that an active image depicting the performance of an operation on the body 302 is depicted at a first portion of the field of view by the display device 510 when displaying the image 800-1, and 2) determining that displaying an image of the body from a viewpoint having a planar orientation different from the planar orientation of the first viewpoint will result in the active image being depicted at a second portion of the field of view that is more optimized to perform the operation than the first portion of the field of view. For example, the system 400 may determine that the first viewpoint displaying the image 800-1 displays the active image at a corner or side of the field of view presented by the display device 510 rather than at a more optimized portion of the field of view (such as at the center). As a result, the system 400 may determine that the more optimized viewpoint will be a viewpoint that displays the active image in a portion of the field of view that is more centered. Thus, system 400 defining a second viewpoint based on the identified condition may include defining the viewpoint to move the active image closer to the center of the field of view. Specifically, system 400 may define the second viewpoint as a viewpoint having a planar orientation different from the planar orientation of the first viewpoint, such that the active image is depicted at a second portion of the field of view that is more optimized for performing the operation.

[0105] exist Figure 8 In the example shown, image 800-1 depicts an active image depicting the performance of an operation on body 302 at a first portion of the field of view of display device 510 near a corner of the field of view. Figure 8 Of the different portions 802-1 through 802-9 of the field of view 802 shown in the top field of view key, the active image depicting the performance of the operation in image 800-1 is shown as being displayed mostly or completely within portion 802-1 of the field of view 802 (i.e., in the upper left corner of the field of view). This positioning of the active image may be suboptimal for performing certain operations, as it may be desirable to have the active image near the center of the field of view (e.g., in or near portion 802-5). Therefore, the system 400 can define a second viewpoint in which the plane orientation is adjusted to display the active image closer to the center of the field of view 802, such as centered about portion 802-5.

[0106] As mentioned above Figure 6 and Figure 7As described above, once system 400 defines the second viewpoint, system 400 can direct display device 510 to display an indication of the second viewpoint in a variety of ways. For example, a fully automatic or semi-automatic change from image 800-1 to image 800-2 can be used, or an overlay graphic can be displayed to allow a user to manually adjust the parameters of imaging device 202 to redirect to the second viewpoint associated with image 800-2. Figure 6 The graticule object 604 described above is very effective in indicating horizontal orientation adjustments. Figure 7 The depicted bounding box 702 is also effective for indicating zoom orientation adjustments (perhaps with relatively small planar orientation adjustments), which these types of overlay objects may not be able to do so effectively. Figure 8 The plane orientation is shown adjusted to move from a suboptimal viewpoint of image 800-1 to a more optimal viewpoint of image 800-2. Additionally, pitch orientation adjustments, yaw orientation adjustments, and / or other orientation adjustments to imaging device 202 may similarly not be particularly well indicated by a two-dimensional overlay object such as reticle object 604 or bounding box 702.

[0107] Thus, in some examples, as the imaging device 202 provides images of the body 302 from different viewpoints, the graphical overlay object indicating the second viewpoint may include a first 3D shape anchored to the field of view of the imaging device 202 and a second 3D shape indicating at least one of the zoom orientation, plane orientation, horizontal orientation, pitch orientation, and yaw orientation of the second viewpoint. For example, while one 3D shape may appear to be anchored to the imaging device 202 itself (e.g., floating in front of the imaging device as the imaging device is zoomed, translated, articulated, etc.), a target 3D shape may be anchored to the body 302 such that, if the first 3D shape matches the target 3D shape, the imaging device 202 will be adjusted to capture images from the second viewpoint (i.e., the more optimized target viewpoint).

[0108] To illustrate, Fig. 9 The display device 510 is shown displaying an image 900-1 from a first viewpoint which would be understood to be suboptimal and subsequently displaying an image 900-2 from a second viewpoint which has an orientation that differs in multiple respects from the orientation of the first viewpoint and would be understood to be more optimal than the first viewpoint. Fig. 9 As shown, the first 3D shape overlay 902 is anchored to the field of view of the imaging device 202 (e.g., in this example, in the lower left corner of the field of view). In this example, the 3D shape overlay 902 is a 3D pyramid shape viewed directly from the top. The face on the right side of the 3D pyramid shape is shown as shaded to illustrate clarity and orientation. In addition, Fig. 9A second 3D shape overlay 904 is shown anchored to the image being displayed and indicating a second viewpoint that has been defined. Specifically, as shown, 3D shape overlay 904 is smaller than 3D shape overlay 902 (e.g., indicating that the scaled orientation will be enlarged in order to align and / or match the shapes), is rotated relative to 3D shape overlay 902 along each of the X, Y, and Z axes (e.g., indicating that the horizontal orientation, pitch orientation, and yaw orientation will be adjusted in order to align the shapes), and is depicted on a different portion of the field of view of display device 510 (e.g., indicating that the planar orientation will be adjusted in order to align the shapes).

[0109] By adjusting each of the different aspects of the orientation of imaging device 202, 3D shape overlay 902 may be aligned or matched with 3D shape overlay 904. When such alignment is achieved, imaging device 202 will be posed to capture image 900-2 from a second, more optimized viewpoint defined by system 400. While each of the zoom, plane, horizontal, pitch, and yaw orientations may be adjusted in this example to move 3D shape overlay 902 to match 3D shape overlay 904, it will be appreciated that in other examples, any single aspect of the orientation of imaging device 202 or any combination of these or other aspects of the orientation of imaging device 202 may be adjusted to align the 3D shape overlay.

[0110] As described above, in some examples, the system 400 can direct the display device 510 to display an indication of the second viewpoint in a manner that does not involve overlaying a graphical object onto an image captured from the first viewpoint. For example, certain embodiments of the system 400 can be configured to direct the display device 510 to indicate the second viewpoint by facilitating a switch from displaying an image from the first viewpoint to displaying an image from the second viewpoint in an automatic or semi-automatic manner. In some embodiments, for example, directing the display device 510 to display an indication of the second viewpoint may include: 1) directing the display device 510 to present (e.g., when the display device 510 displays an image of the body from the first viewpoint) an indication of a defined second viewpoint; 2) receiving (e.g., in response to presentation of an indication that the second viewpoint has been defined) user input indicating that a user of the system has selected to view an image of the body from the second viewpoint rather than from the first viewpoint; and 3) in response to the user input, directing the display device 510 to switch from displaying an image of the body from the first viewpoint to displaying an image of the body from the second viewpoint.

[0111] To illustrate, Fig.10 The display device 510 is shown from above Fig. 9 The same first viewpoint and second viewpoint (i.e., the second less optimal first viewpoint and the second more optimal viewpoint) are shown. However, rather than facilitating manual switching from the first viewpoint to the second viewpoint by means of a graphical overlay object such as 3D shape overlays 902 and 904, Fig.10 An indicator 1002 indicating that the system 400 has defined a second more optimized viewpoint (than the viewpoint currently used) is illustrated. The indicator 1002 can take any form that can be used for a particular embodiment. For example, the indicator 1002 can be implemented as a button, a link, a notification, an alarm, etc. Therefore, a user input indicating that the user chooses to view an image of the body from a second viewpoint rather than a first viewpoint can be provided in any suitable manner, such as by a foot pedal or button press, a gesture, a voice command, or any other suitable form of user input. Once the user input is received, the system 400 can automatically adjust the orientation parameters of the imaging device 202 to move to the second more optimized viewpoint associated with the image 1000-2. In some examples, the system 400 can ensure that the viewpoint is not changed when the operating system 100 is in an operating mode (i.e., a mode in which the instrument follows or imitates the user's hand movements), but only when the operating system 100 is in an imaging adjustment mode.

[0112] In other examples, the change from one viewpoint to another viewpoint may be performed in a fully automatic manner so that no specific user input indicating the selection of the optimized viewpoint is required. In particular, as new users gain more experience with the system, it may be helpful for novice users to practice performing operations using an apparatus of operating system 100 for a period of time using an automatically selected optimized viewpoint before learning how to perform manual or assisted viewpoint selection. In these examples, directing display device 510 to display the indication of the second viewpoint may include directing display device 510 to automatically switch from displaying an image of the body from the first viewpoint to displaying an image of the body from the second viewpoint in response to the definition of the second viewpoint.

[0113] To illustrate, Fig.11 The display device 510 is shown from above Fig. 9 and Fig.10 The same first viewpoint and second viewpoint (i.e., the second best first viewpoint and the second better viewpoint) as shown in FIG. Fig.11 In the example of , neither an overlay object facilitating manual parameter adjustment nor an indicator facilitating semi-automatic parameter adjustment is included. Instead, it should be understood that once the second viewpoint is defined, the system 400 can direct the display device 510 to automatically switch from displaying image 1100-1 to 1100-2. In some examples, the automatic adjustment from the first viewpoint to the second viewpoint can be performed only when the operating system 100 is in the imaging adjustment mode, rather than when the operating system 100 is in the operating mode. For example, a user can operate in the operating mode, then step on the foot pedal to perform automatic imaging adjustments, then readjust his or her wrist posture, and return to the operating mode to perform the next operation in the program using the automatically selected viewpoint.

[0114] In other examples, the system 400 may otherwise facilitate, incentivize, or encourage the user to switch to a more optimal viewpoint. For example, if the system 400 determines that the first viewpoint is highly suboptimal or the second viewpoint is significantly more optimal than the current viewpoint, the system 400 may automatically direct the operating system 100 to switch from the operational mode to the imaging adjustment mode and not allow the operating system 100 to switch back to the operational mode until a more optimal viewpoint is selected. For example, the system 400 may ensure that one or more aspects of the orientation of the imaging device 202 meet certain thresholds before allowing the user to continue to perform operations on the body in the operational mode.

[0115] As another example, the system 400 can promote or incentivize the use of optimized viewpoints by providing an optimization metric or optimization score for each aspect of the orientation of the imaging device 202. For example, during the performance of a particular operation, the zoom orientation can be determined to be within 10% of the optimal value, while the horizontal orientation can be determined to be more than 30° away from the optimal value. These metrics can be used to score users during training exercises, can be stored in a database for later use in analyzing the strengths and weaknesses of a user's particular viewpoint selections (as described in more detail below), or otherwise used to promote viewpoint optimization in any manner that is useful for a particular implementation.

[0116] As yet another example, the system 400 may be configured to automatically adjust the viewpoint based on the detected wrist posture, rather than other ways that may be done conventionally. Specifically, in some conventional embodiments, a user may select a viewpoint when the operating system 100 is in an imaging adjustment mode, and then, in order to switch to the operating mode, the user may need to make his or her wrist posture conform to the wrist posture required for the selected viewpoint and perform a gesture (e.g., pinching, etc.) to enter the operating mode in which the device will follow the user's movement. In some embodiments, the operating system 100 may be configured to facilitate this process by physically moving the main control held by the user to the appropriate position of the specific viewpoint after the viewpoint has been selected. Therefore, instead of conforming the wrist posture to the selected viewpoint in these conventional ways, some embodiments of the system 400 may allow the user to select a wrist posture for one or both of his or her wrists, and then a viewpoint that fully conforms to the wrist posture or at least conforms to the wrist posture to a certain extent may be automatically defined. Once the parameters of the imaging device are set to capture images from this automatically defined viewpoint, a notification may inform the user that the optimal viewpoint is available, and through a gesture (e.g., pinching), the user may continue to operate in the operating mode.

[0117] In the description Figures 7 to 11In the present invention, various examples have been disclosed regarding how optimized viewpoints, once defined by a viewpoint optimization system such as system 400, can facilitate the performance of certain operations. Various methods for assisting a user in switching from a suboptimal viewpoint to a more optimal viewpoint are also described. To achieve these objectives, system 400 can be configured to define optimized viewpoints in any manner and using any information received from any source that may be useful for a particular implementation.

[0118] To illustrate, Fig.12 Exemplary entities 1202 through 1208 are shown that can provide input data to system 400 to allow system 400 to facilitate optimization of the imaging device viewpoint in the manner described herein. Fig.12 As shown, user 1202, expert advisor 1204, automatic viewpoint recommendation algorithm 1206, and viewpoint selection data store 1208 may each provide input to certain embodiments of system 400. Each of entities 1202 through 1208 will now be described in greater detail, along with descriptions of various ways in which these entities may provide data to system 400 to enable system 400 to define a more optimized viewpoint than the currently selected viewpoint.

[0119] User 1202 may perform any of the actions described herein as being performed by a user. In some examples, user 1202 may be a clinician (e.g., clinician 110-1), such as a surgeon who performs a medical procedure or is trained to use operating system 100. In other examples, user 1202 may be another person on a surgical team, other than a surgeon or another user of operating system 100 or another similar computer-assisted operating system.

[0120] The expert advisor 1204 may be a different person than the user 1202 who also provides input to the system 400. The expert advisor 1204 may be an experienced user of the operating system 100 who is believed to have good insight into what makes one viewpoint suboptimal for a particular surgery and another viewpoint more optimal for a particular surgery. For example, if the user 1202 is a novice surgeon being trained to use the operating system 100, the expert advisor 1204 may be a surgeon with more experience using the operating system 100, an instructor training the user on the system 1202, etc. In some cases (e.g., during a training session), the user 1202 may operate the operating system 100 and, when it is desired to switch from one viewpoint to another, may place the operating system 100 into a "training mode," during which the user 1202 may compare his or her currently selected viewpoint with the viewpoint recommended by the expert advisor 1204. To this end, defining a second viewpoint that is more optimized than the first viewpoint for an operation in the plurality of operations may include identifying (e.g., based on input provided in real time by the expert advisor 1204 during the operation session) a recommended viewpoint for the operation and defining the second viewpoint based on the recommended viewpoint.

[0121] The automatic viewpoint recommendation algorithm 1206 can perform similar functions as the expert advisor 1204, but may not require human effort because it is implemented as a computer algorithm running on the system 400, the operating system 100, or another suitable computing system. More specifically, the automatic viewpoint recommendation algorithm 1206 can be configured to generate operation-specific viewpoint recommendations in real time during an operation session. In some examples using the automatic viewpoint recommendation algorithm 1206, the definition of a second viewpoint that is more optimized than the first viewpoint for an operation in the plurality of operations can therefore include identifying (e.g., based on the automatic viewpoint recommendation algorithm 1206) a recommended viewpoint for the operation, and defining the second viewpoint based on the recommended viewpoint.

[0122] The viewpoint selection data store 1208 may be implemented as any suitable type of data store (e.g., a database, a data storage facility, etc.) that is configured to track and store user-specific and / or expert-specific viewpoint selection data. For example, as described above, as the user 1202 performs operations and corresponding viewpoint selection tasks over a period of time (e.g., over the course of a training program, over the course of his or her career, etc.), viewpoint selection data representing effective and less effective viewpoint selection decisions made by the user 1202 may be stored in the viewpoint selection data store 1208. In this way, the data stored in the viewpoint selection data store 1208 may indicate viewpoint selection strengths and weaknesses that the user 1202 is known to have. For example, it may be known that the user 1202 consistently selects an appropriate zoom orientation, but struggles to find the correct horizontal orientation for certain tasks. Thus, in some examples, the system 400 may use data received from the viewpoint selection data store to illustrate the fact that the user 1202 (and not some other user) is performing an operation session, to illustrate the general skill and experience level of the user 1202, to illustrate certain tendencies, strengths, and / or weaknesses of the user 1202, and so on. Specifically, in these examples, identification of conditions associated with the operational session may include determining the identity of user 1202 (i.e., the user who selected the first viewpoint) and accessing (e.g., based on the identity of user 1202) user-specific data representing viewpoint selections performed in the past by user 1202. System 400 may then define a second viewpoint based on the user-specific data representing viewpoint selections performed in the past by user 402. In some examples, automatic viewpoint recommendation algorithm 1206 may be used in conjunction with data received from viewpoint selection data store 1208 to define the second viewpoint.

[0123] In the same or other examples, viewpoint selection data store 1208 may also store various types of viewpoint selection data representing viewpoints recommended by expert advisor 1204 (or other experts) for particular operations. Thus, defining a second viewpoint that is more optimal than a first viewpoint for an operation in the plurality of operations may include accessing data representing viewpoint selections performed in the past by expert advisor 1204, and defining the second viewpoint based on the accessed data representing viewpoint selections performed in the past by expert advisor 1204. System 400 may use this data (e.g., in conjunction with automatic viewpoint recommendations 1206) to determine whether a currently selected viewpoint is suboptimal and whether a historical viewpoint used by an expert is more optimal for the particular operation being performed.

[0124] Fig.13 An exemplary method 1300 is illustrated for facilitating optimization of an imaging device viewpoint during an operation session of a computer-assisted operating system. Fig.13 The diagram illustrates exemplary operations according to one embodiment, but other embodiments may omit, add, reorder, and / or modify Fig.13 Any of the actions shown in . Fig.13 One or more of the operations shown in may be performed by a viewpoint optimization system such as system 400, any components included therein, and / or any implementation thereof.

[0125] In operation 1302, the viewpoint optimization system may identify a condition associated with an operation session. In some examples, during the operation session, the computer-assisted operation system may perform a plurality of operations with respect to a body. Additionally, during the operation session, an imaging device included in the computer-assisted operation system may provide an image of the body from a first viewpoint. For example, the image may be provided for display on a display device during the operation session. Operation 1302 may be performed in any manner described herein.

[0126] In operation 1304, the viewpoint optimization system may define a second viewpoint for the imaging device that is more optimized than the first viewpoint of an operation included in the plurality of operations. For example, the viewpoint optimization system may define the second viewpoint based on the condition identified in operation 1302. Operation 1304 may be performed in any manner described herein.

[0127] In operation 1306, the viewpoint optimization system may direct the display device to display an indication of the second viewpoint defined in operation 1304. Operation 1306 may be performed in any manner described herein.

[0128] In some embodiments, one or more processes described herein may be implemented at least in part as instructions contained in a non-transitory computer-readable medium and executable by one or more computing devices. Typically, a processor (e.g., a microprocessor, etc.) receives instructions from a non-transitory computer-readable medium (e.g., a memory, etc.) and executes those instructions to perform one or more processes (including one or more processes described herein). Any of a variety of known computer-readable media may be used to store and / or transmit such instructions.

[0129] Computer-readable media (also referred to as processor-readable media) include any non-transitory media that participates in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such media can take many forms, including but not limited to non-volatile media and / or volatile media. Non-volatile media can include, for example, optical or magnetic disks and other persistent memories. Volatile media can include, for example, dynamic random access memory (DRAM), which typically constitutes main memory. Common forms of computer-readable media include, for example, disks, hard disks, tapes, any other magnetic media, compact disk read-only memory ("CD-ROM"), digital video disks ("DVD"), any other optical media, random access memory ("RAM"), programmable read-only memory ("PROM"), electrically erasable programmable read-only memory ("EPROM"), FLASH-EEPROM, any other memory magnetic chip or cassette, or any other tangible medium that a computer can read.

[0130] Fig.14 The figure shows an exemplary computing device 1400, which may be specifically configured to perform one or more of the processes described herein. Fig.14 As shown, computing device 1400 may include a communication interface 1402, a processor 1404, a storage device 1406, and an input / output ("I / O") module 1408 communicatively coupled via a communication infrastructure 1410. Fig.14 An exemplary computing device 1400 is shown in FIG. Fig.14 The components shown in the drawings are not intended to be limiting. Additional or alternative components may be used in other embodiments. Fig.14 Components of computing device 1400 shown in .

[0131] Communication interface 1402 may be configured to communicate with one or more computing devices. Examples of communication interface 1402 include, but are not limited to, a wired network interface (such as a network interface card), a wireless network interface (such as a wireless network interface card), a modem, an audio / video connection, and any other suitable interface.

[0132] The processor 1404 generally represents any type or form of processing unit capable of processing data or interpreting, executing and / or directing the execution of one or more of the instructions, processes and / or operations described herein. The processor 1404 may direct the execution of operations according to one or more applications 1412 or other computer-executable instructions such as may be stored in the storage device 1406 or another computer-readable medium.

[0133] The storage device 1406 may include one or more data storage media, devices, or configurations and may take any type, form, and combination of data storage media and / or devices. For example, the storage device 1406 may include, but is not limited to, a hard drive, a network drive, a flash drive, a magnetic disk, an optical disk, RAM, dynamic RAM, other non-volatile and / or volatile data storage units, or a combination or sub-combination thereof. Electronic data (including the data described herein) may be temporarily and / or permanently stored in the storage device 1406. For example, data representing one or more executable applications 1412 configured to direct the processor 1404 to perform any of the operations described herein may be stored in the storage device 1406. In some examples, the data may be arranged in one or more databases residing within the storage device 1406.

[0134] I / O module 1408 may include one or more I / O modules configured to receive user input and provide user output. One or more I / O modules may be used to receive input for a single virtual reality experience. I / O module 1408 may include any hardware, firmware, software, or combination thereof that supports input and output capabilities. For example, I / O module 1408 may include hardware and / or software for capturing user input, including but not limited to a keyboard or keypad, a touch screen component (e.g., a touch screen display), a receiver (e.g., an RF or infrared receiver), a motion sensor, and / or one or more input buttons.

[0135] The I / O module 1408 may include one or more devices for presenting output to a user, including, but not limited to, a graphics engine, a display (e.g., a display screen), one or more output drivers (e.g., a display driver), one or more audio speakers, and one or more audio drivers. In some embodiments, the I / O module 1408 is configured to provide graphical data to the display for presentation to the user. The graphical data may represent one or more graphical user interfaces and / or any other graphical content that may be used for a particular implementation.

[0136] In some examples, any of the facilities described herein may be implemented by or within one or more components of computing device 1400. For example, one or more applications 1412 residing within storage device 1406 may be configured to direct processor 1404 to perform one or more processes or functions associated with processing facility 404 of system 400. Likewise, storage facility 402 of system 400 may be implemented by storage device 1406 or a component thereof.

[0137] In the foregoing description, various exemplary embodiments have been described with reference to the accompanying drawings. However, it will be apparent that various modifications and changes may be made to these embodiments, and additional embodiments may be implemented, without departing from the scope of the invention as set forth in the appended claims. For example, certain features of one embodiment described herein may be combined with features of another embodiment described herein or may replace another embodiment. Therefore, the description and drawings are to be regarded as illustrative rather than restrictive.

Claims

1. A computer-assisted operating system, comprising: a memory for storing instructions; a processor communicatively coupled to the memory and configured to execute the instructions to: identifying a specific operation to be performed by a computer-assisted manipulation system during an manipulation session, the specific operation to be performed on a body while an imaging device included within the computer-assisted manipulation system provides an image of the body from a first viewpoint for display on a display device during the manipulation session, Based on the specific operation, defining a second viewpoint for the imaging device that is more optimized than the first viewpoint for performing the specific operation, directing the display device to display an indication of the second viewpoint; determining that the user is using a first wrist gesture associated with the first viewpoint to guide the computer-assisted operating system to perform the specific operation, and determining that a second wrist gesture associated with a viewpoint having a horizontal orientation different from the horizontal orientation of the first viewpoint is more optimal than the first wrist gesture to direct performance of the particular operation; and Wherein the second viewpoint is defined as the viewpoint associated with the second wrist posture and having the horizontal orientation different from the horizontal orientation of the first viewpoint.

2. The computer-assisted operating system according to claim 1, wherein: The instruction for directing the display device to display the second viewpoint also includes directing the display device to display a graphic object indicating the second viewpoint when the display device is displaying the image of the body from the first viewpoint, and the graphic object is displayed as an overlay graphic integrated with the image of the body displayed from the first viewpoint.

3. The computer-assisted operating system according to claim 2, wherein: The graphics object indicating the second viewpoint includes a reticle object indicating a horizontal orientation of the second viewpoint.

4. The computer-assisted operating system according to claim 2, wherein: The graphics object indicating the second viewpoint includes a bounding box indicating at least one of a scale orientation and a planar orientation of the second viewpoint.

5. The computer-assisted operating system according to claim 2, wherein: The graphic object indicating the second viewpoint comprises: As the imaging device provides images of the body from different viewpoints, a first three-dimensional shape, i.e., a first 3D shape, anchored to the field of view of the imaging device, and A second 3D shape indicating at least one of a zoom orientation, a plan orientation, a horizontal orientation, a pitch orientation, and a yaw orientation of the second viewpoint.

6. The computer-assisted operating system according to claim 1, wherein: The instruction for guiding the display device to display the second viewpoint further includes: directing the display device to present an indication that the second viewpoint has been defined when the display device displays the image of the body from the first viewpoint; and responsive to said presenting of said indication that said second viewpoint has been defined, receiving user input indicating that a user of said system has selected to view an image of said body from said second viewpoint rather than viewing said image of said body from said first viewpoint; Wherein directing the display device to switch from displaying the image of the body from the first viewpoint to displaying the image of the body from the second viewpoint is based on receiving the user input.

7. The computer-assisted operating system according to claim 1, wherein: Directing the display device to display the indication of the second viewpoint includes directing the display device to automatically switch from displaying the image of the body from the first viewpoint to displaying the image of the body from the second viewpoint in response to definition of the second viewpoint.

8. The computer-assisted operating system according to claim 1, wherein: The processor is further configured to execute the instructions to: determining that the display device depicts a moving image depicting performance of the particular operation with respect to the body at a first level of detail when the image of the body is displayed from the first viewpoint, and determining that displaying an image of the body from a viewpoint having a zoom orientation different from that of the first viewpoint will cause the active image to be depicted at a second level of detail that is more optimized than the first level of detail to perform the particular operation; and The second viewpoint is defined as the viewpoint having the zoom orientation different from the zoom orientation of the first viewpoint such that the active image is depicted at the second level of detail that is more optimized for performing the particular operation.

9. The computer-assisted operating system according to claim 1, wherein: The processor is further configured to execute the instructions to: determining that an active image depicting performance of the particular operation with respect to the body is depicted at a first portion of a field of view presented by the display device when the image of the body is displayed from the first viewpoint, and determining that displaying an image of the body from a viewpoint having a planar orientation different from the planar orientation of the first viewpoint will cause the active image to be depicted in a second portion of the field of view that is more optimized than the first portion of the field of view to perform the particular operation; as well as The second viewpoint is defined as the viewpoint having the planar orientation different from the planar orientation of the first viewpoint such that the active image is depicted at the second portion of the field of view that is more optimized to perform the particular operation.

10. The computer-assisted operating system according to claim 1, wherein: The processor is further configured to execute the instructions to: determining the identity of the user who selected the first viewpoint, and accessing user-specific data representing a viewpoint selection performed by the user prior to the operational session based on the identity of the user; and The second viewpoint is further defined based on the user-specific data representing the viewpoint selection performed by the user prior to the operating session.

11. The computer-assisted operating system according to claim 1, wherein: The definition of the second viewpoint being more optimized than the first viewpoint for performing the specific operation comprises: identifying a recommended viewpoint for the particular operation based on input provided in real time during the operation session by an expert advisor different from the user performing the particular operation; and The second viewpoint is defined based on the recommended viewpoint.

12. The computer-assisted operating system according to claim 1, wherein: The definition of the second viewpoint that performs the specific operation better than the first viewpoint includes: accessing data representing viewpoint selections performed prior to the operational session by an expert advisor different from the user performing the particular operation; and The second viewpoint is defined based on access data representing the viewpoint selection of the expert advisor.

13. The computer-assisted operating system according to claim 1, wherein: The definition of the second viewpoint that performs the specific operation better than the first viewpoint includes: identifying a recommended viewpoint for the specific operation based on an automatic viewpoint recommendation algorithm configured to generate operation-specific viewpoint recommendations in real-time during the operation session; and The second viewpoint is defined based on the recommended viewpoint.

14. A computer-assisted operating system, comprising: a memory for storing instructions; a processor communicatively coupled to the memory and configured to execute the instructions to: determining during an operation session that a user uses a first wrist gesture associated with a first viewpoint to direct a computer-assisted operating system to perform an operation included in a plurality of operations, during which the computer-assisted operating system performs the plurality of operations on a body while an imaging device included in the computer-assisted operating system provides an image of the body from the first viewpoint for display to the user via a display device, defining, during the manipulation session, a second viewpoint associated with a second wrist gesture, the second wrist gesture directing the performance of the manipulation more optimally than the first wrist gesture, the second viewpoint having a horizontal orientation different from a horizontal orientation of the first viewpoint, directing the display device to integrate a graticule overlay graphic indicating the horizontal orientation of the second viewpoint with the displayed image of the body from the first viewpoint when the display device displays the image of the body from the first viewpoint, responsive to the integration of the reticle overlay graphic indicating the horizontal orientation of the second viewpoint, receiving user input indicating that the user selects to view an image of the body from the second viewpoint rather than the image of the body from the first viewpoint, and The display device is directed to switch from displaying the image of the body from the first viewpoint to displaying the image of the body from the second viewpoint in response to the user input.

15. A method for computer-assisted operation, comprising: identifying, by a viewpoint optimization system, a specific operation performed on a body by a computer-assisted manipulation system during a manipulation session, while an imaging device included in the computer-assisted manipulation system provides an image of the body from a first viewpoint for display on a display device during the manipulation session; defining, by the viewpoint optimization system, for the imaging device a second viewpoint that is more optimized than the first viewpoint for performing the specific operation based on the specific operation; guiding the display device to display an indication of the second viewpoint by the viewpoint optimization system; determining that the user is using a first wrist gesture associated with the first viewpoint to guide the computer-assisted operating system to perform the specific operation, and determining that a second wrist gesture associated with a viewpoint having a horizontal orientation different from the horizontal orientation of the first viewpoint is more optimal than the first wrist gesture to direct performance of the particular operation; and Wherein the second viewpoint is defined as the viewpoint associated with the second wrist posture and having the horizontal orientation different from the horizontal orientation of the first viewpoint.

16. The method according to claim 15, wherein: The instructions for directing the display device to display the second viewpoint also include directing the display device to display a graphic object indicating the second viewpoint when the display device is displaying the image of the body from the first viewpoint, and the graphic object is displayed as an overlay graphic integrated with the image of the body displayed from the first viewpoint.

17. The method according to claim 15, wherein: The instruction for guiding the display device to display the second viewpoint further includes: directing the display device to present an indication that the second viewpoint has been defined while the display device is displaying the image of the body from the first viewpoint; and responsive to said presenting of said indication that said second viewpoint has been defined, receiving user input indicating that a user of said system has selected to view an image of said body from said second viewpoint rather than from said first viewpoint; Wherein directing the display device to switch from displaying the image of the body from the first viewpoint to displaying the image of the body from the second viewpoint is based on receiving the user input.

18. The method according to claim 15, wherein: Directing the display device to display the indication of the second viewpoint includes directing the display device to automatically switch from displaying the image of the body from the first viewpoint to displaying the image of the body from the second viewpoint in response to the definition of the second viewpoint.

19. A non-transitory computer-readable medium storing instructions that, when executed, direct a processor of a computing device to: identifying a specific operation to be performed by a computer-assisted manipulation system during an manipulation session, the specific operation to be performed on a body while an imaging device included within the computer-assisted manipulation system provides an image of the body from a first viewpoint for display on a display device during the manipulation session, Based on the specific operation, defining a second viewpoint for the imaging device that is more optimized than the first viewpoint for performing the specific operation, directing the display device to display an instruction of the second viewpoint, determining that the user is using a first wrist gesture associated with the first viewpoint to guide the computer-assisted operating system to perform the specific operation, and determining that a second wrist gesture associated with a viewpoint having a horizontal orientation different from the horizontal orientation of the first viewpoint is more optimal than the first wrist gesture to direct performance of the particular operation; and Wherein the second viewpoint is defined as the viewpoint associated with the second wrist posture and having the horizontal orientation different from the horizontal orientation of the first viewpoint.

Images