Systems and methods for content aware user interface overlay

Abstract

A medical system may include a display system for displaying a display area and a control system. The control system may include a processing unit including one or more processors. The processing unit may be configured to generate a field-of-view image for display in the display area, generate a user interface component for display in the display area, determine a priority record of elements in the field-of-view image, determine a display characteristic of the user interface component based on the priority record of elements in the field-of-view, and display the user interface component based on the display characteristic. And displaying the user interface component in a display area overlying the display image of the field of view according to the display characteristics.

Description

[0001] Cross-reference application

[0002] This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 341,881, filed on May 13, 2022, entitled “Systems and Methods for ContentAware User Interface Overlays,” which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to systems and methods for robotically assisted medical procedures, and more particularly to systems and methods for arranging user interface overlays based on knowledge of underlying field of view image content. Background Art

[0004] Minimally invasive medical techniques are intended to reduce the amount of external tissue damaged during diagnostic or surgical procedures, thereby reducing the patient's recovery time, discomfort, and harmful side effects. Such minimally invasive techniques can be performed through natural orifices in the patient's anatomy or through one or more surgical incisions. Through these natural orifices or incisions, clinicians can insert medical tools to reach the target tissue location. Minimally invasive medical tools include instruments such as therapeutic instruments, diagnostic instruments, and surgical instruments. Minimally invasive medical tools can also include imaging instruments, such as endoscopic instruments that provide users with images of the field of view within the patient's anatomy.

[0005] Some minimally invasive medical tools may be robotically assisted, including teleoperation, tele-operation, or other forms of computer assistance. During a medical procedure, a clinician may view a field of view image of a patient's anatomy that may include one or more minimally invasive medical tools. Various user interface components may be overlaid on the field of view image to provide, for example, information, interactive functionality, and tool control. Improved systems and methods are needed to present overlaid user interface components based on knowledge of the underlying content of the field of view. Summary of the invention

[0006] Embodiments of the invention are best summarized by the claims appended to the description.

[0007] In an exemplary embodiment, a medical system may include a display system and a control system for displaying a display area. The control system may include a processing unit including one or more processors. The processing unit may be configured to generate an image of a field of view for display in the display area, generate a user interface component for display in the display area, determine a priority record of elements in the image of the field of view, determine display characteristics of the user interface component based on the priority record of the elements in the field of view, and display the user interface component in the display area overlaid on the displayed image of the field of view according to the display characteristics.

[0008] It should be understood that the above general description and the following detailed description are exemplary and explanatory in nature, and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In this regard, through the following detailed description, additional aspects, features and advantages of the present disclosure will be clear to those skilled in the art. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a flow chart illustrating a method for displaying a user interface component overlay based on system awareness of field of view content, according to some examples.

[0010] Figure 2 A graphical user interface including a field of view image with overlaid user interface components is shown according to some examples.

[0011] Figure 3A According to some examples, the Figure 2 A graphical user interface of an adjusted field of view image with overlaid user interface components on high priority elements in the field of view.

[0012] Figure 3B Shown include Figure 3A A graphical user interface of an image of a field of view with overlaying user interface components moved to cover lower priority elements in the field of view.

[0013] Figure 4A A graphical user interface is shown with overlaid user interface components over a high priority element in a field of view according to some examples.

[0014] Figure 4B Shown include Figure 4A A graphical user interface of a field of view with overlaying user interface components moved to cover lower priority elements in the field of view.

[0015] Figure 5A A graphical user interface including a field of view image with overlaid user interface components is shown according to some examples.

[0016] Figure 5B Shown include Figure 5A A graphical user interface of an adjusted field of view image with overlaid user interface components on top of high priority elements in the field of view.

[0017] Figure 5C Shown include Figure 5B A graphical user interface of a field of view with overlaying user interface components moved to cover lower priority elements in the field of view.

[0018] Figure 6 A graphical user interface with layered user interface components is shown according to some examples.

[0019] Figure 7 A graphical user interface with a semi-transparent user interface component including an overlay of an auxiliary ultrasound image is shown according to some examples.

[0020] Figure 8 A graphical user interface is shown having an overlaid user interface component and having a highlighted border outlining underlying elements in a field of view image according to some examples.

[0021] Fig. 9 Schematic diagrams of medical systems are shown according to some examples.

[0022] Fig.10 Based on some examples Fig. 9 A perspective view of a manipulator assembly of a medical system.

[0023] Fig.11 is a front view of an operator console in a robotic-assisted medical system according to some examples.

[0024] Embodiments of the present disclosure and their advantages can be best understood by referring to the following detailed description. It should be understood that the same reference numerals are used to identify the same elements shown in one or more figures, which are shown to illustrate embodiments of the present disclosure, but not to limit the present disclosure. DETAILED DESCRIPTION

[0025] In a robotic-assisted medical procedure, an endoscopic image of a surgical environment can provide a clinician with a field of view of the patient's anatomy and an image of any medical tools located in the patient's anatomy. Various user interface components can be overlaid on the field of view image. The user interface components can include, for example, menus, alarms, user messages, indicators, digital tools, and auxiliary images. These overlaid components may obscure portions of the field of view. The presentation of these user interface components can be adjusted based on the knowledge of the image content to prioritize the display of segments of the field of view image that are most relevant to the user.

[0026] Figure 11 is a flow chart showing a method 100 for displaying a user interface component overlay based on the recognition of the content of the field of view image. The methods described herein are shown as a set of operations or processes, and are described with reference to the remaining figures. Not all of the processes shown can be performed in all embodiments of the method. In addition, one or more processes not explicitly shown may be included before, after, between, or as part of the process shown. In some embodiments, one or more of the processes shown may be omitted. In some embodiments, one or more processes may be implemented at least in part in the form of executable code stored on a non-transient, tangible, machine-readable medium, which, when executed by one or more processors (e.g., a processing unit of a control system such as control system 320), may cause one or more processors to execute one or more of the processes. In one or more embodiments, these processes may be performed by a control system.

[0027] At process 102, a field of view image may be generated for display. For example, Figure 2 A display system 200 is provided that includes a display area 201 for displaying a graphical user interface 202 that includes a field of view image 204. The field of view image 204 may be provided by an imaging device (e.g., an endoscopic imaging system 715 described below) within an anatomical environment in a patient's anatomy. In this example, the image 204 may be a three-dimensional stereoscopic image, but in other examples, the image may be a two-dimensional image. The image 204 may be an intraoperative real-time video endoscopic image, or may be an image previously recorded (including preoperatively). The field of view image 204 may have, for example, an image reference frame X based on a distal end of the endoscopic imaging system. I , Y I , Z I .

[0028] Reference again Figure 1 In method 100, at process 104, a user interface component can be generated for display. The user interface component can include components of a graphical user interface, such as windows, alphanumeric text, labels, or other artificial graphic elements overlaid on the field of view image or inserted at a three-dimensional location in the field of view image. For example, the user interface component can include menus, alarms, user messages, indicators, digital tools, measurement tools, and auxiliary still or video images. In addition or alternatively, the user interface component can be a digitally modified segment of the field of view image. For example, an endoscopic image of patient tissue can be modified to enhance the appearance of the vasculature by providing higher contrast, amplify local motion of structures in the endoscopic image, or blend fluoroscopic signals with the endoscopic image. Figure 2An example of a user interface component 212 is provided, in which the user interface component can be a window displaying a three-dimensional model of a patient's anatomy generated from a preoperative CT image. The model can be a static three-dimensional image or an interactive element that can be manipulated (e.g., rotated) in response to user interaction.

[0029] Reference again Figure 1 , at process 106, a priority record may be determined for an element in the field of view image. The element in the field of view may include an anatomical structure, a medical structure (such as an instrument or other tool), or an active segment (such as a moving or bleeding segment). Knowledge about the relative importance, criticality, or relevance of the elements in the field of view image may be captured as one or more priority records associated with the elements in the field of view. The priority record may provide a ranking indicator or other indication of the relative importance of the element to a clinician viewing the field of view image or to the safety or effectiveness of a procedure. A relatively high priority record may indicate that the element is a critical structure that a user interface component should avoid covering. A mapping of the elements in the field of view and their priority records may be stored in a memory (e.g., memory 734) and accessed by a control system (e.g., control system 320) of a robotic-assisted medical system (e.g., system 310).

[0030] Process 106 may include distinguishing and / or identifying elements in the field of view image, and may assign a priority record to one or more identified elements. Figure 2 Identified elements in the field of view image 204 are shown, including tool 208, tool 210, surgically exposed tissue 214, and surgically uninvolved tissue 205, 206. In some examples, the identified elements in the field of view image can be identified by lines (e.g., Figure 2 The elements may be demarcated by dashed lines (shown as dashed lines), color modifications, markings, or other graphical indicators. In other examples, the identified elements may be distinguished by the control system without visible or displayed boundaries. A priority record may be assigned to each distinguished element based on predetermined priority categories for various types of structures. Priority levels may also be predefined. For example, a scale of 1 (low criticality or relevance) to 5 (high criticality or relevance) may be used to assign priority records based on priority categories, such as instruments under clinician control, instruments not under clinician control, tissue structures involved in surgery, tissue structures not involved in surgery, tissue motion segments (e.g., beating, pulsating, moved by an instrument), or fluid flow segments (e.g., bleeding, mucus flow). Figure 2In the example of FIG. 1 , activated tools 208 , 210 may be assigned a priority record of 5 (high criticality or relevance), exposed tissue 214 that is a tissue structure involved in the surgical procedure may be assigned a priority record of 3 (medium criticality or relevance), and tissue segments 205 , 206 that are not involved in the surgical procedure may be assigned a priority record of 1 (low criticality or relevance). Any of a variety of linear scales, multidimensional parameters, or combinations of rankings and factors may be used to determine the relevant priority records for elements in the field of view image. Determinations of criticality and relevance may be based on patient health and safety, operator interest, importance to the medical procedure, or other criteria.

[0031] In some examples, priority records can be assigned to elements in the field of view image by a clinician or other operator (such as a surgeon on a surgeon's console or a laparoscopic tool operator on the patient's bedside) who is viewing the display. Operator priority record assignment can include identifying elements in the field of view image by marking, outlining, providing segment boundaries, annotations, or selecting from a menu of image segmentation elements. Priority records can be assigned to elements identified by the operator. In some examples, the field of view image can be graphically segmented (e.g., segmented into pixels or voxels), and priority records can be assigned to segments of the segmented image based on computer vision or computer image analysis of the image and predetermined categories of elements detected by the image analysis. Artificial intelligence can be used for analysis. Image analysis can detect any changes in tissue structure, tissue segment, tool, anatomical landmark, tool or tissue movement, anatomical movement or tool or tissue structure, and can determine the relevance of the structure to identify which should not be covered. In some examples, image analysis can be based on a co-registered preoperative model including identified and prioritized model elements, a co-registered intraoperative imaging modality (e.g., ultrasound, fluorescence imaging), or a physical fiducial or other marker in the field of view image to assign priority records. In some examples, priority records may be generated and assigned by a control system or other processor-based system, and may be modified or supplemented by a clinician or other operator.

[0032] In some examples, the priority record can be dynamic. For example, the priority record of a tool element may change from a low priority record when it is detected to be inactive to a high priority record when it is detected to be activated. The priority record of a tissue segment may change from a low priority record when it is detected not to be involved in a medical procedure to a high priority when it is detected to be adjacent to an activated tool. Tissue elements may also change to a higher priority when exposed, moved, manipulated by a tool, or change state (such as from a non-bleeding state to a bleeding state). Certain segments, regions, or elements in the field of view may not be assigned a priority record.

[0033] Reference again Figure 1, at process 108, program characteristics may optionally be determined. Program characteristics may include a type of procedure or a step or task within a procedure. Program characteristics may include, for example, a type of task performed in an anatomical environment, such as dissection, suturing, stapling, irrigation, or tissue retraction. For example, the type of task may be used to determine a location at which a user interface component is displayed so as not to interfere with an operator's viewing of the task. Program characteristics may also include user preferences of an operator stored in a user profile. User preferences may include, for example, a preferred location for an operator to view a user interface component in a display.

[0034] At process 110, display characteristics of the user interface component can be determined. The display characteristics can include one or more attributes associated with the user interface component, such as placement position, placement orientation, minimum size, maximum size, nominal size, transparency, scaling factor, display duration and / or display interval. The display characteristics can be static, or can be dynamic, in response to changes in the field of view. The display characteristics of the user interface component can be determined based on the priority records of the elements in the field of view to avoid the abrupt interference caused by the user interface component. For example, the display characteristics can be the placement position of the user interface component in the graphical user interface, which avoids covering the field of view elements with relatively high priority records, but covers areas without priority records or with relatively low priority records. The display characteristics can also or alternatively be determined based on program characteristics. For example, the display characteristics can be determined based on the type of task or the user's preferences. At process 112, the user interface component can be displayed on the display image of the field of view according to the conditions of the display characteristics.

[0035] Figure 2-Figure 8 An example of an implementation of the method 100 is shown. Figure 2 , in the field of view image 204, priority records 5 (high criticality) may be assigned to the activated tools 208, 210, priority records 3 (medium criticality) may be assigned to the exposed tissue 214, and priority records 1 (low criticality) may be assigned to the tissue sections 205, 206 not involved in the surgical procedure. The display characteristic of the user interface component 212 may be a display position of the user interface component relative to the field of view image 204. Based on the priority records of the elements in the field of view image 204, the display characteristic may be a display position of overlaying the tissue element 205 having a low priority record. If the field of view includes multiple low priority record elements, the display position may be further determined based on proximity to higher priority elements, based on operator preference (e.g., preference to overlay to the left / right or top / bottom on the display), or based on other factors.

[0036] Reference Figure 3A ,and Figure 2In contrast, the endoscopic imaging instrument can be translated upward, thereby changing the field of view image 204 displayed on the display system 200. The user interface component 212 is located at a position relative to the display system 200. Figure 2 204, but now obscures the view of element 216 in the field of view. Element 216 may be an active device that has been assigned a priority record of 5 (high criticality). The new display characteristics of user interface component 212 may be determined based on the priority record of element 216 that is now present in modified field of view image 204. Figure 3B As shown, the new display characteristics of the user interface component 212 can be a new display position of the covering tissue element 205 relative to the display system 200. In some examples, the new display position can be unchanged relative to the tissue element 205, but in other examples, it can also change relative to the tissue element 205. With the new display characteristics, the user interface component 212 is no longer covered on the element 216 with a high priority record. Therefore, the element in the field of view image with a high priority record is still visible to the clinician. When the image analysis determines a change in the field of view or the endoscopic imaging instrument, the new display characteristics can be determined and dynamically updated by the control system. The change in the field of view image can be determined based on kinematic information related to the endoscopic imaging system, or can be determined based on image analysis of the field of view image. Optionally, the display characteristics of the user interface component can include a nominal size, and in order to maintain the nominal size, the display position of the user interface component 212 can cover a higher priority element (e.g., element 214). The allowed coverage area can be less than a threshold percentage of the visible area of ​​the element 214 in the field of view image 204.

[0037] In some examples, the display characteristic of user interface component 212 can be the position of the user interface element relative to (e.g., pinned or tethered to) an element visible in the field of view. For example, the display characteristic of user interface component 212 can be a display position relative to tissue element 205. In this example, user interface component 212 is displayed in the same relative position and orientation relative to tissue element 205 as long as tissue element 205 is visible in field of view image 204, regardless of the position of tissue element 205 relative to display system 200.

[0038] In some alternative examples, the display characteristic of the user interface component may be a duty cycle for the intermittent display of the user interface component. Figure 3A The position of the covering tool element 216 is shown or Figure 3B The user interface component 212 shown overlying the location of the tissue element 205 may also include a display feature that causes the user interface component 112 to flash or otherwise display with an intermittent duty cycle so that during a portion of the duty cycle, the viewing clinician can see the underlying elements.

[0039] In some alternative examples, the display characteristic of the user interface component can be the duration of time that the user interface component is displayed. Figure 3A The position of the covering tool element 216 is shown or located as shown in Figure 3B The user interface component 212 shown covering the location of the organizing element 205 may also include a display feature that causes the user interface component 212 to be displayed for a limited duration before the user interface component 212 disappears from the display. In some examples, the user may selectively turn off the user interface component so that it disappears from the display.

[0040] In some examples, display characteristics may allow the display position of a user interface component to be overlaid on elements having a predetermined range of allowed priority records (e.g., ranked 1-3), and may limit the display position of overlaying elements having a predetermined restricted range of allowed priority records (e.g., ranked 4-5).

[0041] In some examples, the display characteristics can indicate that the user interface component is obscuring an important underlying object or structure in the field of view or user interface. For example, the display characteristics can include an outline with a unique color, line style, icon, or other visual treatment to warn the user of the obstruction.

[0042] Reference Figure 4A ,and Figure 2 In contrast, the endoscopic imaging device generating the field of view image can be zoomed in and translated upward and leftward, thereby changing the field of view image 204 displayed on the display system 200. The user interface component 212 is located relative to the display system 200. Figure 2 204, but now obscures the view of element 216 in the field of view. Element 216 may be an active device that is assigned a priority record of 5 (high criticality). The new display characteristics of user interface component 212 may be determined based on the priority record of element 216 that is now present in modified field of view image 204. Figure 4B As shown, the new display characteristics of user interface component 212 can be a new display position relative to display system 200 that overlays organizational element 206 (having priority record 1 (low criticality)). With the new display characteristics, user interface component 212 no longer overlays element 216 having a high priority record. Optionally, the display characteristics of the user interface component can include a scaling factor to maintain the nominal size of the user interface component relative to display system 200. Thus, in Figure 4B Even if the elements in the field of view image 204 appear smaller due to the imaging device scaling factor, Figure 2 The size of the user interface component 212 does not scale with the elements in the field of view, but remains the same relative to the display system 200. Figure 2In an alternative example, the display characteristic may be a scaling factor that is proportionally scaled with the instrument scaling factor.

[0043] Figure 5A The display system 200 is shown displaying a graphical user interface 302 that includes a field of view image 304. The image 304 includes a tool 308, a tool 310, a tool 316, surgically exposed tissue 314, and surgically uninvolved tissue 305, 306. The tools 308, 310, 316 may be assigned a priority record of 5 (high criticality), the exposed tissue 314 may be assigned a priority record of 3 (medium criticality), and the surgically uninvolved tissue segments 305, 306 may be assigned a priority record of 1 (low criticality). Figure 5B ,and Figure 5A In contrast, the endoscopic imaging instrument can be translated to the left, thereby changing the field of view image 304 displayed on the display system 200. The user interface component 312 is located at a position relative to the display system 200. Figure 5A 316 in the field of view. Tool element 316 may be an active instrument that is assigned a priority record of 5 (high criticality). The new display characteristics of user interface component 312 may be determined based on the priority record of tool element 316 that is now present in modified field of view image 304. Figure 5C As shown, new display characteristics of user interface component 312 may include a new display position of overlay organization element 306 relative to display system 200. With the new display characteristics, user interface component 312 is no longer overlaid on tool element 316 having a high priority record. The new display characteristics of user interface component 312 may also include a new, smaller zoom relative to display system 200. The zoom display characteristics may be determined based on the area of ​​organization element 306 and, optionally, a minimum size display characteristic for user interface component 312 that prevents the user interface component from becoming too small.

[0044] Figure 6A display system 200 is shown displaying a graphical user interface 402 including a field of view image 404. Image 404 includes tool element 408, tool element 410, tissue element 405 involved in the surgical procedure, and tissue element 406 not involved in the surgical procedure. Tissue element 406 can be assigned a low priority record. In this example, multiple user interface components can include display characteristics, which include display positions overlaid on tissue element 406. For example, user interface component 420 is a message containing alphanumeric text for conveying information to a clinician during a medical procedure. User interface component 422 is a window defining multiple user interface components, including menu 424, indicator 426, measurement tool 428, and interactive tool 430. User interface component 432 includes an interactive anatomical model. User interface component 434 includes a fluoroscopic anatomical image of a patient's anatomy. User interface component 436 includes a high contrast image of an anatomical vascular system. In some examples, the display characteristics of each user interface component also include dynamic layer, order, or depth allocation parameters for determining the sequential layering order of the user interface components. For example, if the area of ​​organizational element 406 is insufficient to fit all user interface components simultaneously in a single visible layer, such a display characteristic may be useful. In this example, user interface component 420 may be assigned a dynamic layer parameter that has a higher value than the dynamic layer parameters of user interface components 422-436. As a result, user interface component 420 may be layered on top of other user interface components 422-436, thereby allowing the contents of user interface component 420 to be fully visible to a viewing clinician.

[0045] Dynamic layer parameters can change based on the current state of the medical procedure or based on operator selection. For example, a user interface component 420 that includes a warning message that warns the clinician of an extracorporeal instrument collision can be assigned a high dynamic layer parameter so that the user interface component 420 dynamically appears and is presented as a top layer, located above other user interface components covering the tissue element 406. When the instrument collision is corrected, a lower dynamic layer parameter can be assigned to the user interface component 420 so that the message window can be dropped to a lower layer or closed, allowing another user interface component with a higher dynamic layer parameter to be displayed on the top layer. In other examples, if image analysis of the field of view image detects a color change in the image that may be associated with patient bleeding, the user interface component including the warning can be displayed on other user interface components, and any displayed user interface component can change the display position so that the color change area (now assigned a high priority record) is not covered by the user interface component.

[0046] Figure 7A display system 200 is shown displaying a graphical user interface 502 including a field of view image 504. The image 504 includes a tissue element 506 that is not involved in a surgical procedure to which a low priority record may be assigned. In this example, a user interface component 512 may include a window displaying an ultrasound image of a patient's anatomical structure, and may include display characteristics including display distortions, such as display transparency or display blur of the window when the window is overlaid on the tissue element 506. The display transparency may be a fixed level of semi-transparency, such as 25% transparency, 50% transparency, 75% transparency, or any transparency level between 0 and 100%, which may allow a viewing clinician to at least partially observe the covered tissue element 506 beneath the user interface component 512. In some examples, a user interface component 512 with display transparency may be used to cover a field of view element with a relatively high priority record (e.g., greater than 3 on a scale of 1-5) so that the user interface component may be visualized below the user interface component.

[0047] Figure 8A display system 200 is shown displaying a graphical user interface 602 including a field of view image 604. Image 604 includes a tissue element 606 not involved in the surgical procedure that can be assigned a low priority record and a tool element 610 that can be assigned a higher priority record than element 606. In this example, a user interface component 612 may include a window displaying an ultrasound image of a patient's anatomy, and may include display characteristics including a display transparency of the window when the window is overlaid on the tissue element 606 and the tool element 610. The display transparency may be a fixed level of transparency, such as 25% transparency, 50% transparency, 75% transparency, or any level of transparency between 0 and 100% that may allow a viewing clinician to at least partially observe the underlying covering tissue element 606 and tool element 610 of the user interface component 512. In an alternative example, the display transparency may be dynamic, becoming more or less transparent based on changing lighting conditions, changes in underlying elements, clinician selections, or other conditions in the field of view. The display characteristics of the user interface component 612 may also include a display outline 614 of the edge boundaries of the structure covered by the user interface component in the field of view. For example, the user interface 612 may include a bold outline, outline overlay, or otherwise highlighted border marking the edge of the tool element 610 underlying the user interface component 612. In some examples, the displayed outline may be displayed only for portions of the user interface component that overlay image elements with sufficiently high priority records (e.g., priority records of 3 or higher on a scale of 1-5). In some examples, the underlying structure (e.g., the instrument outline) may be used to mask the overlay to display only the relevant features of the underlying structure. In alternative examples, the user interface component 612 may provide other types of information (e.g., alphanumeric labels, markings, shading) to indicate, describe, or mark the underlying image elements.

[0048] In some alternative examples, the display characteristic can be associated with or "pinned" to the display of an element in the field of view image or a category or type of element in the field of view image (e.g., a category of tissue elements not involved in the surgical procedure). In some examples, the category of the element can be a structural type that can be distinguished by a pixel or voxel type (e.g., a pixel or voxel associated with the color or texture of body wall tissue), so that the user interface component can be pinned, tethered, attached, or otherwise displayed in association with the structure or type of the identified pixels / voxels. In some examples, the clinician can assign or adjust the display characteristics, such as the display position. For example, the clinician can adjust the imaging system to provide a field of view image that includes tissue not involved in the surgical procedure or other non-critical tissue (such as the body wall), and then the display position can be assigned to the user interface component, pinning or attaching the user interface component to the body wall.

[0049] In some examples, if the image analysis detects tissue movement, instrument movement, color change (e.g., bleeding), or other changes in the field of view image, the display characteristics and / or priority records can be dynamically updated so that the user interface component does not cover the changed area. In some examples, a clinician or other operator may be allowed or required to confirm changes to display characteristics, such as changes in position. In some examples, the display characteristics may allow the user to nudge the user interface component by moving an instrument visible in the field of view. In some examples, nudges may be allowed when the medical system is in instrument control or a "follow" mode of the medical system. In some examples, the display characteristics of the user interface component may depend on the type of display device. For example, the display position or other display characteristics of the user interface component and the field of view image may have a specific set of values ​​for the surgeon's console in the display system. When displayed on a tablet or mobile device, the same field of view image and user interface component may have different display characteristics (e.g., display position).

[0050] Figure 9-11 Together, an overview of a medical system 710 that can be used in medical procedures including, for example, diagnostic, therapeutic, or surgical procedures is provided. The user interface display examples provided above can be used in the context of the medical system 710. The medical system 710 is located in a medical environment 711. Fig. 9 In some embodiments, medical environment 711 is depicted as an operating room. In other embodiments, medical environment 711 may be an emergency room, a medical training environment, a medical laboratory, or some other type of environment in which any number of medical procedures or medical training procedures may be performed. In other embodiments, medical environment 711 may include an operating room and a control area located outside the operating room.

[0051] In one or more embodiments, the medical system 710 may be a robotically assisted medical system under the teleoperational control of an operator (e.g., a surgeon, clinician, physician, etc.). In alternative embodiments, the medical system 710 may be under partial control of a computer programmed to perform a medical procedure or subroutine. In further alternative embodiments, the medical system 710 may be a fully automated medical system under full control of a computer programmed to perform a medical procedure or subroutine with the medical system 710. An example of a medical system 710 that may be used to implement the systems and techniques described in the present disclosure is a da Surgical system (da Surgical System).

[0052] like Fig. 9As shown, medical system 710 generally includes component 712, which can be mounted to or positioned near an operating table T where a patient P is positioned. Component 712 can be referred to as a patient side cart, a surgical cart, or a surgical robot. In one or more embodiments, component 712 can be a remotely operated component. The remotely operated component can be referred to as, for example, a remotely operated arm cart. Medical device system 714 and endoscopic imaging system 715 can be operatively coupled to component 712. Operator input system 716 allows operator O or other types of clinicians to view images of the surgical site or images representing the surgical site and control the operation of medical device system 714 and / or endoscopic imaging system 715.

[0053] The medical device system 714 may include one or more medical devices. In embodiments where the medical device system 714 includes multiple medical devices, the multiple medical devices may include multiple identical medical devices and / or multiple different medical devices. Similarly, the endoscopic imaging system 715 may include one or more endoscopes. In the case of multiple endoscopes, the multiple endoscopes may include multiple identical endoscopes and / or multiple different endoscopes.

[0054] The operator input system 716 may be located at an operator console, which may be located in the same room as the operating table T. In some embodiments, the operator O and the operator input system 714 may be located in a different room than the patient P or in a completely different building. The operator input system 716 typically includes one or more control devices for controlling the medical device system 714. The control device(s) may include one or more of any number of various input devices, such as a handle, a joystick, a trackball, a data glove, a trigger gun, a foot pedal, a manual controller, a voice recognition device, a touch screen, a body motion or presence sensor, and other types of input devices.

[0055] In some embodiments, the control device(s) will be provided with the same degrees of freedom as the medical device(s) of the medical device system 714 to provide telepresence to the operator, which is a perception that the control device(s) are integral with the instrument, giving the operator a strong sense of directly controlling the instrument, as if at the surgical site. In other embodiments, the control device(s) may have more or fewer degrees of freedom than the associated medical instrument and still provide telepresence to the operator. In some embodiments, the control device(s) are manual input devices that can move in six degrees of freedom and may also include an actuatable handle for actuating the instrument (e.g., for closing a grasping clamp end effector, applying an electrical potential to an electrode, delivering a drug therapy, and actuating other types of instruments).

[0056] When the operator O views the surgical site through the operator input system 716, the component 712 can support and manipulate the medical device system 714. The image of the surgical site can be obtained by the endoscopic imaging system 715, which can be manipulated by the component 712. The component 712 can include the endoscopic imaging system 715, and can similarly include multiple medical device systems 714. The number of medical device systems 714 used at one time generally depends on factors such as the diagnostic or surgical procedure to be performed and the space limitations in the operating room. The component 712 can include one or more non-servo control links (e.g., one or more links that can be manually positioned and locked in place, generally referred to as a setting structure) and the kinematic structure of the manipulator. When the manipulator takes the form of a remotely operated manipulator, the component 712 is a remotely operated component. The component 712 includes multiple motors that drive inputs on the medical device system 714. In one embodiment, these motors move in response to commands from a control system (e.g., control system 720). The motor includes a drive system that, when coupled to the medical device system 714, can advance the medical device into a natural or surgically created anatomical orifice. Other motorized drive systems can move the distal end of the medical device in multiple degrees of freedom, which can include three degrees of freedom for linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three degrees of freedom for rotational motion (e.g., rotation about the X, Y, and Z Cartesian axes). In addition, a motor can be used to actuate an articulated end effector of the medical device to grasp tissue in a fixture of a biopsy device or similar device. The medical device of the medical device system 714 may include an end effector having a single working member (such as a scalpel, a blunt blade, an optical fiber, or an electrode). Other end effectors may include, for example, forceps, a grasper, scissors, or a clip applier.

[0057] The medical system 710 also includes a control system 720. The control system 720 includes at least one memory 724 and at least one processor 722 for implementing control between the medical device system 714, the operator input system 716, and other auxiliary systems 726, which may include, for example, imaging systems, audio systems, fluid delivery systems, display systems, lighting systems, steering control systems, irrigation systems, and / or suction systems. The clinician can circulate within the medical environment 711 and can access, for example, the components 712 during a setup procedure, or view a display of the auxiliary system 726 (e.g., display system 200) from the patient's bedside.

[0058] Despite Fig. 9712, in some embodiments, the control system 720 may be completely contained within the component 712. The control system 720 also includes programming instructions (e.g., stored on a non-transitory computer readable medium) to implement some or all of the methods described in accordance with aspects disclosed herein. Fig. 9 Although shown as a single block in the simplified schematic of , control system 720 may include two or more data processing circuits, with a portion of the processing optionally being performed on or near component 712 and another portion of the processing being performed at operator input system 716, and so on.

[0059] Any of a variety of centralized or distributed data processing architectures may be employed. Similarly, the programming instructions may be implemented as a plurality of separate programs or subroutines, or they may be integrated into a variety of other aspects of the systems described herein, including remote operating systems. In one embodiment, the control system 720 supports wireless communication protocols such as Bluetooth, IrDA (infrared data communication), HomeRF (home radio frequency), IEEE 802.11, DECT (digital enhanced communication system), and wireless telemetry.

[0060] In some embodiments, the control system 720 may include one or more servo controllers that receive force and / or torque feedback from the medical device system 714. In response to the feedback, the servo controller sends a signal to the operator input system 716. The (one or more) servo controllers may also send signals instructing the assembly 712 to move the (one or more) medical device systems 714 and / or the endoscopic imaging system 715, which extend through an opening in the body to an internal surgical site in the patient's body. Any suitable conventional or dedicated servo controller may be used. The servo controller may be separate or integrated with the assembly 712. In some embodiments, the servo controller and assembly 712 are provided as part of a teleoperated arm cart positioned near the patient's body.

[0061] The control system 720 can be coupled to the endoscopic imaging system 715 and can include a processor for processing the captured images for subsequent display, such as on an operator console or on another suitable display located locally and / or remotely to an operator. For example, where a stereoscopic endoscope is used, the control system 720 can process the captured images to present a coordinated stereoscopic image of the surgical site to the operator. Such coordination can include alignment between relative images and can include adjusting the stereoscopic working distance of the stereoscopic endoscope.

[0062] In alternative embodiments, the medical system 710 may include more than one component 712 and / or more than one operator input system 716. The exact number of components 712 will depend on the surgical procedure and space constraints within the operating room, as well as other factors. The operator input systems 716 may be collocated or located in separate locations. Multiple operator input systems 716 allow more than one operator to control one or more components 712 in various combinations. The medical system 710 may also be used to train and rehearse medical procedures.

[0063] Fig.10 712 is a perspective view of an embodiment of an assembly 712, which may be referred to as a patient side cart, a surgical cart, a teleoperated arm cart, a manipulator assembly, or a surgical robot. The assembly 712 shown provides manipulation of three surgical tools 730a, 730b, and 730c (e.g., a medical instrument system 714) and an imaging device 728 (e.g., an endoscopic imaging system 715), such as a stereo endoscope for capturing images of the site of the procedure. The imaging device may transmit signals to the control system 720 via a cable 756. Manipulation is provided by a teleoperated mechanism having multiple joints. The imaging device 728 and the surgical tools 730a-c may be positioned and manipulated through an incision in the patient's body, thereby maintaining a kinematic remote center at the incision to minimize the size of the incision. When the surgical tools 730a-c are positioned within the field of view of the imaging device 728, the image of the surgical site may include an image of the distal end of the surgical tools 730a-730c.

[0064] Assembly 712 includes a drivable base 758. Drivable base 758 is connected to a telescopic column 757, which allows the height of arm 754 to be adjusted. Arm 754 can include a swivel joint 755, which can rotate and move up and down. Each arm 754 can be connected to an orientation platform 753. Arm 754 can be marked to facilitate troubleshooting. For example, each arm 754 can be printed with different numbers, letters, symbols, other identifiers or combinations thereof. Orientation platform 753 can rotate 760 degrees. Assembly 712 can also include a telescopic horizontal cantilever 752 for moving orientation platform 753 in a horizontal direction.

[0065] In this example, each arm 754 is connected to a manipulator arm 751. Manipulator arm 751 can be directly connected to a medical instrument, such as one of surgical tools 730a-c. Manipulator arm 751 can be remotely operable. In some examples, an arm 754 connected to a directional platform 753 may not be remotely operable. Instead, such an arm 754 can be positioned as needed before operator O begins operating using the teleoperation assembly. Throughout the surgical procedure, medical instruments may be removed and replaced with other instruments, so that the association of instruments with arms may change during the procedure.

[0066] Endoscopic imaging systems (e.g., endoscopic imaging system 715 and imaging device 728) can be provided in various configurations, including rigid or flexible endoscopes. Rigid endoscopes include a rigid tube that houses a relay lens system for transmitting images from the distal end of the endoscope to the proximal end of the endoscope. Flexible endoscopes use one or more flexible optical fibers to transmit images. Digital image-based endoscopes have a "chip-on-the-tip" design in which a distal digital sensor (such as one or more charge-coupled devices (CCDs) or complementary metal oxide semiconductor (CMOS) devices) stores image data. Endoscopic imaging systems can provide two-dimensional or three-dimensional images to an observer. Two-dimensional images can provide limited depth perception. Three-dimensional stereoscopic endoscopic images can provide a more accurate depth perception to an observer. Stereoscopic endoscopic instruments use stereo cameras to capture stereoscopic images of a patient's anatomy. Endoscopic instruments can be a fully sterilizable assembly in which the endoscope cable, handle, and shaft are all rigidly coupled and hermetically sealed.

[0067] Fig.11 7 is a perspective view of an embodiment of an operator input system 716 at an operator console. The operator input system 716 includes a display system having a left eye display 732 and a right eye display 734 for presenting a coordinated stereoscopic view of a surgical environment that enables depth perception to the operator O. The left eye display 732 and the right eye display 732 can be components of a display system 735 (e.g., display system 200). In other embodiments, the display system 735 can include one or more other types of displays. The display system 735 can present images captured, for example, by the imaging system 715 to display the endoscopic field of view to the operator. The endoscopic field of view can be enhanced by virtual or synthetic menus, indicators, and / or other graphical or textual information to provide additional information to the observer.

[0068] The operator input system 716 also includes one or more input control devices 736, which in turn enable the assembly 712 to manipulate one or more instruments of the endoscopic imaging system 715 and / or the medical instrument system 714. The input control device 736 can provide the same degrees of freedom as the instrument with which it is associated to provide telepresence to the operator O, or provide a sense that the input control device 732 is integral with the instrument so that the operator has a strong sense of direct control of the instrument. To this end, position, force, and tactile feedback sensors (not shown) can be used to transmit position, force, and tactile sensations from the medical instrument (e.g., surgical tools 730a-c or imaging device 728) back to the operator's hand through the input control device 736. The input control device 739 is a foot pedal that receives input from the user's foot. Aspects of the operator input system 716, assembly 712, and auxiliary system 726 can be adjustable and customizable to meet the physical needs, skill level, or preferences of the operator O.

[0069] Whenever feasible, elements described in detail with reference to one embodiment, implementation, or application may optionally be included in other embodiments, implementations, or applications that are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment, but not described with reference to a second embodiment, the element may still be required to be included in the second embodiment. Therefore, in order to avoid unnecessary repetition in the following description, unless otherwise specifically described, unless one or more elements will make the embodiment or implementation inoperative, or unless two or more elements provide conflicting functions, one or more elements shown and described in association with one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects.

[0070] Any changes and further modifications to the described devices, systems, instruments, methods, and any further applications of the principles of the present disclosure are fully contemplated, as would be expected by a person skilled in the art of the present disclosure. In particular, it is fully contemplated that the features, components, and / or steps described with respect to one embodiment may be combined with the features, components, and / or steps described with respect to other embodiments of the present disclosure. In addition, the dimensions provided herein are for specific example purposes, and it is contemplated that different sizes, dimensions, and / or ratios may be utilized to implement the concepts of the present disclosure. In order to avoid unnecessary descriptive repetitions, one or more components or actions described according to an illustrative embodiment may be used or omitted depending on the applicable circumstances of other illustrative embodiments. For the sake of brevity, multiple iterations of these combinations will not be described separately.

[0071] Various systems and portions of systems have been described in terms of their states in three-dimensional space. As used herein, the term "position" refers to the position of an object or portion of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, Z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or portion of an object (three rotational degrees of freedom, such as roll, pitch, and yaw). As used herein, the term "pose" refers to the position of an object or portion of an object in at least one translational degree of freedom, as well as the orientation of the object or portion of an object in at least one rotational degree of freedom (up to six total degrees of freedom).

[0072] Although some of the examples described herein relate to surgical procedures or instruments, or medical procedures and medical instruments, the disclosed techniques are optionally applicable to non-medical procedures and non-medical instruments. For example, the instruments, systems, and methods described herein can be used for non-medical purposes, including industrial uses, general robotic uses, and sensing or manipulating non-tissue artifacts. Other example applications include cosmetic improvements, imaging of human or animal anatomical structures, collecting data from human or animal anatomy, and training medical or non-medical personnel. Additional example applications include procedures for tissues removed from human or animal anatomical structures (not returned to human or animal anatomy), and procedures on human or animal corpses. In addition, these techniques can also be used for surgical and non-surgical medical or diagnostic procedures.

[0073] A computer is a machine that follows programmed instructions to perform mathematical or logical functions on input information to produce processed output information. A computer includes logic units that perform mathematical or logical functions, and memory that stores programmed instructions, input information, and output information. The term "computer" and similar terms such as "processor" or "controller" or "control system" are similar.

[0074] Although certain exemplary embodiments of the present invention have been described and shown in the accompanying drawings, it should be understood that these embodiments are merely illustrative of the present invention rather than limiting thereof, and that the embodiments of the present invention are not limited to the specific constructions and arrangements shown and described, since various other modifications may be made by those skilled in the art.

Claims

1. A medical system, comprising: A display system, which is used to display a display area; and A control system, wherein the control system comprises a processing unit, the processing unit comprising one or more processors, and wherein the processing unit is configured to: generating an image of the field of view for display in the display area; generating a user interface component for display in the display area; determining a priority record of elements in the image of the field of view; determining a display characteristic of the user interface component based on the priority record of the element in the field of view; and The user interface component is displayed in the display area overlaid on a displayed image of the field of view in accordance with the display characteristic. 2 . The medical system of claim 1 , wherein the user interface component comprises at least one of a menu, an alert, an indicator, a digital tool, or an auxiliary image. 3 . The medical system of claim 1 , wherein the user interface component includes modified image data based on an image analysis of image content. The medical system of claim 1 , wherein the priority record includes a ranking indicator of the priority.

5. The medical system of claim 1, wherein the priority record is determined for anatomical structures in the image of the field of view.

6. The medical system of claim 1, wherein the priority record is determined for a tool in the image of the field of view.

7. The medical system of claim 1, wherein the priority record is dynamic based on activation detected in the field of view.

8. The medical system of claim 1, wherein determining a priority record of the element in the image of the field of view comprises distinguishing a plurality of elements in the image of the field of view, the plurality of elements comprising the element in the field of view.

9. The medical system of claim 8, wherein distinguishing the plurality of elements comprises identifying boundaries between the plurality of elements.

10. The medical system of claim 8, wherein distinguishing the plurality of elements comprises receiving a graphical segmentation of the image of the field of view.

11. The medical system of claim 8, wherein determining the priority record for the element in the image of the field of view comprises receiving an indication of a selection from a priority menu for the element. 12 . The medical system of claim 1 , wherein determining a priority record for the elements in the image of the field of view comprises receiving a priority assignment.

13. The medical system of claim 1, wherein determining a priority record for the elements in the image of the field of view comprises receiving a priority indication based on an image analysis of the image of the field of view.

14. The medical system of claim 13, wherein the image analysis comprises identification of model priority records from a model co-registered with the image of the field of view.

15. The medical system according to claim 1, wherein the field of view comprises a plurality of elements, the plurality of elements comprising the element as a first element and a second element, wherein the first element has a higher priority value than the second element, and Wherein displaying the user interface component having the display characteristic in the display area includes displaying the user interface component overlaying the second element based on a higher priority value of the first element.

16. The medical system of claim 1, wherein the processing unit is further configured to determine a procedure characteristic, and wherein determining the display characteristic is further based on the procedure characteristic.

17. The medical system of claim 16, wherein the procedure characteristic is a type of procedure performed in the field of view.

18. The medical system of claim 16, wherein the procedure characteristic is associated with an instrument in the image of the field of view.

19. The medical system of claim 1, wherein the display characteristic comprises a transparency level or a blur level of the user interface component.

20. The medical system of claim 1, wherein the display characteristic comprises an outline of edge boundaries of structures in the field of view that are obscured by the user interface component.

21. The medical system of claim 1, wherein the display characteristic is a duty cycle for intermittently displaying the user interface component.

22. The medical system of claim 1, wherein the display characteristic is a display duration of the user interface component.

23. The medical system of claim 1, wherein the display characteristic is a depth assignment.

24. The medical system of claim 1, wherein the display characteristic is a layer assignment.

25. The medical system of claim 1, wherein the display characteristic is an association with a structure in the field of view.

26. The medical system of claim 1, wherein the display characteristic is adapted to be dynamically updated based on kinematic information about an imaging system that generated the image.

27. The medical system of claim 1, wherein the display characteristics are adapted to be determined based on image analysis that the field of view has changed and to be dynamically updated.

28. The medical system of claim 1, wherein the display characteristic is adapted to be dynamically updated based on image analysis determinations of movement of structures in the field of view.

29. The medical system of claim 1, wherein displaying the user interface component comprises nudging the displayed user interface component based on receiving a user command.

30. The medical system of claim 1, wherein displaying the user interface component comprises displaying an alert if image analysis of the image detects a change in the appearance of a structure in the field of view.

31. The medical system of claim 1, further comprising: a second display system for displaying a second display area, wherein the display system is the first display system for displaying the first display area; and The processing unit is configured as follows: displaying the user interface component on the first display area at a first position relative to structures in the image of the field of view, and A user interface component is displayed on the second display area at a second position relative to a structure in the image of the field of view, wherein the first position relative to the structure is different than the second position relative to the structure.

32. The medical system of claim 1, wherein the processing unit is further configured to: A confirmation of the proposed change to the display characteristic of the user interface component is received.

33. The medical system of claim 1, wherein the processing unit is further configured to: receiving user input to move the user interface component to a position outside the field of view when the imaging device used to generate the image is in a first orientation; and An imaging device is adjusted to a second orientation, wherein a movable user interface component is within the image of the field of view when the imaging device used to generate the image is in the second orientation.

34. The medical system of claim 1, wherein the display characteristic is display orientation.

35. The medical system of claim 1, wherein the display characteristic is display orientation.

36. The medical system of claim 1, wherein the display characteristic includes a warning that an underlying structure is obscured.

Images