Systems and methods for anatomy segmentation and anatomical structure tracking

Abstract

A system and method for anatomical structure segmentation and anatomical structure tracking is provided. The system receives an image from a camera assembly of the system. The image may include a representation of one or more anatomical structures of a target. The system extracts a visual representation from the image. The system determines position and orientation data associated with a robotic assembly of the system. The position and orientation data indicates a pose of the robotic assembly. The system generates a state representation based at least in part on the pose and the visual representation. The state representation indicates a state of the system. The system identifies one or more anatomical landmarks in an anatomical space in which the robotic assembly is operating. The system generates a plurality of segmentation maps. Each segmentation map identifies which anatomical structures should avoid contact with the robotic assembly.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 430,513, filed December 6, 2022, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] Surgical robotic systems allow a user (also referred to herein as an "operator" or "user") to perform actions using robotic controls to perform tasks and functions during a procedure. However, the user (e.g., a surgeon) is still limited to visual feedback of the patient's interior. Such visual feedback can limit the user's ability to navigate and orient themselves within a small and often confusing operating room.

[0003] Navigation is typically performed using prior knowledge of the target area and anatomical landmarks. While prior knowledge can be acquired using significant preoperative planning time, recognizing anatomical landmarks can be a challenge if the user does not know the camera orientation relative to the patient or lacks a sufficient amount of expertise. This problem can significantly increase intraoperative time and the likelihood of potential injury, for example, by resecting the wrong structure. Summary of the Invention

[0004] A surgical robotic system is presented. The surgical robotic system includes a robotic assembly. The robotic assembly includes a camera assembly configured to generate one or more images (e.g., a series of video frames, live video footage, a series of photographs captured in burst mode, a single photograph, and / or other suitable images) of an internal cavity of a subject (e.g., a patient) and a robotic arm assembly positioned within the internal cavity to perform a surgical procedure. The surgical robotic system also includes a memory that stores one or more instructions and a processor configured or programmed to read the one or more instructions stored in the memory. The processor is operably coupled to the robotic assembly. The processor is configured to receive images from the camera assembly. The images may include representations of one or more anatomical structures of the subject (e.g., organs, organ ducts, blood vessels (arteries and veins), nerves, other delicate structures, and / or obstacle structures obstructing the robotic arm assembly). The processor is further configured to extract a visual representation thereof from the images. The visual representation is a compact representation (e.g., a vector representing the image by reducing the dimensions of the image to one or any other suitable representation that represents the image in a compact manner, such as reducing data size). The processor is further configured to determine position and orientation data associated with the robotic assembly. The position and orientation data indicates a pose of the robotic assembly. The processor is further configured to generate a pose representation of the robotic assembly based at least in part on the position and orientation data. The processor is further configured to generate a state representation based at least in part on the pose representation and the visual representation. The state representation indicates a state of the surgical robotic system. The processor is further configured to identify, based at least in part on the state representation, one or more anatomical landmarks (e.g., inguinal triangle, triangle of doom, triangle of pain, etc.) within an anatomical space in which the robotic assembly is operating. The processor is further configured to generate a plurality of segmentation maps. Each segmentation map identifies which of one or more anatomical structures should avoid contact with the robotic assembly.

[0005] In some embodiments, the processor is further configured to determine a similarity between the state representation and a previous state representation. The previous state representation is generated based at least in part on a previous pose representation extracted from the previous position and orientation data and a previous visual representation extracted from the previous image. In response to determining that the similarity is equal to or greater than a similarity threshold, the processor is further configured to average the state representation and the previous state representation to generate an averaged state representation. The processor is further configured to identify one or more anatomical landmarks based at least in part on the averaged state representation. The processor is further configured to generate a plurality of segmentation maps, each segmentation map identifying which of one or more anatomical structures should avoid contact with the robotic assembly. In some embodiments, the processor is further configured to determine a three-dimensional (3D) reconstruction of an anatomical space in which the robotic assembly is operating. The processor is further configured to determine a location of each of the one or more identified anatomical structures within the anatomical space based at least in part on the 3D reconstruction.

[0006] These and other features and advantages of the present invention will be more fully understood by reference to the following detailed description taken in conjunction with the accompanying drawings, in which like reference characters refer to like elements throughout the various views, illustrating the principles of the invention and showing relative dimensions, although not to scale, in which: [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an exemplary surgical robotic system, according to some embodiments. [Figure 2A] FIG. 2A is an exemplary perspective view of a patient cart including a robotic support system coupled to a robotic subsystem of a surgical robotic system, according to some embodiments. [Figure 2B]FIG. 2B is an exemplary perspective view of an exemplary operator console of the surgical robotic system of the present disclosure, according to some embodiments. [Figure 3A] FIG. 3A illustrates an exemplary side view of a surgical robotic system performing a surgical procedure within an internal cavity of a subject, according to some embodiments. [Figure 3B] FIG. 3B illustrates an exemplary top view of a surgical robotic system for performing a surgical procedure within the internal cavity of the object of FIG. 3A, according to some embodiments. [Figure 4A] FIG. 4A is an exemplary perspective view of a single robotic arm subsystem, according to some embodiments. [Figure 4B] FIG. 4B is an exemplary side perspective view of a single robotic arm of the single robotic arm subsystem of FIG. 4A, according to some embodiments. [Figure 5] FIG. 5 is a front perspective view of a camera assembly and a robotic arm assembly, according to some embodiments. [Figure 6] FIG. 6 is an exemplary graphical user interface of a robot pose view including a frustum view, a robot pose view, and a camera view of a patient cavity and a pair of robotic arms of a surgical robot system, according to some embodiments. [Figure 7] FIG. 7 illustrates an exemplary anatomy segmentation and tracking module for anatomy segmentation and anatomical structure tracking, according to some embodiments. [Figure 8] FIG. 8 is a flowchart illustrating steps for anatomy segmentation and anatomical landmark identification performed by a surgical robotic system, according to some embodiments. [Figure 9] FIG. 9 is a flowchart illustrating steps for anatomy tracking performed by a surgical robotic system, according to some embodiments. [Figure 10]FIG. 10 is a flow chart illustrating steps for training a surgical robotic system to automatically identify anatomical structures, according to some embodiments. [Figure 11A] FIG. 11A is a diagram of an example computing module that may be used to implement one or more steps of the method provided by the example embodiments. [Figure 11B] FIG. 11B illustrates exemplary system code that may be executable by the computing module of FIG. 11A, according to some embodiments. [Figure 12] FIG. 12 is a diagram illustrating computer hardware and network components on which the system may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0008] Embodiments taught and described herein provide systems and methods for anatomy segmentation and anatomical structure tracking. In some embodiments, the surgical robotic systems taught and described herein can provide an augmented view of a complex surgical environment by displaying anatomically relevant structures currently within the field of view of the surgical robotic system's camera assembly, such as by highlighting and tracking anatomical landmarks, organs, organ ducts, blood vessels (arteries and veins), nerves, other delicate structures, and obstructing structures (e.g., structures obstructing the robotic arm assembly) in a real-time video stream (e.g., live video footage). For example, the surgical robotic system can provide anatomy segmentation and tracking, including anatomical landmark identification, anatomical structure segmentation, and anatomical structure tracking, which can reduce the intraoperative time a user spends orienting themselves and increase the user's awareness of surgical stages. The anatomical structure segmentation and tracking taught and described herein can assist users during surgical procedures, thereby reducing the training time and skill required for users to use surgical robotic systems, thereby increasing the accessibility of surgical robotic systems to a wider range of users. The anatomical landmark identification taught and described herein can also enable advanced user interfaces that provide users with tools such as tissue and anatomical structure identification, shape navigation, and user-definable safety keepout zones to prevent unwanted tissue collisions. The anatomical structure segmentation taught and described herein can further enable surgical robotic systems to automatically define safety keepout zones that protect subjects from on- and off-camera damage, and can enable surgical robotic systems to become semi-autonomous surgical robotic systems that can interact with human tissue while navigating and performing surgical procedures within subjects, such as suturing and tissue dissection, safely and efficiently under user supervision.

[0009] In comparison to conventional anatomy segmentation systems and methods that are limited by the type of anatomy, the anatomically constrained region, the tracking environment, the tracking object, and the data modality, the anatomy segmentation and tracking taught and described herein uses an artificial intelligence (AI)-based framework (e.g., machine learning / deep learning models) to segment various types of anatomy (e.g., blood vessels, nerves, other delicate structures, obstacle structures, or the like) and can provide information about multiple anatomy simultaneously. The anatomy segmentation and tracking taught and described herein can also perform anatomy segmentation regardless of the location of the anatomy within the body by segmenting the anatomy at various anatomical locations. Additionally, the anatomy segmentation and tracking taught and described herein can track anatomy over time and simultaneously localize them in a reconstructed three-dimensional (3D) map of an internal body cavity. Furthermore, the anatomy segmentation and tracking taught and described herein can utilize multiple data modalities (e.g., robotic arm assembly pose information, visual information, dye and fluorescence data, or other suitable modality data) to perform anatomical landmark identification, anatomical structure segmentation, and anatomical structure tracking.

[0010] Before providing additional specific details of anatomy segmentation and tracking with respect to FIGS. 7-12, a surgical robotic system in which some embodiments may be employed is described below with respect to FIGS.

[0011] While various embodiments have been taught and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions may occur to those skilled in the art without departing from the invention. It will be understood that various alternatives to the embodiments of the invention described herein may be used.

[0012] As used in this specification and claims, the singular forms "a," "an," and "the" include plural referents unless the content clearly dictates otherwise. It will be further understood that the terms "comprises" and / or "comprising," or "include" and / or "including," as used herein, indicate the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The term "and / or," as used herein, includes any and all combinations of one or more of the associated listed items.

[0013] Unless specifically stated or apparent from the context, the term "about" as used herein is understood to mean within normal tolerances in the art, for example, within two standard deviations of the mean. "About" may be understood to mean within 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, 0.5%, 0.1%, 0.05%, or 0.01% of the stated value. Unless otherwise apparent from the context, all numerical values ​​provided herein are modified by the term "about."

[0014] While exemplary embodiments are described herein or in documents incorporated by reference as employing multiple units to perform exemplary processes, it is understood that exemplary processes may also be performed by one or more modules. Additionally, it is understood that the term controller / controller may refer to a hardware device, including a memory and a processor, specifically programmed to perform processes according to some embodiments described herein. In some embodiments, the memory is configured to store modules, and the processor is specifically configured to execute the modules to perform one or more processes described further below. In some embodiments, multiple different controllers or controllers, or multiple different types of controllers or controllers, may be employed to perform one or more processes. In some embodiments, different controllers or controllers may be implemented in different portions of the surgical robot system.

[0015] Surgical Robot System Some embodiments may be used with surgical robotic systems. A system for robotic surgery may include a robotic subsystem. The robotic subsystem includes at least a portion, which may be referred to herein as a robotic assembly, that may be inserted into a patient via a trocar through a single incision point or site. The portion inserted into the patient via the trocar is small enough to be deployed in vivo at a surgical site and is sufficiently maneuverable when inserted into the body so that it can be moved within the body to perform various surgical procedures at multiple different points or sites. Herein, the portion inserted into the body that performs a functional task may be referred to as a surgical robot module, a surgical robot module, or a robotic assembly. A surgical robot module may include multiple different sub-modules or components that may be separately inserted into a trocar. A surgical robot module, a surgical robot module, or a robotic assembly may include multiple separate robotic arms that can be deployed within a patient along different or separate axes. These multiple separate robotic arms may be collectively referred to herein as a robotic arm assembly. Additionally, a surgical camera assembly may also be deployed along a separate axis. The surgical robot module, surgical robot module, or robot assembly may also include a surgical camera assembly. Thus, a surgical robot module or robot assembly employs multiple distinct components, such as a pair of robotic arms and a surgical or robotic camera assembly, each of which is deployable along a different axis and separately operable, steerable, and movable. A robotic arm and camera assembly that are disposable along separate operable axes is referred to herein as a split-arm (SA) architecture. The SA architecture is designed to simplify and increase the efficiency of the insertion of robotic surgical instruments through a single trocar at a single insertion site, while also assisting in the deployment of the surgical instruments into a surgical-ready state and their subsequent removal through the trocar. As an example, surgical instruments can be inserted through the trocar to access a patient's abdominal cavity and perform surgery in vivo.In some embodiments, a variety of surgical instruments may be used or employed, including, but not limited to, robotic surgical instruments and other surgical instruments known in the art.

[0016] The systems, devices, and methods disclosed herein may be incorporated into and utilized in conjunction with, for example, the robotic surgical devices and related systems disclosed in U.S. Patent No. 10,285,765 and PCT Patent Application No. PCT / US2020 / 39203, the camera assemblies and systems disclosed in U.S. Patent Publication No. 2019 / 0076199, and / or the systems and methods for exchanging surgical tools in implantable surgical robotic systems disclosed in PCT Patent Application No. PCT / US2021 / 058820, the entire contents and teachings of which are incorporated herein by reference. A surgical robot module forming part of the present invention may, in some embodiments, form part of a surgical robotic system that includes a user workstation including appropriate sensors and displays, and a robotic support system (RSS) for supporting and interacting with the robotic subsystem of the present invention. The robotic subsystem, in some embodiments, includes a motor and a surgical robot module including one or more robotic arms and one or more camera assemblies. The robotic arm and camera assembly may form part of a single-support axis robotic system, a split-arm (SA) architecture robotic system, or other arrangements. The robotic support system may provide multiple degrees of freedom so that the robotic module can be maneuvered to a single position or multiple different positions within a patient. In one embodiment, the robotic support system may be attached directly to the operating table or to the floor or ceiling within the operating room. In another embodiment, attachment is achieved by various fastening means, including, but not limited to, clamps, screws, or combinations thereof. In other embodiments, the structure may be freestanding. The robotic support system may attach a motor assembly coupled to the surgical robot module, including the robotic arm assembly and camera assembly. The motor assembly may include gears, motors, drivetrains, electronics, etc., for powering the components of the surgical robot module.

[0017] The robotic arm assembly and camera assembly are capable of multiple degrees of freedom of movement. According to some embodiments, when the robotic arm assembly and camera assembly are inserted into a patient through a trocar, they are capable of movement in at least the axial, yaw, pitch, and roll directions. The robotic arm of the robotic arm assembly is designed to incorporate and utilize a multi-degree of freedom of movement robotic arm having an end effector attached to its distal end that corresponds to a user's wrist region or joint. In other embodiments, the working end (e.g., end effector end) of the robotic arm is designed to incorporate, use, or employ other robotic surgical instruments, such as, for example, the surgical instruments described in U.S. Publication No. 2018 / 0221102, the entire contents of which are incorporated herein by reference.

[0018] Similar numerical identifiers are used throughout the figures to refer to the same elements.

[0019] 1 is a schematic diagram of a surgical robotic system 10 in which aspects of the present disclosure may be employed according to some embodiments of the present disclosure. The surgical robotic system 10 includes an operator console 11 and a robotic subsystem 20 according to some embodiments.

[0020] The operator console 11 includes a display 12, an image computation module 14 (which may be a three-dimensional (3D) computation module), a hand controller 17 having a sensing and tracking module 16, and a computation module 18. Additionally, the operator console 11 may include a foot pedal array 19 including a plurality of pedals. The image computation module 14 may include a graphical user interface 39. The graphical user interface 39, the controller 26, or the image rendering device 30, or both, may provide one or more images or one or more graphical user interface elements on the graphical user interface 39. For example, pillar boxes associated with modes of operating the surgical robotic system 10 or any of the various components of the surgical robotic system 10 may be rendered on the graphical user interface 39. Live video footage captured by a camera assembly 44 may also be rendered on the graphical user interface 39 by the controller 26 or the image rendering device 30.

[0021] The operator console 11 may include a visualization system 9 including a display 12, which may be any selected type of display for displaying information, images, or video generated by the image computation module 14, the computation module 18, and / or the robotic subsystem 20. The display 12 may include or form part of, for example, a head-mounted display (HMD), an augmented reality (AR) display (e.g., an AR display or AR glasses combined with a screen or display), a screen or display, a two-dimensional (2D) screen or display, a three-dimensional (3D) screen or display, etc. The display 12 may also include an optional sensing and tracking module 16A. In some embodiments, the display 12 may include an image display for outputting images from a camera assembly 44 of the robotic subsystem 20.

[0022] The hand controller 17 is configured to sense the movement of the operator's hand and / or arm to operate the surgical robotic system 10. The hand controller 17 may include a sensing and tracking module 16, circuitry, and / or other hardware. The sensing and tracking module 16 may include one or more sensors or detectors that sense the movement of the operator's hand. In some embodiments, the one or more sensors or detectors that sense the movement of the operator's hand are disposed within the hand controller 17, which is grasped or engaged by the operator's hand. In some embodiments, the one or more sensors or detectors that sense the movement of the operator's hand are coupled to the operator's hand and / or arm. For example, the sensors of the sensing and tracking module 16 may be coupled to regions of the hand and / or arm, such as the fingers, wrist region, elbow region, and / or shoulder region. In some embodiments, additional sensors may also be coupled to the operator's head and / or neck region. In some embodiments, the sensing and tracking module 16 may be external and coupled to the hand controller 17 via electrical components and / or mounting hardware. In some embodiments, the optional sensor and tracking module 16A may sense and track movement of one or more of the operator's head, the operator's eyes, or at least a portion of the operator's neck based at least in part on imaging of the operator, in addition to or instead of sensors attached to the operator's body.

[0023] In some embodiments, the sensing and tracking module 16 may use sensors coupled to the operator's torso or any other body part. In some embodiments, the sensing and tracking module 16 may use, in addition to sensors, an inertial momentum unit (IMU) having, for example, an accelerometer, a gyroscope, a magnetometer, and a motion processor. The addition of a magnetometer may reduce sensor drift around the vertical axis. In some embodiments, the sensing and tracking module 16 also includes sensors placed within surgical materials such as gloves, surgical scrubs, or a surgical gown. The sensors may be reusable or disposable. In some embodiments, the sensors may be disposed external to the operator, such as in a fixed location in a room such as an operating room. The external sensors 37 may generate external data 36 that may be processed by the computing module 18 and thus used by the surgical robotic system 10.

[0024] The sensors generate position and / or orientation data indicative of the position and / or orientation of the operator's hands and / or arms. The sensing and tracking modules 16 and / or 16A may be utilized to control the movement (e.g., changes in position and / or orientation) of the camera assembly 44 and the robotic arm assembly 42 of the robot subsystem 20. The tracking and position data 34 generated by the sensing and tracking modules 16 may be communicated to the computing module 18 for processing by at least one processor 22.

[0025] The computing module 18 can determine or calculate the position and / or orientation of the operator's hands or arms, and in some embodiments, the operator's head, from the tracking and position data 34 and 34A and communicate the tracking and position data 34 and 34A to the robotic subsystem 20. The tracking and position data 34, 34A can be processed by the processor 22 and stored, for example, in the storage 24. The tracking and position data 34 and 34A can also be used by the controller 26, which can responsively generate control signals to control the movement of the robotic arm assembly 42 and / or the camera assembly 44. For example, the controller 26 can change the position and / or orientation of at least a portion of the camera assembly 44, at least a portion of the robotic arm assembly 42, or both. In some embodiments, the controller 26 can also adjust the pan and tilt of the camera assembly 44 to follow the movement of the operator's head.

[0026] The robotic subsystem 20 can include a robotic support system (RSS) 46 having a motor 40 and a trocar 50 or trocar mount, a robotic arm assembly 42, and a camera assembly 44. The robotic arm assembly 42 and camera assembly 44 can form part of a single support axis robotic unit as disclosed and described in U.S. Pat. No. 10,285,765, or can form part of a split-arm (SA) architecture robotic system as disclosed and described in PCT Patent Application No. PCT / US2020 / 039203, both of which are incorporated herein by reference in their entireties.

[0027] The robotic subsystem 20 may employ multiple distinct robotic arms deployable along different or separate axes. In some embodiments, a camera assembly 44, which may employ multiple distinct camera elements, may also be deployed along a common, separate axis. Thus, the surgical robot system 10 may employ multiple distinct components, such as a pair of distinct robotic arms and camera assemblies 44 deployable along different axes. In some embodiments, the robotic arm 42 and camera assembly 44 are independently operable, steerable, and movable. The robotic subsystem 20, including the robotic arm assembly 42 and camera assembly 44, is disposable along separate operable axes and is referred to herein as an SA architecture. The SA architecture is designed to simplify and increase the efficiency of insertion of robotic surgical instruments through a single trocar at a single insertion point or site, while also assisting in the deployment of the surgical instruments into a surgical-ready state and subsequent removal of the surgical instruments through the trocar 50, as further described below.

[0028] The RSS 46 may include a motor 40 and a trocar 50 or trocar mount. The RSS 46 may further include a support member supporting the motor 40 coupled to its distal end. The motor 40 may be coupled to each of the camera assembly 44 and the robotic arm assembly 42. The support member may be configured and controlled to move one or more components of the robotic subsystem 20 linearly or in any other selected direction or orientation. In some embodiments, the RSS 46 may be freestanding. In some embodiments, the RSS 46 may include a motor 40 coupled at one end to the robotic subsystem 20 and at an opposite end to an adjustable support member or element.

[0029] The motors 40 can receive control signals generated by the controller 26. The motors 40 can include gears, one or more motors, drive trains, electronics, etc., for powering and driving the robotic arm assembly 42 and the camera assembly 44, individually or together. The motors 40 can also provide mechanical power, electrical power, mechanical communications, and electrical communications to the robotic arm assembly 42, the camera assembly 44, and / or the RSS 46 and other components of the robotic subsystem 20. The motors 40 can be controlled by the computing module 18. Thus, the motors 40 can generate signals to control one or more motors that can control and drive the robotic arm assembly 42, including, for example, the position and orientation of each articulating joint of each arm, and the camera assembly 44. The motors 40 can further provide translational or linear degrees of freedom that are primarily utilized to insert and remove each component of the robotic subsystem 20 through the trocar 50. The motors 40 can also be employed to adjust the insertion depth of each robotic arm 42 of the robotic arm assembly as it is inserted into the patient 100 through the trocar 50.

[0030] The trocar 50 is a medical device that, in some embodiments, may consist of a claw (which may be a metal or plastic, sharp or non-bladed tip), a cannula (essentially a hollow tube), and a seal. The trocar 50 may be used to position at least a portion of the robotic subsystem 20 within an internal cavity of a subject (e.g., a patient) and may withdraw gases and / or fluids from the body cavity. The robotic subsystem 20 may be inserted through the trocar 50 to access the patient's body cavity and perform surgery in vivo. In some embodiments, the robotic subsystem 20 of the present invention may be supported, at least in part, by the trocar 50 or a trocar mount with multiple degrees of freedom so that the robotic arm assembly 42 and camera assembly 44 may be maneuvered within the patient to a single position or multiple different positions. In some embodiments, the robotic arm assembly 42 and camera assembly 44 may be supported by the trocar 50 or a trocar mount with multiple degrees of freedom so that the robotic arm assembly 42 and camera assembly 44 may be maneuvered within the patient to a single position or multiple different positions.

[0031] In some embodiments, RSS 46 may further include an optional controller for processing input data from one or more of the system components (e.g., display 12, sensing and tracking module 16, robotic arm assembly 42, camera assembly 44, etc.) and for generating control signals in response thereto. Motor 40 may also, in some embodiments, include a memory element for storing data.

[0032] In some embodiments, and in some modes of operation, the robotic arm assembly 42 may be controlled to follow scaled-down movements or motions of an operator's arm and / or hand, as sensed by associated sensors. The robotic arm assembly 42 includes a first robotic arm including a first end effector having an instrument tip disposed at the distal end of the first robotic arm, and a second robotic arm including a second end effector having an instrument tip disposed at the distal end of the second robotic arm. In some embodiments, the robotic arm assembly 42 may have portions or regions associated with movements that may be associated with shoulder, elbow, and wrist joints, as well as the fingers of an operator. For example, a robotic elbow joint may track the position and orientation of a human elbow, and a robotic wrist joint may track the position and orientation of a human wrist. The robotic arm assembly 42 may also have a terminal region associated therewith, which in some embodiments may terminate in an end effector that tracks the movement of one or more fingers of the operator, such as the index finger, when the user pinches the index finger and thumb together. In some embodiments, the robotic arm assembly 42 may follow the movement of the operator's arms in some control modes, while the virtual chest of the robotic assembly may remain stationary (e.g., in an instrument control mode). In some embodiments, the position and orientation of the operator's torso is subtracted from the position and orientation of the operator's arms and / or hands. This subtraction allows the operator to move their torso without the robotic arms moving. Further disclosure of control of the movement of the individual arms of the robotic assembly is provided in International Patent Applications WO 2022 / 094000 A1 and WO 2021 / 231402 A1, each of which is incorporated herein by reference in its entirety.

[0033] The camera assembly 44 is configured to provide the operator with image data 48, such as a live video feed of the procedure or surgical site, as well as to allow the operator to operate and control the cameras forming part of the camera assembly 44. In some embodiments, the camera assembly 44 may include one or more cameras (e.g., a pair of cameras) whose optical axes are axially spaced a selected distance apart, known as the inter-camera distance, to provide a stereoscopic view or image of the surgical site. In some embodiments, the operator can control camera movement through hand movement, via a sensor coupled to the operator's hand or via a hand controller 17 grasped or held by the operator's hand, thus allowing the operator to obtain a desired view of the surgical site in an intuitive and natural manner. In some embodiments, the operator can additionally control camera movement through movement of the operator's head. The camera assembly 44 is movable in multiple directions, including, for example, yaw, pitch, and roll, relative to the direction of view. In some embodiments, the stereoscopic camera components may be configured to provide a natural and comfortable user experience. In some embodiments, the axial distance between the cameras can be modified to adjust the depth of the surgical site as perceived by the operator.

[0034] Image or video data 48 generated by camera assembly 44 may be displayed on display 12. In embodiments, if display 12 includes an HMD, the display may include an embedded sensing and tracking module 16A that acquires raw orientation data in the yaw, pitch, and roll directions of the HMD, as well as position data in Cartesian space (x, y, z) of the HMD. In some embodiments, position and orientation data for the operator's head may be provided via a separate head tracking module. In some embodiments, sensing and tracking module 16A may be used to provide supplemental position and orientation tracking data for the display instead of, or in addition to, the HMD's embedded tracking system. In some embodiments, operator head tracking is not used or employed. In some embodiments, an image of the operator may be used by sensing and tracking module 16A to track at least a portion of the operator's head.

[0035] 2A shows an exemplary robotic assembly 20 (also referred to herein as a robotic subsystem) of a surgical robotic system 10 integrated into or mounted on a mobile patient cart, according to some embodiments. In some embodiments, the robotic subsystem 20 includes an RSS 46, which in turn includes motors 40, a robotic arm assembly 42 having an end effector 45, a camera assembly 44 having one or more cameras 47, and may also include a trocar 50 or a trocar mount.

[0036] 2B shows an example of an operator console 11 of the presently disclosed surgical robotic system 10, according to some embodiments. The operator console 11 includes a display 12, a hand controller 17, and one or more additional controllers, such as a foot pedal array 19, for controlling the robotic arm assembly 42, the camera assembly 44, and other aspects of the system.

[0037] 2B also illustrates the left and right hand controller subsystems 23A and 23B of the operator console. The left hand controller subsystem 23A includes and supports the left hand controller 17A, and the right hand controller subsystem 23B includes and supports the right hand controller 17B. In some embodiments, the left hand controller subsystem 23A can be removably connected or engaged with the left hand controller 17A, and the right hand controller subsystem 23B can be removably connected or engaged with the right hand controller 17A. In some embodiments, the connections can be both physical and electronic, such that the left and right hand controller subsystems 23A and 23B can receive signals from the left and right hand controllers 17A and 17B, respectively, including signals conveying input received from user selections on buttons or touch input devices of the left or right hand controller 17A or 17B.

[0038] Each of the left hand controller subsystem 23A and the right hand controller subsystem 23B can include components that enable a range of motion for the respective left hand controller 17A and right hand controller 17B, such that the left hand controller 17A and the right hand controller 17B can translate or displace in three dimensions and can additionally move in roll, pitch, and yaw directions. Additionally, each of the left hand controller subsystem 23A and the right hand controller subsystem 23B can register the movement of the respective left hand controller 17A and right hand controller 17B in each of the aforementioned directions and can send signals providing such movement information to the processor 22 (as shown in FIG. 1 ) of the surgical robotic system 10.

[0039] In some embodiments, each of the left hand controller subsystem 23A and the right hand controller subsystem 23B may be configured to receive and connect to or engage a different hand controller (not shown). For example, hand controllers having different configurations of buttons and touch input devices may be provided. In addition, hand controllers having different shapes may be provided. The hand controllers may be selected to be compatible with a particular surgical robotic system or a particular surgical robotic procedure, or may be selected based on operator preferences for buttons and input devices or for hand controller shape to provide greater comfort and ease for the operator.

[0040] FIG. 3A schematically illustrates a side view of the surgical robotic system 10 performing surgery within an internal cavity 104 of an object 100, according to some embodiments and for some surgical procedures. FIG. 3B schematically illustrates a perspective top view of the surgical robotic system 10 performing surgery within an internal cavity 104 of an object 100. The object 100 (e.g., a patient) is positioned on a surgical table 102 (e.g., surgical table 102). In some embodiments and for some surgical procedures, an incision is made in the patient 100 to gain access to the internal cavity 104. A trocar 50 is then inserted into the patient 100 at a selected location to provide access to the internal cavity 104 or surgical site. The RSS 46 can then be manipulated into position on the patient 100 and the trocar 50. In some embodiments, the RSS 46 includes a trocar mount that couples to the trocar 50. The camera assembly 44 and the robotic arm assembly 42 are coupled to the motor 40 and may be inserted individually and / or sequentially through the trocar 50 into the patient 100, and thus into the internal cavity 104 of the patient 100. While the camera assembly 44 and the robotic arm assembly 42 may include portions that remain external to the subject's body during use, references to inserting the robotic arm assembly 42 and / or the camera assembly 44 into the internal cavity of the subject and disposing the robotic arm assembly 42 and / or the camera assembly 44 within the internal cavity of the subject refer to the portions of the robotic arm assembly 42 and the camera assembly 44 that are intended to be within the internal cavity of the subject during use. The sequential insertion method has the advantage of supporting smaller trocars, thus allowing for smaller incisions to be made in the patient 100, thus reducing trauma experienced by the patient 100. In some embodiments, the camera assembly 44 and the robotic arm assembly 42 may be inserted in any order or in a specific order. In some embodiments, the camera assembly 44 may be followed by a first robotic arm 42A of the robotic arm assembly 42, followed by a second robotic arm 42B of the robotic arm assembly 42, all of which may be inserted into the trocar 50 and thus into the internal cavity 104.Once inserted into the patient 100 , the RSS 46 can move the robotic arm assembly 42 and camera assembly 44 to the surgical site controlled manually or automatically by the operator console 11 .

[0041] Disclosures regarding management of movement of individual arms of a robotic arm assembly are provided in International Patent Application Publication Nos. 2022 / 094000A1 and 2021 / 231402A1, each of which is incorporated herein by reference in its entirety.

[0042] 4A is a perspective view of the robotic arm subassembly 21, according to some embodiments. The robotic arm subassembly 21 includes a robotic arm 42A, an end effector 45 having an instrument tip 120 (e.g., monopolar scissors, a needle driver / holder, a bipolar grasper, or any other suitable tool), and a shaft 122 that supports the robotic arm 42A. A distal end of the shaft 122 is coupled to the robotic arm 42A, and a proximal end of the shaft 122 is coupled to the housing 124 of the motor 40 (as shown in FIG. 2A). At least a portion of the shaft 122 can be external to the internal cavity 104 (as shown in FIGS. 3A and 3B). At least a portion of the shaft 122 can be inserted into the internal cavity 104 (as shown in FIGS. 3A and 3B).

[0043] 4B is a side view of the robotic arm assembly 42. According to some embodiments, the robotic arm assembly 42 includes a shoulder joint 126 that forms a virtual shoulder, an elbow joint 128 having a position sensor 132 (e.g., a capacitive proximity sensor) that forms a virtual elbow, a wrist joint 130 that forms a virtual wrist, and an end effector 45. In some embodiments, the shoulder joint 126, elbow joint 128, and wrist joint 130 can include a series of hinges and revolute joints to provide seven positionable degrees of freedom for each arm, along with an additional grasping degree of freedom for the end effector 45.

[0044] 5 illustrates a perspective front view of a portion of robotic assembly 20 configured for insertion into a patient's internal body cavity. Robotic assembly 20 includes robotic arm 42A and robotic arm 42B. The two robotic arms 42A and 42B may, in some embodiments, define a virtual chest 140 of robotic arm 20. In some embodiments, virtual chest 140 may be defined by a chest plane extending between a first pivot point 142A of the most proximal joint of robotic arm 42A (e.g., shoulder joint 126), a second pivot point 142B of the most proximal joint of robotic arm 42B, and a camera imaging center point 144 of camera 47. A pivot center 146 of virtual chest 140 is at the center of virtual chest 140.

[0045] In some embodiments, sensors on one or both of the robotic arms 42A and 42B may be used by the system 10 to determine a change in location in three-dimensional space of at least a portion of each or both of the robotic arms 42A and 42B. In some embodiments, sensors on one or both of the first robotic arm 42A and the second robotic arm 42B may be used by the surgical system 10 to determine a location in three-dimensional space of at least a portion of one robotic arm relative to a location in three-dimensional space of at least a portion of the other robotic arm.

[0046] In some embodiments, the camera assembly 44 is configured to acquire images that enable the surgical robotic system 10 to determine relative positions in three-dimensional space. For example, the camera assembly 44 can include multiple cameras, at least two of which are laterally offset from one another relative to an imaging axis, and the system can be configured to determine distances to features within an internal body cavity. Further disclosure regarding surgical robotic systems including camera assemblies and associated systems for determining distances to features can be found in International Patent Application Publication No. WO 2021 / 159409, entitled "System and Method for Determining Depth Perception In Vivo in a Surgical Robotic System," published August 12, 2021, and incorporated herein by reference in its entirety. Information regarding distances to features and information regarding optical properties of the cameras can be used by the system to determine relative positions in three-dimensional space.

[0047] 6 illustrates graphical user interface 150 formatted to include left and right pillarboxes 198, 199, respectively, to the left and right of live video feed 168 of a patient's cavity. Graphical user interface 150 may be overlaid on live video feed 168. In some embodiments, live video feed 168 is formatted by controller 26 to accommodate left and right pillarboxes 198, 199. In some embodiments, live video feed 168 may be displayed on display 12 at a predetermined size and location on display 12, and left and right pillarboxes 198, 199 may be displayed on either side of live video feed 168 at a particular size based on the remaining area on display 12 not occupied by live video feed 168. Graphical user interface 150 includes several different graphical user interface elements, which are described in more detail below.

[0048] The robotic arms 42B and 42A are also visible in the live video feed. The left pillarbox 198 may include a state identifier 173, for example, an engaged or disengaged state identifier associated with the instrument tip 120 of the robotic arm 42B. The “engaged” state identifier 173 indicates that the user's left hand and arm are engaged with the left hand controller 201, and therefore the instrument tip 120 is also engaged. The “disengaged” state identifier 173 indicates that the user's left hand and arm are not engaged with the hand controller 201, and therefore the instrument tip 120 is also disengaged. When the user's left hand and arm are disengaged from the left hand controller 201, the surgical robotic system 10 can be completely disengaged. That is, the surgical robotic system 10 may remain on but will not respond until the user's hand re-engages the hand controller. The instrument tip 120 may be represented by a symbol 179 containing the name of the instrument tip 120 to provide the user with confirmation of what type of end effector or instrument tip is currently being used. 6, the instrument tip 120, represented by iconographic symbol 179, is a bipolar grasper. Notably, the present disclosure is not limited to the bipolar graspers or scissors shown in FIG.

[0049] Similarly, the right pillar box 199 may include a state identifier 175, such as an engaged or disengaged state identifier, associated with the instrument tip 120 of the robotic arm 42A. In some embodiments, based on the state of the end effector, the graphical user interface may also provide a visual representation of the state in addition to text. For example, the end effector iconography may be "grayed out" or made less prominent if it is disengaged.

[0050] The state identifier 175 may be "engaged," thereby indicating that the user's right hand and arm are engaged with the right hand controller 202, and therefore the instrument tip 120 is also engaged. Alternatively, the state identifier 175 may be "disengaged," thereby indicating that the user's right hand and arm are not engaged with the right hand controller 202, and therefore the instrument tip 120 is also disengaged. The instrument tip 120 may be represented by a graphical symbol 176 that includes the name of the instrument tip 120 to provide the user with confirmation of what type of end effector or instrument tip is currently being used. In FIG. 6 , the instrument tip 120 represented by the graphical symbol 176 is a monopolar scissors. Notably, the present disclosure is not limited to the monopolar scissors shown in FIG. 6 .

[0051] The left pillarbox 198 may also include a robot pose view 171. The robot pose view 171 includes a simulated view of the robot arms 42B and 42A, the camera assembly 44, and the support arm, thereby allowing the user to obtain a third-person perspective of the robot arm assembly 42, the camera assembly 44, and the robot support system 46. A simulated view of the robot arms 42B and 42A is represented by a pair of simulated robot arms 191 and 192. A simulated view of the camera assembly 44 is represented by a simulated camera 193. The robot pose view 171 also includes a simulated camera view associated with the patient's cavity or portion of the cavity, which represents the placement or location of the pair of robot arms 151 and 172 relative to the frustum 151. More specifically, the camera view may be the field of view of the camera assembly 44, which is equivalent to the frustum 151.

[0052] The right pillarbox 199 may also include a robot pose view 172 that includes a simulated view of the robot arms 42B and 42A, the camera assembly 44, and the support arm, thereby allowing the user to obtain a third-person perspective of the robot arm assembly 42, the camera assembly 44, and the support arm. The simulated view of the robot arms 42B and 42A is a pair of simulated robot arms 165 and 166. The simulated view of the camera assembly 44 is represented by a simulated camera 193. The robot pose view 172 also includes a simulated camera view associated with the patient cavity or portion of the cavity, which is the placement or location of the pair of robot arms 165 and 166 relative to a frustum 167. More specifically, the camera view may be the field of view of the camera, which is the frustum 167. The robot pose view 172 provides elbow height awareness and situational awareness, especially when maneuvering in a face-up / face-down configuration.

[0053] Situational awareness can be characterized as a way of understanding particular robotic elements with respect to time and space when robotic arms 42A and 42B are inside the patient cavity. For example, as shown in robot pose view 171, the elbow of simulated robotic arm 192 is bent downward, thereby providing the user with the ability to know how the elbow of actual robotic arm 42A is actually oriented and positioned within the patient cavity. Note that due to the positioning of camera assembly 44 relative to robotic arms 42A and 42B, the full length of robotic arms 42A and 42B may not be visible in live video footage 168. As a result, the user may not have visualization of how robotic arms 42A and 42B are oriented and positioned within the patient cavity. Simulated robotic arms 165 and 166, as well as simulated robotic arms 191 and 192, provide the user with situational awareness of at least the position and orientation of actual robotic arms 42A and 42B within the patient cavity.

[0054] There may be two separate views (robot pose view 171 and robot pose view 172) from two different perspectives on each side of the graphical user interface 150 presented on the display 12. The robot pose views 171 and 172 are automatically updated to stay centered on the trocar 50 while keeping the robot arms 42A and 42B in view. The robot pose views 171 and 172 also provide spatial awareness to the user.

[0055] Spatial awareness may be characterized as the placement or location of robot arms 42A and 42B when viewed in robot pose views 171 and 172 relative to other objects within the cavity and the cavity itself. Robot pose views 171 and 172 provide the user with the ability to determine where the actual robot arms 42A and 42B are located within the cavity by viewing simulated robot arms 191 and 192 in robot pose view 171 and simulated robot arms 165 and 166 in robot pose view 172. For example, robot pose view 171 shows the position and location of simulated robot arms 191 and 192 relative to frustum 151. Robot pose view 171 depicts simulated robot arms 191 and 192 relative to frustum 151 from a side view of the support arm and the simulated robot arms 191 and 192 attached to the support arm. This particular robot pose provides the user with the ability to better ascertain proximity to anatomical features within the cavity.

[0056] The robot pose view 172 can also provide the user with the ability to better ascertain how close the actual robot arms 42A and 42B are to one another or how far apart they are from one another. Furthermore, the robot pose view 172 can also show where the actual robot arms 42A and 42B can be positioned or located relative to the interior of the patient's cavity to the left and right of the robot arms 42A and 42B, thereby providing the user with spatial awareness of where the robot arms 42A and 42B are within the cavity and where they are relative to anatomical features within the cavity. As mentioned above, because the full length of the robot arms 42A and 42B is not visible in the live video feed 168, the simulated robot arms 165 and 166 can provide the user with spatial awareness of how close or far apart the actual robot arms 42A and 42B are to one another. The view provided by the robot pose view 172 is a view that appears as if the user is looking into the interior of the cavity. The robot pose view 172 provides the user with spatial awareness to know how close the virtual elbows 128 are to each other and how close the actual robot arms 42A and 42B are to each other when the user manipulates the right hand controller 202 and the left hand controller so that the virtual elbows 128 move closer to each other. For example, if the user manipulates the left hand controller 201 and the right hand controller 202 to straighten the robot arms 42A and 42B, the simulated robot arms 166 and 165 will become parallel to each other and the distance between the elbows of the simulated robot arms 165 and 166 will decrease. Conversely, if the user manipulates the left hand controller 201 and the right hand controller 202 to bend the robot arms 42A and 42B, the result will be that the virtual elbows 128 of the robot arms 42A and 42B will move further apart, the simulated robot arms 166 and 165 will no longer be parallel to each other, and the distance between the elbows of the simulated robot arms 165 and 166 will increase. Because the live video feed 168 does not provide full length visualization of the robotic arms 42A and 42B, the robot pose views 171 and 172 provide the user with spatial awareness during the surgical procedure.

[0057] 6, simulated robotic arms 191 and 192 are shown within the field of view of camera assembly 14 associated with frustum 151, which provides the user with situational and spatial awareness of where robotic arms 42B and 42A are located or positioned within a captured portion of the patient's actual cavity. The camera view associated with robot pose view 171 is a simulated view of robotic arms 42B and 42A as if the user were looking at an actual view of robotic arms 42B and 42A from a side view within the patient's cavity. As noted above, the camera view can be the field of view of camera assembly 44, which is frustum 167. That is, robot pose view 171 provides the user with a side view of simulated robotic arms 191 and 192, which are simulated views corresponding to robotic arms 42B and 42A, respectively.

[0058] In some embodiments, the graphical user interface 150 can display live video feed 168 from a single viewpoint that includes the field of view of the cavity and the robotic arms 42B and 42A for different regions within the cavity as shown in Figure 6. As a result, the user may not always be able to determine how the virtual elbows 128 of the robotic arms 42B and 42A are positioned because the camera assembly 44 may not always include video feed of the virtual elbow 128 of the robotic arm 42B and video feed of the elbows of the robotic arms 42A, and therefore the user may not be able to determine how to adjust the right hand controller 202 and the left hand controller 201 if they wish to operate within the patient's cavity. Because the left situation awareness camera view panel includes a simulated view of the entire length of the robot arms 191 and 192, the simulated view of the robot arm 42B (robot arm 191) and the simulated view of the robot arm 42A (robot arm 192) provide the user with a perspective that enables the user to determine the positioning of the virtual elbows 128 of the robot arms 42A and 42B. Because the simulated view of the robot pose view 171 includes views of the virtual elbows 128 of the robot arms 191 and 192, the user can adjust the positioning of the robot arms 42B and 42A by manipulating the left hand controller 201 and the right hand controller 202 and carefully watching how the robot arms 191 and 192 move in accordance with the manipulation of the left hand controller 201 and the right hand controller 202.

[0059] The graphical user interface 150 may include a robot pose view 172 within which is a frustum 167 that is the field of view of the camera assembly 44 associated with a portion of the patient's cavity, and within which are the simulated camera 158 and the robot arms 165 and 166 having a simulated robot support arm supporting the robot arms 165 and 166.

[0060] 6 , simulated robotic arms 165 and 166 are shown within frustum 167, which represents the location and orientation of robotic arms 42B and 42A within the patient's actual cavity. The view shown in robot pose view 172 is a simulated view of robotic arms 42B and 42A as if the user were looking at robotic arms 42B and 42A from a top-down view within the patient's cavity. That is, robot pose view 172 provides the user with a top-down view of simulated robotic arms 165 and 166, which are corresponding simulated views of robotic arms 42B and 42A, respectively. The top-down view provides the user with the ability to maintain a level of situational awareness of robotic arms 42B and 42A as they perform a procedure within the cavity. Because the robot pose view 172 includes a simulated top-down view of the robot arms 165 and 166, the camera 158, and the support arms of the robot assembly, the view of the simulated robot arm 165 corresponding to the robot arm 42B and the view of the simulated robot arm 166 corresponding to the robot arm 42A provide a user with an overhead perspective that enables the positioning of the robot arms 42B and 42A to be determined. Because the simulated field of view of the camera assembly 44, outlined by the frustum 167, includes a top-down view of the simulated robot arms 165 and 166, the user can adjust the positioning of the robot arms 42B and 42A by manipulating the left hand controller 201 and the right hand controller 202 and noting how the simulated robot arms 165 and 166 move forward or backward within the portion of the cavity within the frustum 167 in accordance with the manipulation of the left hand controller 201 and the right hand controller 202.

[0061] The simulated views of the robot arms 42B and 42A in the robot pose views 171 and 172 are automatically updated to keep the robot arms 42B and 42A in view while remaining centered on the trocar 50. In some embodiments, this can be achieved based on one or more sensors from the sensing and tracking module 16 on the robot arms 42B and 42A, which provide information to the right hand controller 202 and the left hand controller 201. The sensors can be encoders or Hall effect sensors or other suitable sensors.

[0062] Anatomy Segmentation and Tracking The anatomy segmentation and tracking described herein can be used in conjunction with any of the surgical robotic systems described above, or any other suitable surgical robotic system. Additionally, some embodiments described herein can be employed with semi-robotic endoscopic surgical systems that are only partially robotic.

[0063] Examples of the anatomy segmentation and tracking taught herein may be understood with reference to the embodiments shown in Figures 7-12 described below. For convenience, unless otherwise noted, like reference numerals will be used to refer to similar features of the various embodiments shown in the drawings.

[0064] FIG. 7 illustrates an exemplary anatomy segmentation and tracking module 300 for anatomy segmentation and anatomical structure tracking, according to some embodiments. The anatomy segmentation and tracking module 300 may be executed by the processor 22 of the computing module 18 of the surgical robotic system 10. In some embodiments, the anatomy segmentation and tracking module 300 may be part of the system code 2000 as described with respect to FIGS. 11A and 11B . The anatomy segmentation and tracking module 300 may include a machine learning model 320, which is provided by input 310 and generates output 340, a confidence module 346, and a tracking module 360. In some embodiments, the anatomy segmentation and tracking module 300 may include different components as shown in FIG. 7 . For example, the anatomy segmentation and tracking module 300 may further include a training module for training the machine learning model. Some embodiments may include the confidence module 346, while some embodiments may not include the confidence module 346.

[0065] The input 310 includes position and orientation data 312 indicating the pose of the robot assembly 20, which includes the robotic arm assembly 42 and the camera assembly 44. For example, the surgical robot system 10 can determine the position and orientation data 312 using sensors coupled to the robotic arm assembly 42 and the camera assembly 44. With reference to FIGS. 1 and 4-6, the robotic arm assembly 42 can include sensors (e.g., sensor 132 in FIG. 4B) that detect the position and orientation (e.g., roll, pitch, yaw) of one or more portions of each of the robotic arms (e.g., various joints, end effectors, or other portions) in 3D space (e.g., within the abdominal cavity of a patent). Sensors from the sensing and tracking module 16 on the robotic arms 42B and 42A and the camera assembly 44 can detect the position and orientation data 312 of the robotic arms 42B and 42A and the camera assembly 44.

[0066] The input 310 may also include an image 314 including a representation of one or more anatomical structures of the subject. In some embodiments, as shown in FIG. 6 , an image 168 (e.g., a video frame of a live video stream) captured by the camera assembly 44 within the field of view of the camera assembly 44 may include an incarcerated hernia 200 and a blood vessel 210 within an anatomical space 220 (e.g., the abdominal cavity) of the subject (e.g., a patient). For example, the anatomy segmentation and tracking module 300 may access the video stream from the camera assembly 44 in real time to obtain video showing content within the surgical field (e.g., the field of view of the camera assembly 44), which may include organs, vasculature, and other tissues. The anatomy segmentation and tracking module 300 may decompose the obtained video into a series of video frames recorded at a particular rate (e.g., 30 Hz). The anatomy segmentation and tracking module 300 may input each video frame to a machine learning model 320, as described below. In some embodiments, image 314 may be a single photograph or one of a series of photographs captured in burst mode, or other suitable image depicting the subject's anatomy.

[0067] In some embodiments, input 310 may also include data from other systems, such as fluorescence data (e.g., from indocyanine green (ICG) imaging and / or from a laparoscopic system), 3D data or maps of the environment captured by camera assembly 44, and / or other suitable data associated with the internal body cavity in which surgical robotic system 10 is operating. Notably, the present disclosure is not limited to the input data described herein.

[0068] The machine learning model 320 may include an encoder-decoder architecture and a classifier architecture. The encoder-decoder architecture may include a pose encoder 322, a visual encoder 326, and a decoder 322. The classifier architecture may include a classifier 324 (e.g., a multi-class classifier). The machine learning model 320 may segment multiple types of anatomical structures (e.g., blood vessels, nerves, other vessel-like structures, organs, other delicate structures, and / or obstruction structures) within the surgical field (e.g., the field of view of the camera assembly 44) across various anatomical locations and may identify anatomical landmarks (e.g., in real time) as described below.

[0069] In some embodiments, machine learning model 320 may include one or more neural networks (e.g., a convolutional neural network or other type of neural network). Pose encoder 322, visual encoder 326, decoder 322, and classifier 324 may be part of a single neural network or may be different neural networks. In some embodiments, machine learning model 320 may be generated using a training process based on supervised learning, semi-supervised learning, unsupervised learning, reinforcement learning, and / or other suitable learning methods.

[0070] The pose encoder 322 is fed by the position and orientation data 312 of the robot assembly 20 and can extract a compact representation of the pose of the robot assembly 20 from the position and orientation data 312. The extracted compact representation can be referred to as a pose representation 324. The extracted compact representation can be a vector representing the pose of the robot assembly 20 by reducing the dimensionality of the position and orientation data 312, or any other suitable representation that represents the position and orientation data 312 in a compact manner (e.g., reduced data size, etc.).

[0071] The visual encoder 326 is fed by the image 314 and may extract from the image 314 a compact representation of the image 314. The extracted compact representation may be referred to as a visual representation 328. The compact representation may be a vector that represents the image by reducing the dimensionality of the image 314, or any other suitable representation that represents the image in a compact manner (e.g., reducing data size, etc.).

[0072] The machine learning model 320 can aggregate the pose representations 324 and the visual representations 328 into a single state representation 330 that reflects the current state of the robotic assembly 20 and can be used to accomplish multi-subject tasks, such as identifying anatomical landmarks and delineating delicate structures within a surgical field. In some embodiments, the machine learning model 320 can aggregate the pose representations 324 and the visual representations 328 by averaging the representations and weighing each representation equally. In some embodiments, the machine learning model 320 can aggregate the pose representations 324 and the visual representations 328 using a weighted average of the representations. The weight of each representation is determined according to the relative importance of each data modality. For example, if the content of the image 314 changes only slightly as the robotic assembly 20 moves, the pose representation 324 may have a greater weight than the visual representation 328. In some embodiments, the machine learning model 320 can concatenate the representations such that the size of the state representation 330 increases with the addition of each representation.

[0073] In some embodiments, state representation 330 is extensible in that it can describe any number of modality-specific representations and is not limited to pose representation 324 and visual representation 328. For example, machine learning model 320 may include more encoders to extract respective compact representations from additional inputs, such as fluorescence data, 3D map data, and / or other suitable data, as described with respect to input 310.

[0074] The decoder 332 is fed by the state representation 330 and can generate multiple segmentation maps 342. Each segmentation map 342 can identify which of one or more anatomical structures should avoid contact with the robotic arm assembly 42. In some embodiments, the decoder 332 can identify multiple anatomical structures. For example, there may be multiple sensitive structures that the surgeon wants to keep track of. For example, the surgeon may want to avoid damaging a nerve (anatomical structure 1) while being guided along an artery (anatomical structure 2). To enable segmentation of multiple anatomical structures, each decoder 332 can be trained to identify a particular anatomical structure (e.g., nerve, artery, etc.). Each decoder 332 can output a probability that each pixel of the image 314 represents a particular anatomical structure. Pixels whose value exceeds a value threshold can be considered to depict that anatomical structure within the field of view. For example, if there are N sensitive structures, the machine learning model 320 can have N segmentation decoders. In some embodiments, a single decoder can generate multiple segmentation maps 342.

[0075] The classifier 334 is fed by the state representation 330 and can identify one or more anatomical landmarks within the anatomical space in which the robotic assembly 20 is operating. Examples of anatomical landmarks can include the inguinal triangle, which refers to an area of ​​the abdominal wall, the triangle of doom, which refers to an anatomical triangle bounded medially by the vas deferens, laterally by the testicular artery and vein, and inferiorly by the peritoneal folds, and the triangle of pain, which is the area bounded by the iliopubic cord, the testicular artery and vein, and the peritoneal folds.

[0076] In some embodiments, the classifier 334 can output a probability that the area viewed by the camera assembly 44 belongs to one of multiple anatomical landmarks. For example, the classifier 334 can be a multi-class classifier for classifying the area viewed by the camera assembly 44 into different anatomical landmarks. The multi-class classifier can be trained with a predefined list of anatomical landmarks within an internal body cavity (e.g., the abdominal cavity or other internal space). In some embodiments, the predefined list can be based on existing surgical practice and model how surgeons currently navigate the abdominal cavity. For example, in the context of surgery to repair an inguinal hernia, a condition in which organs protrude through the abdominal muscles, key anatomical landmarks include the inguinal triangle, the triangle of doom, and the triangle of pain. The classifier 334 can determine whether the area viewed by the camera assembly 44 belongs to the inguinal triangle, the triangle of doom, or the triangle of pain.

[0077] In some embodiments, the machine learning model can be trained by a training module (e.g., training module 2400 of FIG. 11B ) to generate the machine learning model 320. In some embodiments, the training module 2400 can include a training set having images with labeled anatomical structures and labeled anatomical landmarks and known position and orientation data. In some embodiments, the training module 2400 can access a training set stored on a remote server. The training module 2400 can provide the training set to the machine learning model to be trained. The training module 124 can adjust weights and other parameters in the machine learning model during the training process to reduce the difference between the output of the machine learning model and an expected output. The trained machine learning model 320 can be stored in a database or in the anatomy segmentation and tracking module 300. In some embodiments, the training module 2400 can select a group of the training set as a validation set and apply the trained machine learning model 320 to the validation set to evaluate the trained machine learning model 320.

[0078] The confidence module 346 can create interdependencies between the various outputs 340, including the segmentation maps 342 and the identified anatomical landmarks 344. The confidence module 346 can be bidirectional, whereby the identified anatomical landmarks 344 can affect the relative confidence that can be placed in the segmentation maps 342 generated for different anatomical structures. For example, the presence of a particular anatomical landmark (e.g., one that does not contain nerves or has limited nerves) in a region within the surgical field can automatically eliminate the possibility that the region is a nerve. The anatomy segmentation and tracking module 300 can downweight segmentation maps 342 associated with nerves. Because the confidence module 346 is bidirectional, in some cases the segmentation maps 342 can also affect the relative confidence of the identified anatomical landmarks 344. For example, if the anatomy segmentation and tracking module 300 segments multiple structures (e.g., blood vessels and nerves), the anatomy segmentation and tracking module 300 may have a higher confidence that the robotic assembly 20 is seeing a neurovascular bundle that is an entity that matches a subset of the identified anatomical landmarks. The anatomy segmentation and tracking module 300 may downweight identified anatomical landmarks that are unlikely to contain such a neurovascular bundle (e.g., based on knowledge of the surgical and anatomical domains).

[0079] The tracking module 360 ​​can update the segmentation map 342 and the identified anatomical landmarks 334 to reflect changes in the field of view of the camera assembly 44. For example, video frames are recorded over time at a particular sampling rate (e.g., 30 Hz). During the recording period, the camera assembly 44 is moved by a user and / or objects in the field of view change position. Thus, the field of view of the camera assembly 44 may change over time. The tracking module 360 ​​can update the segmentation map 342 and the identified anatomical landmarks 334 to reflect the changes such that the anatomy segmentation and tracking module 300 can identify anatomical landmarks and anatomical structures over time (e.g., in real time).

[0080] The tracking module 360 ​​may acquire a series of inputs 310 at predetermined time intervals (e.g., a video frame rate, a sampling rate, or a user-defined time interval) over a period of time and apply a machine learning model 320 to each input 310 to generate an output 340. For example, at time slot T1, the tracking module 360 ​​may perform data processing 362A by applying the machine learning model 320 to the inputs 310 acquired at T1 and saving the corresponding outputs 340 and intermediate outputs (e.g., state representation 330, pose representation 324, and / or visual representation 328). At time slot T2, the tracking module 360 ​​may perform data processing 362B by applying the machine learning model 320 to the inputs 310 acquired at T2 and saving the corresponding outputs 340 and intermediate outputs. n Then, the tracking module 360 n The data processing 362N can be performed by applying the machine learning model 320 to the input 310 obtained in and storing the corresponding output 340 and intermediate outputs.

[0081] In some embodiments, the tracking module 360 ​​may n) and a previous state representation from a previous time slot (T2 or T1). The previous state representation may be generated using a previous pose representation extracted from previous position and orientation data acquired in the previous time slot and a previous visual representation extracted from previous images acquired in the previous time slot. In some embodiments, the current time slot is adjacent to the previous time slot. For example, video frames are recorded over time at a particular sampling rate or frame rate (e.g., 30 Hz). The current time slot and the previous time slot are adjacent time slots for acquiring adjacent frames. In some embodiments, there is a predetermined time interval (e.g., based on the frame rate) between the current time slot and the previous time slot, such as a time slot for acquiring discontinuous frames. If the tracking module 360 ​​determines that the similarity is equal to or greater than a similarity threshold, the tracking module 360 ​​may average the state representation and the previous state representation to generate an averaged state representation. The tracking module 360 ​​can feed the averaged state representation to the machine learning model 320 to identify one or more anatomical landmarks within the anatomical space in which the robotic assembly 20 is currently operating and generate multiple segmentation maps.

[0082] For example, the tracking module 360 ​​may determine whether adjacent video frames over a period of time exhibit similar fields of view (i.e., minimal change). The tracking module 360 ​​can quantify the similarity of state representations corresponding to adjacent frames. If the similarity exceeds a threshold, the tracking module 360 ​​determines that the frames have minimal change in field of view and propagates the previous state representation of the previous frame forward in time. The tracking module 360 ​​can average the state representations before providing them to the machine learning model 320. Thus, the tracking module 360's capabilities can utilize historical information to more reliably segment anatomical structures.

[0083] In some embodiments, the tracking module 360 ​​may enable localization of the segmented anatomical structure within the target anatomy. The tracking module 360 ​​may provide the user with the location of the segmented anatomical structure relative to other structures. For example, the tracking module 360 ​​may determine a 3D reconstruction of the anatomical space in which the robotic assembly 20 is operating. For example, the camera assembly 44 may include a light detection and ranging (LIDAR) or a dot matrix projector to obtain a 3D representation or map of at least a portion of a body cavity. The tracking module 360 ​​may use the 3D reconstruction to determine the location of the robotic assembly 20 and the location of the segmented anatomical structure within the anatomy so that the user can easily control the robotic assembly 20 to avoid and / or manipulate the anatomical structure.

[0084] In some embodiments, the trust module 346 and / or the tracking module 360 ​​may be included in the machine learning model 320. In some embodiments, the tracking module 360 ​​may include the machine learning model 320.

[0085] 8 is a flowchart illustrating steps 400 for anatomical structure segmentation and anatomical landmark identification performed by the surgical robotic system 10, according to some embodiments. In step 402, the surgical robotic system 10 receives an image from the camera assembly 44 of the surgical robotic system 10. The image may include a representation of one or more anatomical structures of a subject. An example is described with respect to image 314 of FIG. 7.

[0086] In step 404, the surgical robot system 10 extracts a visual representation from the image. The visual representation may be a compact representation of the image. An example is described with respect to the visual encoder 326 in FIG. 7.

[0087] In step 406, the surgical robotic system 10 determines position and orientation data associated with the robotic assembly 20. The robotic assembly 20 includes a robotic arm assembly 42 and a camera assembly 44. The position and orientation data indicates the pose of the robotic assembly 20. An example is described with respect to the position and orientation data 312 of FIG.

[0088] At step 408, the surgical robotic system 10 generates a pose representation of the robotic assembly 20 based at least in part on the position and orientation data. An example embodiment is described with respect to the pose encoder 322 of FIG.

[0089] At step 410, the surgical robotic system 10 generates a state representation based at least in part on the visual representation and the pose representation. An example is described with respect to the state representation 330 of FIG.

[0090] At step 412, the surgical robotic system 10 identifies one or more anatomical landmarks within the anatomical space in which the robotic assembly 20 is operating based at least in part on the state representation. An example is described with respect to the classifier 334 and identified anatomical landmarks 344 in FIG. 7.

[0091] At step 414, the surgical robotic system 10 generates a plurality of segmentation maps, each identifying which of one or more anatomical structures should avoid contact with the robotic assembly 20. An example is described with respect to the decoder 322 and segmentation map 342 of FIG.

[0092] 9 is a flowchart illustrating steps 500 for anatomical structure tracking performed by surgical robotic system 10, according to some embodiments. In step 502, surgical robotic system 10 receives one image of a plurality of images captured by camera assembly 44. The image includes a representation of one or more anatomical structures of a subject. An example is described with respect to image 314 and tracking module 360 ​​of FIG. 7.

[0093] In step 504, the surgical robot system 10 extracts a visual representation from the image. The visual representation may be a compact representation of the image. An example is described with respect to the visual encoder 326 and tracking module 360 ​​of FIG. 7.

[0094] In step 506, the surgical robotic system 10 determines position and orientation data associated with the robotic assembly 20. An example is described with respect to the position and orientation data 312 and tracking module 360 ​​of FIG.

[0095] At step 508, the surgical robotic system 10 generates a pose representation of the robotic assembly 20 based at least in part on the position and orientation data. An embodiment is described with reference to the pose encoder 322 and tracking module 360 ​​of FIG.

[0096] At step 510, the surgical robotic system 10 generates a state representation based at least in part on the visual representation and the pose representation. An example is described with respect to the state representation 330 and tracking module 360 ​​of FIG. 7.

[0097] In step 512, the surgical robot system 10 determines a similarity between the state representation and a previous state representation. The previous state representation is generated based at least in part on a previous pose representation extracted from previous position and orientation data and a previous visual representation extracted from a previous image that is one of the multiple images and that is temporally prior to the image. An example embodiment is described with respect to tracking module 360 ​​in FIG. 7.

[0098] In step 514, in response to determining that the similarity is greater than or equal to the similarity threshold, the surgical robotic system 10 averages the state representation and the previous state representation to generate an averaged state representation, an example of which is described with respect to tracking module 360 ​​in FIG.

[0099] In step 516, the surgical robotic system 10 identifies one or more anatomical landmarks within the anatomical space in which the robotic assembly 20 is operating based at least in part on the averaged state representation. An example is described with respect to the classifier 334, identified anatomical landmarks 344, and tracking module 360 ​​of FIG. 7.

[0100] In step 518, the surgical robotic system 10 generates a plurality of segmentation maps, each identifying which of one or more anatomical structures should avoid contact with the robotic assembly 20. An example is described with respect to the decoder 322, segmentation map 342, and tracking module 360 ​​of FIG.

[0101] FIG. 10 is a flowchart illustrating steps of a method 600 for training the surgical robotic system 10 to automatically identify anatomical structures, according to some embodiments.

[0102] In step 602, a computing device, e.g., computing module 18, trains a machine learning model based at least in part on a training set having a plurality of labeled images and known position and orientation data associated with robot assembly 20 of surgical robot system 10. For example, training module 2400 of FIG. 11B may include a training set having labeled images with labeled anatomical structures and labeled anatomical landmarks and known position and orientation data. In some embodiments, training module 2400 may access a training set stored on a non-transitory computer-readable medium, e.g., a server. Each labeled image may have one or more labeled anatomical structures and, in some embodiments, one or more labeled anatomical landmarks. The known position and orientation data indicates the pose of robot assembly 20, such as the pose of robot arm assembly 42, the pose of camera assembly 44, or both. The position and orientation data is one of the inputs to the machine learning model, along with an image, as described in FIG. 7. The position and orientation data can provide additional information to images captured by the camera assembly 44, such as the direction, position, orientation, or any other suitable information regarding how the robotic arm assembly 42 and camera assembly 44 are positioned while the images are being captured. The training module 2400 can supply the training set to the machine learning model to be trained to generate a trained machine learning model 320. The training module 2400 can adjust weights and other parameters in the machine learning model during the training process to reduce differences between the output of the machine learning model and expected outputs. For example, the training module 2400 can adjust one or more parameters in the machine learning model to reduce differences between one or more anatomical landmarks identified by the machine learning model and corresponding labeled anatomical landmarks, and between one or more anatomical structures in each segmentation map generated by the machine learning model and corresponding labeled anatomical structures. The trained machine learning model 320 can be stored on a non-transitory storage medium or as a component of the anatomy segmentation and tracking module 300.In some embodiments, the training module 2400 may select a group of the training set as a validation set and apply the trained machine learning model 320 to the validation set to evaluate the trained machine learning model 320.

[0103] In step 604, the surgical robotic system 10 deploys the trained machine learning model 320. For example, the trained machine learning model 320 can identify one or more anatomical landmarks within the anatomical space in which the robotic assembly 20 is operating. An example is described with reference to the classifier 334 and identified anatomical landmarks 344 in FIG. 7. The trained machine learning model 320 can generate multiple segmentation maps, each identifying which of one or more anatomical structures the robotic assembly 20 should avoid contacting. An example is described with reference to the decoder 322 and segmentation map 342 in FIG. 7.

[0104] In some embodiments, the training process described with respect to Figure 12 may be performed on the remote computational servers 1102a-n of Figure 10. The computational servers 1102a-n may send the trained machine learning model 320 to the surgical robotic system 10 to train the machine learning model and automatically identify anatomical structures.

[0105] FIG. 11A is a diagram of an exemplary computing module 18 that may be used to perform one or more steps of a method provided by an exemplary embodiment. The computing module 18 includes one or more non-transitory computer-readable media for storing one or more computer-executable instructions or software for implementing the exemplary embodiments. The non-transitory computer-readable media may include, but are not limited to, one or more types of hardware memory, non-transitory tangible media (e.g., one or more magnetic storage disks, one or more optical disks, one or more USB flash drives), etc. For example, the memory 1006 included in the computing module 18 may store computer-readable and computer-executable instructions or software for implementing the exemplary embodiments (e.g., system code 2000). The computing module 18 also includes a processor 22 and associated cores 1004 for executing the computer-readable and computer-executable instructions or software stored in the memory 1006, as well as other programs for controlling the system hardware. The processor 22 may be a single-core processor or a multi-core (1004) processor.

[0106] The memory 1006 may include computer system memory or random access memory, such as DRAM, SRAM, EDO RAM, etc. The memory 1006 may also include other types of memory, or combinations thereof. A user may interact with the computing module 18 through a display 12, such as a touchscreen display or computer monitor capable of displaying a graphical user interface (GUI) 39. The display 12 may also display other aspects, transducers, and / or information or data associated with the exemplary embodiments. The computing module 18 may include other I / O devices for receiving input from a user, such as a keyboard or any suitable multi-point touch interface 1008, a pointing device 1010 (e.g., a pen, stylus, mouse, or trackpad), etc. The keyboard 1008 and pointing device 1010 may be coupled to the visual display device 12. The computing module 18 may include other appropriate conventional I / O peripherals.

[0107] The computing module 18 may also include one or more storage devices 24, such as a hard drive, CD-ROM, or other computer-readable medium, for storing data and computer-readable instructions, applications, and / or software (which may be executed to generate the GUI 39 on the display 12) that perform the exemplary operations / steps of the surgical robotic system 10, or portions thereof, described herein. The exemplary storage device 24 may also store one or more databases for storing any appropriate information needed to implement the exemplary embodiments. The databases may be updated by a user to add, delete, or update one or more items in the databases, or automatically at any appropriate time. The exemplary storage device 24 may include one or more databases 1026 for storing provisioned data and other data / information used to implement the exemplary embodiments of the systems and methods described herein.

[0108] Computing module 18 may include a network interface 1012 configured to interface with one or more networks, such as a local area network (LAN), a wide area network (WAN), or the Internet, through various connections (including, but not limited to, standard telephone lines, LAN or WAN links (e.g., 802.11, T1, T3, 56kb, X.25), broadband connections (e.g., ISDN, Frame Relay, ATM), wireless connections, controller area networks (CAN), or combinations of any or all of the above) via one or more network devices 1020. Network interface 1012 may include an internal network adapter, a network interface card, a PCMCIA network card, a card bus network adapter, a wireless network adapter, a USB network adapter, a modem, or any other device suitable for interfacing computing module 18 to any type of network with which it can communicate and for performing the operations described herein. Furthermore, computing module 18 may be any computer system, such as a workstation, desktop computer, server, laptop, handheld computer, tablet computer (e.g., an iPad® tablet computer), mobile computing or communication device (e.g., an iPhone® communication device), or other form of computing or communication device capable of communications and having sufficient processor power and memory capacity to perform the operations described herein.

[0109] Computing module 18 may run any operating system 1016, such as any version of the Microsoft® Windows® operating system, different releases of Unix and Linux operating systems, any version of MacOS® for Macintosh computers, any embedded operating system, any real-time operating system, any open source operating system, any proprietary operating system, any operating system for mobile computing devices, or any other operating system capable of running on a computing device and performing the operations described herein. In some embodiments, operating system 1016 may run in native mode or in an emulated mode. In some embodiments, operating system 1016 may run on one or more cloud machine instances.

[0110] The computing module 18 may also include an antenna 1030 that may transmit wireless transmissions to and receive wireless transmissions from a radio frequency (RF) front end.

[0111] 11B illustrates exemplary system code 2000 that may be executable by computing module 18, according to some embodiments. System code 2000 (non-transitory, computer-readable instructions) may be stored on a computer-readable medium, such as storage device 24 and / or memory 1006, and may be executable by hardware processor 22 of computing module 18. System code 2000 may include various custom-written software modules that perform the steps / processes described herein, including, but not limited to, data collection module 2100 for collecting input 310 of FIG. 7 , anatomy segmentation and tracking module 300, encoder 2200 including pose encoder 322 and visual encoder 326, state representation aggregation module 2300 for aggregating pose representation 324 and visual representation 328, structure segmentation decoder 332, classifier 334, confidence module 346, tracking module 360, and training engine 2400. Each component of system 100 is described with respect to FIG. 7 .

[0112] The system code 2000 may be programmed using any suitable programming language, including, but not limited to, C, C++, C#, Java, Python, or any other suitable language. Additionally, the system code 2000 may be distributed across multiple computer systems in communication with each other via a communications network, and / or stored and executed on a cloud computing platform and remotely accessed by computer systems in communication with the cloud platform.

[0113] 12 is a diagram illustrating computer hardware and network components upon which the system 1100 may be implemented. The system 1100 may include a surgical robot system 10, multiple computational servers 1102a-1102n having at least one processor (e.g., one or more graphics processing units (GPUs), microprocessors, central processing units (CPUs), tensor processing units (TPUs), application-specific integrated circuits (ASICs), etc.), and memory for executing the computer instructions and methods described above (which may be embodied as system code 2000). The system 1100 may also include multiple data storage servers 1104a-1104n for storing data. The computational servers 1102a-1102n, the data storage servers 1104a-1104n, and the surgical robot system 10 accessed by a user 1112 may communicate via a communications network 1108. [Explanation of symbols]

[0114] 10. Surgical Robot System 11 Operator Console 12 Display 14 Image Computation Module 16 Tracking Module 16A Tracking Module 17 Hand Controller 17A Left Hand Controller 17B Right Hand Controller 18 Computational Module 19 Foot Pedal Array 20 Robot Subsystem 21 Robot arm subassembly 22 processors 23A Left Hand Controller Subsystem 23B Right Hand Controller Subsystem 24 Storage 26 Controller 30 Image rendering device 34, 34A Position data 36 External Data 37 External Sensor 39 Graphical User Interface 40 Motor 42 Robot arm assembly 42A First Robot Arm 42B Second Robot Arm 44 Camera Assembly 45 End Effector 46 Robot Support System 47 Camera 48 Image Data 50 trocars 100 patients 102 Operating Table 104 Internal cavity 120 Instrument Tip 122 Shaft 124 Housing 126 Shoulder Joint 128 Elbow Joint 130 Wrist Joint 132 sensors 140 Virtual Chest 142A First Pivot Point 142B Second Pivot Point 144 Camera imaging center point 146 Turning center 150 Graphical User Interface 151 frustum 158 Simulated Camera 165 and 166 Simulated robot arm 167 frustum 168 Live Video Footage 171 and 172 Robot Posture View 173 State Identifiers 175 State Identifiers 176 Iconographic Symbols 179 Iconographic Symbols 191 and 192 Simulated robot arms 193 Simulated Camera 198 left pillar box 199 Right pillar box 200 Incarcerated hernia 201 Left Hand Controller 202 Right Hand Controller 210 Blood vessels 220 Anatomical Space 300 Tracking Module 310 Input 312 Orientation Data 314 images 320 Machine Learning Models 322 Attitude Encoder 324 Classifier 326 Visual Encoder 328 Visual Representation 330 State Expressions 332 decoder 334 Classifier 340 output 342 Segmentation Map 344 Anatomical Landmarks 346 Trust Module 360 Tracking Module 362A Data Processing 362B Data Processing 362N Data Processing 1004 cores 1006 memory 1008 Multi-point Touch Interface 1010 pointing device 1012 network interface 1016 Operating Systems 1020 Network Devices 1026 databases 1030 Antenna 1100 System 1102a~1102n remote computing servers 1104a~1104n Data storage servers 1108 Communication Network 1112 users 2000 System Code 2100 Data Acquisition Module 2200 Encoder 2300 State Representation Aggregation Module 2400 Training Module

Claims

1. 1. A surgical robotic system, comprising:

1. A robotic assembly comprising: a camera assembly configured to generate one or more images of an interior cavity of the object; a robotic arm assembly disposed within the internal cavity to perform a surgical procedure; a memory storing one or more instructions; a processor configured or programmed to read the one or more instructions stored in the memory, the processor operably coupled to the robotic assembly; receiving an image from the camera assembly, the image including a representation of one or more anatomical structures of the subject; extracting from said image a visual representation thereof, said visual representation being a compact representation of said image; determining position and orientation data associated with the robot assembly, the position and orientation data indicating a pose of the robot assembly; generating a pose representation of the robotic assembly based at least in part on the position and orientation data; generating a state representation based at least in part on the visual representation and the pose representation, the state representation representing a state of the surgical robotic system; identifying one or more anatomical landmarks within an anatomical space in which the robotic assembly is operating based at least in part on the state representation; a processor that generates a plurality of segmentation maps, each segmentation map identifying which of the one or more anatomical structures should avoid contact with the robot assembly.

2. the processor is further configured or programmed to read the one or more instructions stored in the memory and execute a machine learning model; identifying the one or more anatomical landmarks in the image based on the state representation; The surgical robot system of claim 1 , wherein the plurality of segmentation maps are generated, each of the segmentation maps identifying which of the anatomical structures should avoid contact with the robotic assembly.

3. The surgical robot system of claim 1 , wherein the state representation is generated by aggregating the visual representation and the pose representation.

4. the visual representation and the pose representation are: by averaging the visual and pose representations and weighting each of the visual and pose representations equally; or The surgical robot system of claim 3 , wherein the visual representation and pose representation are aggregated by generating a weighted average of the visual representation and pose representation.

5. 4. The surgical robot system of claim 3, wherein the visual representations and pose representations are aggregated by concatenating the visual representations and pose representations such that the size of the state representation increases with the addition of each visual representation and pose representation.

6. 2. The surgical robot system of claim 1, wherein the one or more anatomical landmarks include at least one of the inguinal triangle, which refers to an area of ​​the abdominal wall; the triangle of doom, which refers to an anatomical triangle bounded medially by the vas deferens, laterally by the testicular artery and vein, and inferiorly by the peritoneal folds; or the triangle of pain, which is the area bounded by the iliopubic cord, the testicular artery and vein, and the peritoneal folds.

7. the processor is further configured or programmed to read the one or more instructions stored in the memory; creating an interdependency between the plurality of segmentation maps and the identified one or more anatomical landmarks; The surgical robot system of claim 1 , further comprising: adjusting weights of the plurality of segmentation maps and the identified one or more anatomical landmarks based at least in part on the interdependence.

8. 1. A surgical robotic system, comprising:

1. A robotic assembly comprising: a camera assembly configured to generate a plurality of images of an interior cavity of the object; a robotic arm assembly disposed within the internal cavity to perform a surgical procedure; a memory storing one or more instructions; a processor configured or programmed to read the one or more instructions stored in the memory, the processor operably coupled to the robotic assembly; receiving an image of the plurality of images, the image including a representation of one or more anatomical structures of the subject; extracting from said image a visual representation thereof, said visual representation being a compact representation of said image; determining position and orientation data associated with the robot assembly, the position and orientation data indicating a pose of the robot assembly; generating a pose representation of the robotic assembly based at least in part on the position and orientation data; generating a state representation based at least in part on the visual representation and the pose representation, the state representation representing a current state of the surgical robotic system; determining a similarity between the state representation and a previous state representation, the previous state representation being generated based at least in part on a previous pose representation extracted from previous position and orientation data and a previous visual representation extracted from a previous image that is one image of the plurality of images and that is temporally prior to the image; responsive to determining that the similarity is greater than or equal to a similarity threshold, averaging the state representation and the prior state representation to generate an averaged state representation; identifying one or more anatomical landmarks based at least in part on the averaged state representation; a processor that generates a plurality of segmentation maps, each segmentation map identifying which of the one or more anatomical structures should avoid contact with the robot assembly.

9. the processor is further configured or programmed to read the one or more instructions stored in the memory; determining a three-dimensional (3D) reconstruction of the anatomical space in which the robotic assembly is operating; The surgical robot system of claim 8 , further comprising: determining a location of each of one or more identified anatomical structures within the anatomical space based at least in part on the 3D reconstruction.

10. The surgical robot system of claim 8 , wherein the plurality of images comprises a plurality of video frames.

11. the processor is further configured or programmed to read the one or more instructions stored in the memory and execute a machine learning model; identifying the one or more anatomical landmarks based at least in part on the averaged state representation; The surgical robot system of claim 8 , wherein the plurality of segmentation maps are generated, each segmentation map identifying which of the one or more anatomical structures should avoid contact with the robotic assembly.

12. The surgical robot system of claim 8 , wherein the state representation is generated by aggregating the visual representation and the pose representation.

13. the visual representation and the pose representation are: by averaging the visual and pose representations and weighting each of the visual and pose representations equally; or The surgical robot system of claim 12 , wherein the visual and pose representations are aggregated by generating a weighted average of the visual and pose representations.

14. 15. The surgical robot system of claim 14, wherein the visual representations and pose representations are aggregated by concatenating the visual representations and pose representations such that the size of the state representation increases with the addition of each visual representation and pose representation.

15. 9. The surgical robot system of claim 8, wherein the one or more anatomical landmarks include at least one of the inguinal triangle, which refers to an area of ​​the abdominal wall; the triangle of doom, which refers to an anatomical triangle bounded medially by the vas deferens, laterally by the testicular artery and vein, and inferiorly by the peritoneal folds; or the triangle of pain, which is the area bounded by the iliopubic cord, the testicular artery and vein, and the peritoneal folds.

16. the processor is further configured or programmed to read the one or more instructions stored in the memory; creating an interdependency between the plurality of segmentation maps and the identified one or more anatomical landmarks; The surgical robot system of claim 8 , further comprising: adjusting weights of the plurality of segmentation maps and the identified one or more anatomical landmarks based at least in part on the interdependence.

17. The similarity between the state representation and a previous state representation is: The surgical robot system of claim 8 , wherein the imaging is determined by determining whether adjacent images of the plurality of images over a period of time exhibit similar fields of view.

18. 9. The surgical robot system of claim 8, wherein the processor is further configured or programmed to read the one or more instructions stored in the memory and update the plurality of segmentation maps and the identified one or more anatomical landmarks in real time to reflect changes in a field of view of the camera assembly.

19. 1. A computer-implemented method for training a surgical robotic system to automatically identify anatomical structures, comprising: training a machine learning model based at least in part on a training set having a plurality of labeled images and known position and orientation data associated with a robotic assembly of the surgical robotic system, wherein each labeled image has one or more labeled anatomical structures and one or more labeled anatomical landmarks, and the known position and orientation data indicates a pose of the robotic assembly; Deploying the trained machine learning model; Identifying one or more anatomical landmarks within an anatomical space in which the robotic assembly is operating; generating a plurality of segmentation maps, each segmentation map identifying which of the one or more anatomical structures should avoid contact with the robotic assembly.

20. 2. The computer-implemented method of claim 1, wherein training the machine learning model comprises adjusting one or more parameters in the machine learning model to reduce differences between one or more anatomical landmarks identified by the machine learning model and corresponding labeled anatomical landmarks, and between the one or more anatomical structures in each segmentation map generated by the machine learning model and the corresponding labeled anatomical structures.

Images