Co-manipulation surgical system with backend processing for data handling and analytics

Abstract

Co-manipulation robotic systems are described herein that may be used for assisting with surgical procedures, including laparoscopic surgery. The co-manipulation robotic systems allow a surgeon to use commercially available surgical tools while providing benefits associated with surgical robotics. Advantageously, the ecosystem incorporating the co-manipulation robotic systems utilizes data received by the system's sensors during surgical procedures in combination with stored and learned surgeon preferences for enhancing the surgical experience for the surgeon, the patient, and the healthcare system involved in the surgeries.

Description

225887-081001CO-MANIPULAHON SURGICAL SYSTEM WITH BACKEND PROCESSING FOR DATA HANDLING AND ANALYTICSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Appl. No. 63 / 738,477, filed December 23, 2024, U.S. Provisional Patent Appl. No. 63 / 683,487, filed August 15, 2024, and EP Patent Appl. No. 24306359.1, filed August 14, 2024, the entire contents of each of which are incorporated herein by reference.FIELD OF USE

[0002] This technology relates to co-manipulation robotic systems, such as those designed to be coupled to clinician-selected surgical instruments to permit movement of the robot arm(s) via movement at the handle of the surgical instrument(s), along with enhanced features for setup and automatic intraoperative movements.BACKGROUND

[0003] Managing vision and access during a surgical procedure, e.g., a laparoscopic procedure, is a challenge. The surgical assistant paradigm is inherently imperfect, as the assistant is being asked to anticipate and see with the surgeon’s eyes, without standing where the surgeon stands, and similarly to anticipate and adjust how the surgeon wants the tissue of interest exposed, throughout the procedure. For example, during a laparoscopic procedure, one assistant may be required to hold a retractor device to expose tissue for the surgeon, while another assistant may be required to hold a scope device to provide a field of view of the surgical space within the patient to the surgeon during the procedure, either one of which may be required to hold the respective tools in an impractical position, e.g., from between the arms of the surgeon while the surgeon is actively operating additional surgical instruments.

[0004] Various attempts have been made at solving this issue. For example, a rail-mounted orthopedic retractor, which is a purely mechanical device that is mounted to the patient bed / table,713732261 v3 1225887-081001 may be used to hold a scope device in position during a laparoscopic procedure, and another railmounted orthopedic retractor may be used to hold a retractor device in position during the laparoscopic procedure. However, the rail-mounted orthopedic retractor requires extensive manual interaction to unlock, reposition, and lock the tool in position.

[0005] Complex robot-assisted systems such as the Da Vinci Surgical System (made available by Intuitive Surgical, Sunnyvale, California) have been used by surgeons to enhance laparoscopic surgical procedures by permitting the surgeon to tele-operatively perform the procedure from a surgeon console remote from the patient console holding the surgical instruments. Such complex robot-assisted systems are very expensive and have a very large footprint and take up a lot of space in the operating room. Moreover, such robot-assisted systems typically require unique system-specific surgical instruments that are compatible with the system, and thus surgeons may not use standard off-the-shelf surgical instruments that they are used to. As such, the surgeon is required to learn an entirely different way of performing the laparoscopic procedure.

[0006] Moreover, it may be challenging for surgeons to learn to apply the right amount of forces during a surgical procedure, e.g., a laparoscopic surgery where one or more trocars are inserted through the body wall of a patient. For example, whereas in traditional open surgery where surgeons can directly feel and manipulate tissues with their hands, thereby providing valuable tactile feedback, in laparoscopic surgery, however, the surgeon uses long, slender instruments with limited tactile sensation. This lack of direct touch can make it difficult to gauge the amount of force applied, increasing the risk of tissue damage due to forces applied thereto by the instrument. In addition, laparoscopic surgery relies on 2D video images from a camera inside the patient’s body, which can lead to a loss of depth perception, thereby making it challenging to accurately discern the distance between instruments and tissues. Misjudging depth can result in excessive or insufficient force application, potentially causing harm to nearby anatomical structures. In addition, laparoscopic instruments have limited degrees of freedom compared to the human hand, which can make precise and delicate movements more challenging to execute, and surgeons must adapt to these limitations when applying forces. Further, different tissues in the body have varying properties, such as thickness, elasticity, and fragility, and713732261 v3 2225887-081001 understanding how to adapt force application based on the tissue being manipulated is crucial to avoid injury to the patient.

[0007] Incorrect force application during laparoscopic surgery can lead to complications such as bleeding, perforations, or damage to adjacent organs. Trocar placement complications include vascular injury, bowel / visceral injury. The bowel and vascular injuries are often due to placement of the primary trocar or Veress needle because they are done blindly. However, injuries also can occur with secondary trocar insertion if the trocars are not properly visualized throughout their insertion. Surgeons must be cautious and precise to minimize these risks. Despite the recent advances in minimally invasive techniques, new technologies, and evidencebased guidelines, no single technique or instrument has been proven to completely eliminate laparoscopic entry associated injury. Limiting the force applied during access may be beneficial to prevent injuries. To overcome these challenges, surgeons often undergo extensive training using simulators and virtual reality tools. These platforms may help surgeons practice force application and refine their skills in a controlled environment before operating on real patients. However, learning to differentiate between tissues and tailor force accordingly to apply the right forces during trocar placement can be challenging, and is a skill that takes time to develop.

[0008] Another significant challenge in abdominal surgeries is the unintentional retention of surgical items (RSIs) within the patient’s abdomen or pelvis, which may lead to severe postoperative complications including infection and pain, which may require additional surgeries to remove these items. Clinically, a retained surgical sponge may be asymptomatic or cause a granulomatous response with abscess development, intestinal obstruction, or fistula formation. Radiologically, gossypibomas may be mistaken for postoperative collections or tumors, especially with the increasing use of absorbable hemostatic materials to control hemorrhage. While the most common RSIs are sponges or gauze (gossypiboma or textiloma), RSIs also may include surgical instruments and needles. According to recent studies, unintentional RSIs cause 70% of re- interventions, with a morbidity rate of 80% and a mortality rate of 35%. Despite rigorous protocols and counting procedures, the fast-paced and complex nature of surgical environments may still result in these critical errors, e.g., a “correct” count may not rule out the possibility of an RSI. For example, in one study, a sponge count was reported as correct in 22 of 29 patients with retained sponges in the abdomen (76%). These oversights not only harm patient713732261 v3 3225887-081001 health but may also lead to legal liabilities and increased healthcare costs. For example, studies have shown that retained surgical sponges or swabs in the abdomen or pelvis accounts for 50% of malpractice claims for RSIs. An analysis of government records, clinical studies, and two major malpractice claim databases suggests that a single case of a retained sponge can cost a hospital and the surgeon over half a million dollars in indemnity payouts and legal fees.

[0009] In view of the foregoing drawbacks of previously known systems and methods, there exists a need for a system that provides the surgeon with the ability to seamlessly position and manipulate various surgical instruments as needed, thus avoiding the workflow limitations inherent to both human and mechanical solutions.

[0010] In addition, there exists a need for a reliable way to extract usage information from the data generated from the co-manipulation surgical system, upload the data to the cloud, and connect that data source to an invoicing system.SUMMARY

[0011] The present technology overcomes the drawbacks of previously known systems and methods by providing a co-manipulation surgical system to assist with a surgical procedure. The co-manipulation surgical system may comprise a robot arm comprising a plurality of links and joints and a distal end configured to be removably coupled to a surgical instrument for performing the surgical procedure, an optical sensor configured to collect image data, and a controller operatively coupled to the optical sensor. The controller may have instructions that, when executed by one or more processors of the controller, cause the controller to: receive information indicative of a type of the surgical procedure to be performed; receive image data collected by the optical sensor, the image data indicative of a surgical site on a patient’s body; identify a location of one or more trocar ports disposed within the surgical site; identify an optimal robot arm configuration based on the location of the one or more trocar ports and the type of the surgical procedure, the optimal robot arm configuration comprising an optimal position of the distal end of the robot arm relative to the one or more trocar ports; and adjust the robot arm via the plurality of links and joints to the optimal robot arm configuration.713732261 v3 4225887-081001

[0012] The controller may be configured to execute a machine learning algorithm to identify the optimal robot arm configuration, the machine learning algorithm trained with a dataset of previously identified optimal robot arm configurations relative to one or more previously identified trocar ports for a same type of surgical procedure as the surgical procedure. Additionally, the controller may be configured to generate a surgical site heat map comprising one or more previously identified trocar ports for a same type of surgical procedure as the surgical procedure, and cause a display to display the surgical site heat map. For example, the controller may be configured to calculate an optimal trocar port placement location based on an average center of the one or more previously identified trocar ports for the same type of surgical procedure as the surgical procedure, and generate a recommendation comprising the optimal trocar port placement location.

[0013] The controller may further be configured to determine an optimal trocar port of the one or more trocar ports for the surgical procedure based on a type of the surgical instrument. Moreover, the controller may be configured to identify a patient bed as a reference point, the patient’s body disposed on the patient bed, and execute one or more algorithms to delineate the surgical site. For example, the controller may be configured to determine an angle of the patient bed, and automatically adjust the robot arm via the plurality of links and joints to the optimal robot arm configuration based on the angle of the patient bed. In addition, the controller may be configured to determine a height of the patient bed, and automatically adjust the robot arm via the plurality of links and joints to the optimal robot arm configuration based on the height of the patient bed. The controller further may be configured to identify a boundary of the surgical site, and identify the location of the one or more trocar ports disposed within the boundary of the surgical site. The optimal robot arm configuration may be configured to maximize a workspace of the robot arm relative to surgical site. Further, the controller may be configured to automatically adjust the robot arm via the plurality of links and joints to the optimal robot arm configuration responsive to user input. In addition, the optimal robot arm configuration may comprise an optimal position of the plurality of links and joints of the robot arm relative to the one or more trocar ports.

[0014] In accordance with another aspect, the co-manipulation surgical system may comprise a surgical platform comprising a plurality of wheels configured to permit mobility of the surgical713732261 v3 5225887-081001 platform, an optical sensor mounted on the surgical platform and configured to collect image data, and a controller operatively coupled to optical sensor. The controller may have instructions that, when executed by one or more processors of the controller, cause the controller to: receive image data collected by the optical sensor, the image data indicative of an operating room environment comprising a patient bed and one or more objects within an operating room; generate a 3D reconstruction of the operating room environment based on the image data, the 3D reconstruction comprising graphical representations of the surgical platform and the one or more objects relative to the patient bed within the operating room; cause a graphical user interface to display the 3D reconstruction of the operating room environment; receive user input data indicative of a surgeon preference of an organization of at least one of the surgical platform or the one or more objects relative to the patient bed within the operating room for a surgical procedure; generate a preferred 3D reconstruction of the operating room environment based on the user input data; and cause a display to display the preferred 3D reconstruction of the operating room environment to facilitate setup of the surgical platform and the one or more objects relative to the patient bed within the operating room for the surgical procedure.

[0015] For example, the one or more objects may comprise at least one of a camera control unit tower, a surgical table, or surgical tools. Moreover, the user input data may comprise a surgeon preference of an organization of one or more persons relative to the patient bed within the operating room, and the preferred 3D reconstruction of the operating room environment generated by the controller may comprise graphical representations of the one or more persons relative to the patient bed within the operating room. The controller further may be configured to save the preferred 3D reconstruction of the operating room environment for the surgical procedure in a surgeon profile associated with the surgeon. Additionally, the controller may be configured to analyze the user input data and generate a warning if the surgeon preference of the organization of the at least one of the surgical platform or the one or more objects relative to the patient bed within the operating room is determined to be suboptimal for the surgical procedure based on at least one of spatial constraints, positioning of equipment, or accessibility issues.

[0016] The controller further may be configured to generate a recommendation for an optimal organization of the at least one of the surgical platform or the one or more objects relative to the patient bed within the operating room based on the user input data and at least one713732261 v3 6225887-081001 of spatial constraints, positioning of equipment, or accessibility issues. Accordingly, the controller may be configured to receive additional user input data indicative of an informed surgeon preference of the organization of the at least one of the surgical platform or the one or more objects relative to the patient bed within the operating room based on the recommendation, and generate an optimal 3D reconstruction of the operating room environment based on the additional user input data. Moreover, the controller may be configured to generate an optimal trajectory for movement of the surgical platform within the operating room by a user during an environmental scan to permit the optical sensor to collect comprehensive image data indicative of the operating room environment. In addition, the co-manipulation surgical system further may comprise a mobile application operatively coupled to the controller, such that the controller may be configured to cause, via the mobile application, a mobile device comprising the graphical user interface to display the 3D reconstruction of the operating room environment. Additionally, or alternatively, the controller is configured to cause, via the mobile application, a mobile device comprising the display to display the preferred 3D reconstruction of the operating room environment.

[0017] In accordance with another aspect, a co-manipulation surgical system to assist with a surgical procedure comprising a setup stage, an intraoperative stage, and a teardown stage is provided. The co-manipulation surgical system may comprise one or more robot arms each comprising a plurality of links and joints and a distal end configured to be removably coupled to a surgical instrument for performing the surgical procedure, and a controller operatively coupled to the one or more robot arms. The controller may have instructions that, when executed by one or more processors of the controller, cause the controller to: detect when the co-manipulation surgical system is in at least one of the setup stage, the intraoperative stage, or the teardown stage; determine, upon detection that the co-manipulation surgical system has been in at least one of the setup stage, the intraoperative stage, or the teardown stage, that a surgical procedure has been performed by the co-manipulation surgical system; aggregate a total count of surgical procedures determined to have been performed by the co-manipulation surgical system within a predetermined period; and generate information indicative of the total count of surgical procedures performed by the co-manipulation surgical system within the predetermined period for transmission to a payment system configured to generate an invoice based on the total count.713732261 v3 7225887-081001

[0018] The controller may be configured to transmit the information indicative of the total count to the payment system at a predetermined frequency. In some embodiments, the comanipulation surgical system may comprise two robot arms. Moreover, the controller may be configured to detect that the co-manipulation surgical system is in the intraoperative stage when the distal end of at least one of the one or more robot arms transitions from being decoupled from the surgical instrument to being coupled to the surgical instrument. In addition, the controller may be configured to detect that the co-manipulation surgical system is in the teardown stage when the distal end of each of the one or more robot arms coupled to a surgical instrument transitions from being coupled to the surgical instrument to being decoupled from the surgical instrument. Further, the distal end of each of the one or more robot arms may comprise a coupler interface configured to be removably coupled to a coupler body. The coupler body may be configured to be removably coupled to the surgical instrument to thereby couple the surgical instrument to the distal end of the respective robot arm.

[0019] The controller may be configured to detect that the co-manipulation surgical system is in the setup stage when the coupler interface of at least one of the one or more robot arms transitions from being decoupled from the coupler body to being coupled to the coupler body. Additionally, or alternatively, the controller may be configured to detect that the co-manipulation surgical system is in the teardown stage when the coupler interface of each of the one or more robot arms coupled to a coupler body transitions from being coupled to the coupler body to being decoupled from the coupler body. The co-manipulation surgical system further may comprise a graphical user interface operatively coupled to the controller, the graphical user interface comprising a plurality of selectable preset surgical procedure configurations of the one or more robot arms. Accordingly, the controller may be configured to detect that the co-manipulation surgical system is in the setup stage or the teardown stage upon selection of at least one of the plurality of selectable preset surgical procedure configurations. For example, the plurality of selectable preset surgical procedure configurations may comprise a drape mode, such that the controller may be configured to, upon selection of the drape mode, cause the one or more robot arms to transition to a predetermined drape pose via the plurality of links and joints of the one or more robot arms to facilitate draping of the one or more robot arms. Moreover, the controller713732261 v3 8225887-081001 may be configured to detect that the co-manipulation surgical system is in the setup stage upon selection of the drape mode.

[0020] The co-manipulation surgical system further may comprise a platform coupled to a base of each of the one or more robot arms. In addition, the plurality of selectable preset surgical procedure configurations may comprise a stow mode, such that the controller may be configured to, upon selection of the stow mode, cause the one or more robot arms to transition to a retracted stow pose above the platform via the plurality of links and joints of the one or more robot arms and rotate about the respective base of the one or more robot arms such that the distal end of each of the one or more robot arms extends towards the rear of the platform and the one or more robot arms are within a footprint of the platform in the retracted stow pose above the platform. Accordingly, the controller may be configured to detect that the co-manipulation surgical system is in the teardown stage upon selection of the stow mode.

[0021] Additionally, the plurality of selectable preset surgical procedure configurations may comprise a compact mode, such that the controller may be configured to, upon selection of the compact mode, cause the one or more robot arms to transition to a semi-retracted compact pose via the plurality of links and joints of the one or more robot arms and rotate about the respective base of the one or more robot arms such that each of the one or more robot arms extends away from the platform to facilitate transportation of the co-manipulation surgical system, the semiretracted compact pose being less retracted than the retracted stow pose. Accordingly, the controller may be configured to detect that the co-manipulation surgical system is in the teardown stage upon selection of the compact mode. The co-manipulation surgical system further may comprise a graphical user interface operatively coupled to the controller, the graphical user interface configured to permit selection of at least one of a surgeon profile or a procedure type. Accordingly, the controller may be configured to detect that the comanipulation surgical system is in the setup stage upon selection of the at least one of the surgeon profile or the procedure type.

[0022] The co-manipulation surgical system further may comprise an optical sensor configured to collect image data, such that the controller may be configured to detect when the co-manipulation surgical system is in the at least one of the setup stage, the intraoperative stage,713732261 v3 9225887-081001 or the teardown stage based on the image data and associated timestamps of the image data. Moreover, the co-manipulation surgical system may comprise an audio sensor configured to collect audio data, such that the controller may be configured to execute a speech recognition technique to identify one or more procedural phases of the surgical procedure, and detect when the co-manipulation surgical system is in the intraoperative stage based on the one or more identified procedural phases. The controller further may be configured to determine that the surgical procedure has been performed by the co-manipulation surgical system upon detection of a transition from at least one of the setup stage to the intraoperative stage or the intraoperative stage to the teardown stage. At least one of the one or more processors may comprise Cloudbased software. In addition, the controller may be configured to transmit, via a Field Service Action, the information indicative of the total count for upload to a Cloud server.

[0023] In accordance with another aspect, another co-manipulation surgical system to assist with a surgical procedure is provided. The co-manipulation surgical system may comprise a robot arm comprising a plurality of links and joints and a distal end configured to be removably coupled to a scope configured to collect image data, an optical sensor configured to collect image data, and a controller operatively coupled to the scope and the optical sensor. The controller may have instructions that, when executed by one or more processors of the controller, cause the controller to: receive image data collected by the optical sensor indicative of an external surgical site of the surgical procedure; receive image data collected by the scope indicative of an internal surgical site of the surgical procedure within a patient; execute an object segmentation algorithm to detect one or more predefined surgical items within the external and internal surgical sites based on the image data collected by the optical sensor and the scope, respectively; track, for each detected predefined surgical item, a number of times the detected predefined surgical item enters the internal surgical site from the external surgical site and a number of times the detected predefined surgical item is removed from the internal surgical site to the external surgical site; and generate, when the surgical procedure is completed, an alert if the number of times the detected predefined surgical item is removed from the internal surgical site to the external surgical site is less than the number of times the detected predefined surgical item enters the internal surgical site from the external surgical site. For example, the one or more predefined surgical items may comprise at least one of a sponge, a gauze, or a needle.713732261 v3 10225887-081001

[0024] The co-manipulation surgical system further may comprise a platform coupled to a base of the robot arm, such that the optical sensor may be mounted on the platform. Moreover, the controller may be configured to determine one or more procedural phases of the surgical procedure based on the image data collected by at least one of the optical sensor or the scope, and determine when the surgical procedure is completed based on the one or more procedural phases of the surgical procedure. In addition, the controller may be configured to identify, if the number of times the detected predefined surgical item is removed from the internal surgical site to the external surgical site is less than the number of times the detected predefined surgical item enters the internal surgical site from the external surgical site, a timestamp of the image data associated with when the detected predefined surgical item was last detected within the internal surgical site. Accordingly, the controller may be configured to cause a display to display the image data associated with the identified timestamp.

[0025] In accordance with another aspect, another co-manipulation surgical system to assist with a surgical procedure is provided. The co-manipulation surgical system may comprise a robot arm comprising a plurality of links and joints and a distal end configured to be removably coupled to a surgical instrument for performing the surgical procedure, an optical sensor configured to collect image data, and a controller operatively coupled to the optical sensor. The controller may have instructions that, when executed by one or more processors of the controller, cause the controller to: receive the image data collected by the optical sensor indicative of the surgical procedure; execute an object segmentation algorithm to detect one or more predefined surgical instruments and / or one or more predefined anatomical structures associated with the surgical procedure based on the image data; and determine a surgical phase of the surgical procedure based on the one or more predefined surgical instruments and / or one or more predefined anatomical structures.

[0026] The controller may be configured to execute a machine learning algorithm to determine the surgical phase of the surgical procedure based on the one or more predefined surgical instruments and / or one or more predefined anatomical structures, the machine learning algorithm trained via a dataset of known surgical phases for a plurality of surgical procedures comprising the one or more predefined surgical instruments and / or one or more predefined anatomical structures. The co-manipulation surgical system further may comprise one or more713732261 v3 11225887-081001 microphones configured to receive audio data. Accordingly, the controller may be configured to receive audio data from the one or more microphones, the audio data indicative of the surgical phase of the surgical procedure, and determine a surgical phase of the surgical procedure based at least partially on the audio data. For example, the audio data may be indicative of a verbal description by a user of the surgical phase of the surgical procedure, such that the controller may be configured to record the determined surgical phase and known kinematics of the robot arm during the determined surgical phase to a surgeon profile associated with the user.

[0027] Moreover, the controller may be configured to prompt a user to provide the verbal description, the verbal description comprising information of when the user transitions from one surgical phase of the surgical procedure to another surgical phase of the surgical procedure. In addition, the surgeon profile may be configured to be uploaded to another co-manipulation surgical system, such that the user may access data recorded to the surgeon profile via the another co-manipulation surgical system. Moreover, the controller may be configured to execute a machine learning algorithm to determine the surgical phase of the surgical procedure based on the one or more predefined surgical instruments, the one or more predefined anatomical structures, and / or known kinematics of the robot arm during the surgical procedure, the machine learning algorithm trained at least partially via data recorded to the surgeon profile. Additionally, the controller may be configured to determine the surgical phase of the surgical procedure based at least partially on known kinematics of the robot arm during the surgical procedure. Further, the controller may be configured to automatically adjust a configuration of the robot arm based on the surgical phase of the surgical procedure.

[0028] In addition, the controller may be configured to cause, when the robot arm is removably coupled to a scope in an instrument centering mode, the robot arm via the plurality of links and joints to automatically track a surgical instrument within a field of view of the scope based on the surgical phase of the surgical procedure. Additionally, or alternatively, the controller may be configured to cause, when the robot arm is removably coupled to a scope in an instrument centering mode, the robot arm via the plurality of links and joints to automatically track an anatomical structure within a field of view of the scope based on the surgical phase of the surgical procedure. The controller further may be configured to identify a predefined surgical task based on the one or more predefined surgical instruments and / or one or more713732261 v3 12225887-081001 predefined anatomical structures. For example, the predefined surgical task may comprise grasping, retracting, or cutting. In addition, the controller may be configured to estimate a surgical procedure end time based on the surgical phase of the surgical procedure, and communicate the estimated surgical procedure end time to operating room staff to facilitate preparation of a subsequent surgical procedure.

[0029] In accordance with another aspect, another co-manipulation surgical system to assist with a surgical procedure is provided. The co-manipulation surgical system may comprise a robot arm comprising a plurality of links and joints and a distal end configured to be removably coupled to a surgical instrument for performing the surgical procedure, an optical sensor configured to collect image data, and a controller operatively coupled to the optical sensor. The controller may have instructions that, when executed by one or more processors of the controller, cause the controller to: receive the image data collected by the optical sensor, the image data indicative of a surgical scene during the surgical procedure; generate an interactive virtual 3D reconstruction of the surgical scene during the surgical procedure based on the image data, the interactive virtual 3D reconstruction comprising graphical representations of the robot arm and one or more objects or persons associated with performing the surgical procedure; and permit a user to remotely interact with the interactive virtual 3D reconstruction of the surgical scene via a virtual display device operatively coupled to the controller.

[0030] In accordance with another aspect, another co-manipulation surgical system to assist with a surgical procedure is provided. The co-manipulation surgical system may comprise a robot arm comprising a plurality of links and joints and a distal end configured to be removably coupled to a surgical instrument for performing the surgical procedure on a patient, and a controller operatively coupled to the robot arm. The controller may have instructions that, when executed by one or more processors of the controller, cause the controller to: calculate a force applied by the surgical instrument to one or more tissues of the patient during the surgical procedure; generate a graphical representation indicative of a magnitude of the force applied by the surgical instrument to one or more tissues of the patient during the surgical procedure; and cause a display to display the graphical representation in real-time during the surgical procedure.713732261 v3 13225887-081001

[0031] In addition, the controller may be configured to receive a video feed from a scope operatively coupled to the controller, the video feed indicative of a surgical site of the surgical procedure. The controller further may be configured to cause the display to display the graphical representation overlaid on the video feed in real-time during the surgical procedure. Moreover, the co-manipulation surgical system further may comprise an optical sensor configured to collect image data, such that the controller may be configured to receive timestamped image data from the optical sensor indicative of the surgical procedure, synchronize the timestamped image data with the video feed from the scope, and cause a graphical user interface to display the timestamped image data alongside the video feed from the scope. The controller further may be configured to receive timestamped telemetry data associated with the robot arm, synchronize the timestamped telemetry data with the video feed from the scope, and cause a graphical user interface to display the timestamped telemetry data alongside the video feed from the scope.

[0032] Additionally, the controller may be configured to receive timestamped data indicative of kinematics of the robot arm during the surgical procedure, generate a timestamped 3D reconstruction of the robot arm during the surgical procedure, synchronize the timestamped 3D reconstruction with the video feed from the scope, and cause a graphical user interface to display the timestamped 3D reconstruction of the robot arm alongside the video feed from the scope. Accordingly, the controller may be configured to generate a graphical representation of a force profile indicative of the force applied by the surgical instrument to one or more tissues of the patient during the surgical procedure, synchronize the timestamped 3D reconstruction and the video feed from the scope with the force profile, and cause the graphical user interface to display the graphical representation of the force profile alongside the timestamped 3D reconstruction of the robot arm alongside the video feed from the scope. The controller may be configured to generate an alert if the force applied by the surgical instrument to one or more tissues of the patient during the surgical procedure exceeds a predetermined threshold.

[0033] In accordance with another aspect, another co-manipulation surgical system to assist with a surgical procedure is provided. The co-manipulation surgical system may comprise a robot arm comprising a plurality of links and joints and a distal end configured to be removably coupled to a surgical instrument for performing the surgical procedure, one or more microphones configured to receive audio data, and a controller operatively coupled to the microphones, the713732261 v3 14225887-081001 controller having instructions that, when executed by one or more processors of the controller, cause the controller to: receive audio data from the one or more microphones, the audio data indicative of a user command; and pass the audio data into an algorithm configured to identify the user command and generate data indicative of a response to the user command.

[0034] For example, the response to the user command may comprise an audio response, such that the controller may be configured to cause one or more speakers to emit an audio response corresponding to the response to the user command. Additionally, or alternatively, the response to the user command may comprise an action to be performed by the robot arm, such that the controller may be configured to cause the robot arm to move via the plurality of links and joints in accordance with the action. Accordingly, the controller may be configured to access a surgeon profile associated with a surgeon, the surgeon profile comprising saved surgeon preferences from previous surgical procedures, and cause the robot arm to move via the plurality of links and joints in accordance with the action and the saved surgeon preferences. The comanipulation surgical system further may comprise an optical sensor configured to collect image data, such that the controller may be configured to generate data indicative of the response to the user command based at least partially on image data received by the optical sensor.

[0035] In some embodiments, the algorithm may comprise a large language model. Moreover, the controller may be configured to receive audio data from the one or more microphones indicative of a surgical phase of the surgical procedure, and determine the surgical phase of the surgical procedure based on the audio data. In addition, the controller may be configured to receive audio data from the one or more microphones indicative of verbal notes by a user, generate textual case notes based on the audio data, and store the textual case notes in a surgeon profile associates with the user. The co-manipulation surgical system further may comprise an optical sensor configured to collect image data. Accordingly, the controller may be configured to receive timestamped image data from the optical sensor indicative of a surgical procedure, use computer vision to detect one or more surgical instruments associated with the surgical procedure based on the timestamped image data, and generate textual case notes indicative of a surgical task associated with the one or more surgical instruments during the surgical procedure.713732261 v3 15225887-081001

[0036] In accordance with another aspect, a co-manipulation surgical system to assist with a surgical procedure is provided. The co-manipulation surgical system may comprise a robot arm comprising a plurality of links, a plurality of joints, a proximal region operatively coupled to a base, and a distal region configured to be removably coupled to the surgical instrument, a plurality of motors operatively coupled to corresponding joints of the plurality of joints, and a controller operatively coupled to the robot arm. The controller may have instructions that, when executed by one or more processors of the controller, cause the controller to: receive fingerprint data indicative of an operational fingerprint of the system; establish a baseline value associated with the operational fingerprint, the baseline value corresponding to a healthy operational state of the system; and aggregate the fingerprint data over time, wherein an inspection or maintenance service is predicted to be required if the fingerprint data deviates from the baseline value by more than a predetermined threshold value. For example, the controller may be configured to establish the baseline value based on manufacture and / or installation baseline data associated with the operational fingerprint of the system. The predetermined threshold value may be selected such that fingerprint data that deviates from the baseline value by more than the predetermined threshold value indicates that the system is closer to an unhealthy operational state than the healthy operational state. The fingerprint data may comprise telemetry data, 3D depth data, color image data, audio data, laparoscopic video data, and / or application event logs.

[0037] Moreover, the controller may be configured to automatically transition, upon selection of a second preset configuration, the robot arm from a first preset configuration to the second preset configuration via the plurality of motors, the second preset configuration different from the first preset configuration. For example, the fingerprint data may comprise a motor current reading across a motor of the plurality of motors as the robot arm transitions from the first preset configuration to the second preset configuration, such that the baseline value may comprise a baseline motor current reading across the motor as the robot arm transitions from the first preset configuration to the second preset configuration under normal operating conditions. Accordingly, the controller may be configured to calculate an average, maximum, and / or integral value of the motor current reading over time, such that the inspection or maintenance service may be predicted to be required if the average, maximum, and / or integral value of the motor current reading deviates from the baseline value by more than the predetermined threshold value.713732261 v3 16225887-081001Motor current readings that deviate from the baseline value by more than the predetermined threshold value may indicate that the system requires an increasing amount of effort to transition the robot arm from the first preset configuration to the second preset configuration. The first and second preset configurations may be selectable from a plurality of preset configurations. For example, the plurality of preset configurations may comprise a compact configuration, a stow configuration, and / or a drape configuration. Moreover, the fingerprint data may comprise a motor voltage reading across a motor of the plurality of motors as the robot arm transitions from the first preset configuration to the second preset configuration, such that the baseline value may comprise a baseline motor voltage reading across the motor as the robot arm transitions from the first preset configuration to the second preset configuration under normal operating conditions.

[0038] In addition, the fingerprint data may comprise a tool sensor value indicative of whether the surgical instrument is removably coupled to the distal region of the robot arm, such that the baseline value may comprise a baseline tool sensor value indicative of whether the surgical instrument is removably coupled to the distal region of the robot arm under normal operating conditions. For example, the surgical instrument may be configured to be removably coupled to the distal region of the robot arm via a coupler body configured to be removably coupled to the surgical instrument and removably coupled to a coupler interface at the distal region of the robot arm. Accordingly, the tool sensor value may be indicative of whether the coupler body is removably coupled to the coupler interface when the surgical instrument is removably coupled to the coupler body and whether the coupler body is removably coupled to the coupler interface when the surgical instrument is not removably coupled to the coupler body. Further, the baseline tool sensor value may be indicative of whether the coupler body is removably coupled to the coupler interface when the surgical instrument is removably coupled to the coupler body under normal operating conditions and whether the coupler body is removably coupled to the coupler interface when the surgical instrument is not removably coupled to the coupler body under normal operating conditions. In addition, the controller may be configured to calculate an average, minimum, maximum, and / or standard deviation value of the tool sensor value, such that the inspection or maintenance service may be predicted to be required if the average, minimum, maximum, and / or standard deviation value of the tool sensor value deviates from the baseline value by more than the predetermined threshold value. In some embodiments,713732261 v3 17225887-081001 the tool sensor value may comprise a first tool sensor value generated by a first tool sensor and a second tool sensor value generated by a second tool sensor. Additionally, the controller may be configured to receive data indicative of angulation between the surgical instrument and a distal- most link of the plurality of links of the robot arm, the angulation configured to facilitate contextualization of the tool sensor value.

[0039] Moreover, the controller may be configured to enable a function of the system upon manual actuation of an actuator by a user. For example, the fingerprint data may comprise data indicative of performance of the actuator, such that the baseline value may comprise a baseline value indicative of performance of the actuator under normal operating conditions. For example, the actuator may comprise first and second underlying actuators, each configured to be actuated via actuation of the actuator, such that the data indicative of performance of the actuator may comprise data indicative of inconsistencies between performance of the first underlying actuator and performance of the second underlying actuator upon manual actuation of the actuator by the user. The data indicative of inconsistencies between performance of the first underlying actuator and performance of the second underlying actuator may comprise an error rate of the first and second underlying actuators, such that the inspection or maintenance service may be predicted to be required if the error rate of the first underlying actuator deviates from the error rate of the second underlying actuator by more than the predetermined threshold value. Additionally, or alternatively, performance of the first and second underlying actuators may comprise a duration of the actuation of the first and second underlying actuators in response to the manual actuation of the actuator by the user.

[0040] In some embodiments, the data indicative of performance of the actuator may comprise a current total amount of travel of the actuator responsive to actuation by the user over a current life of the actuator, such that the baseline value may comprise an expected total amount of travel of the actuator over an expected lifespan of the actuator under normal operating conditions. Accordingly, the inspection or maintenance service may be predicted to be required if the current total amount of travel of the actuator approaches the expected total amount of travel of the actuator by more than a predetermined threshold. Moreover, the actuator may be operatively coupled to the robot arm, and the system further may comprise a second robot arm operatively coupled to a second actuator configured to be actuated to enable a second function of713732261 v3 18225887-081001 the system, such that the fingerprint data may comprise data indicative of inconsistencies between performance of the actuator and performance of the second actuator. For example, the data indicative of inconsistencies between performance of the actuator and performance of the second actuator may comprise an error rate of the actuator and the second actuator, such that the inspection or maintenance service may be predicted to be required if the error rate of the actuator deviates from the error rate of the second actuator by more than the predetermined threshold value.

[0041] The controller may be configured to generate an alert if a proximity sensor of the system switches from an inactive state to an active state. Accordingly, the fingerprint data may comprise data indicative of when the proximity sensor of the system switches between the active state and the inactive state within a predetermined time period, such that the alert may be determined to be a false alert if the proximity sensor does not switch from the active state back to the inactive state within the predetermined time period or if an amount of times that the proximity sensor switches from the inactive state to the active state within the predetermined time period exceeds a predetermined threshold. The predetermined time period may begin when the proximity sensor initially switches from the inactive state to the active state. Moreover, the controller may be configured to adjust the predetermined time period based on a time period that the proximity sensor remains in the active state and / or the amount of times that the proximity sensor switches from the inactive state to the active state within the predetermined time period.

[0042] In addition, the controller may be configured to generate an alert if a braking mechanism of the system is in a disengaged state for a time period that exceeds a predetermined time period. Accordingly, the fingerprint data may comprise data indicative of when the braking mechanism switches between an engaged state and the disengaged state, such that the alert may be determined to be a false alert if the braking mechanism switches from the disengaged state to the engaged state back to the disengaged state and back to the engaged state within a predetermined time period threshold. Moreover, the controller may be configured to detect when operation of the system by a user deviates from a recommended system workflow. For example, the controller may be configured to automatically transition, upon selection of a preset configuration, the robot arm to the preset configuration via the plurality of motors, such that operation of the system by the user deviates from the recommended system workflow when the713732261 v3 19225887-081001 transition of the robot arm to the preset configuration is interrupted. For example, the preset configuration may comprise a retracted configuration of the robot arm, such that operation of the system by the user deviates from the recommended system workflow when the preset configuration is not selected by the user prior to shutdown of the system. Further, the controller may be configured to generate an alert if a number of times that the operation of the system by the user deviates from the recommended system workflow exceeds a predetermined threshold.

[0043] The system further may comprise a stage assembly configured to move the base of the robot arm in one or more degrees of freedom. Accordingly, the controller may be configured to detect when movement of the base in the one or more degrees of freedom via the stage assembly exceeds a predetermined movement threshold, and adjust the predetermined movement threshold based on an amount of times that the movement of the base in the one or more degrees of freedom via the stage assembly exceeds the predetermined movement threshold. Moreover, the fingerprint data may comprise a current total amount of travel of the stage assembly in the one or more degrees of freedom over a current life of the stage assembly, such that the baseline value may comprise an expected total amount of travel of the stage assembly in the one or more degrees of freedom over an expected lifespan of the stage assembly under normal operating conditions. Accordingly, the inspection or maintenance service may be predicted to be required if the current total amount of travel of the stage assembly in the one or more degrees of freedom approaches the expected total amount of travel of the stage assembly in the one or more degrees of freedom by more than a predetermined threshold. In some embodiments, the fingerprint data may comprise a temperature of the system, such that the baseline value may comprise a baseline temperature of the system under normal operating conditions. Additionally, the controller may be configured to detect an occurrence of a fault condition of the system, determine an amount of time between the occurrence of the fault condition and a resolution of the fault condition, and determine a total number of the occurrence of the fault condition over a current life of the system. For example, the fingerprint data may comprise a frequency of the occurrence of the fault condition, such that the baseline value comprises a baseline frequency value of the occurrence of the fault condition under normal operating conditions.

[0044] Moreover, the controller may be configured to execute a predictive maintenance algorithm to determine if the fingerprint data deviates from the baseline value by more than the713732261 v3 20225887-081001 predetermined threshold value, and generate an alert if the fingerprint data deviates from the baseline value by more than the predetermined threshold value, the alert indicative that the inspection or maintenance service is required. For example, the controller may be configured to analyze the fingerprint data over time to identify one or more patterns associated with the operational fingerprint, such that the predictive maintenance algorithm may be configured to compare the one or more patterns to the baseline value to determine if the fingerprint data deviates from the baseline value by more than the predetermined threshold value. The predictive maintenance algorithm may comprise a machine learning algorithm configured to perform predictive analysis, the machine learning algorithm trained with historical data indicative of the operational fingerprints from previous surgical procedures. Additionally, the controller may be configured to cause a display to display the alert. In addition, the controller may be configured to generate a graphical representation of the aggregated fingerprint data over time and the baseline value, and cause a display to display the graphical representation, such that the inspection or maintenance service may be determined to be required based on the displayed graphical representation. The system further may comprise a graphical user interface operatively coupled to the controller, the graphical user interface comprising the display. For example, the graphical user interface may comprise a mobile device comprising a mobile application configured to display the graphical representation.

[0045] In accordance with another aspect, a co-manipulation surgical system to assist with a surgical procedure is provided. The co-manipulation surgical system may comprise a robot arm comprising a plurality of links and joints and a distal end configured to be removably coupled to a surgical instrument for performing the surgical procedure, an optical sensor configured to collect image and / or depth data, and a controller operatively coupled to the optical sensor. The controller may have instructions that, when executed by one or more processors of the controller, cause the controller to: receive the image and / or depth data collected by the optical sensor, the image and / or depth data indicative of at least one object or person within a field of view of the optical sensor; identify the at least one object or person within the field of view of the optical sensor based on the image and / or depth data; determine whether the co-manipulation surgical system is within a predefined authorized clinical zone based on the identified at least one object or person; and automatically provide, if the co-manipulation surgical system is within the713732261 v3 21225887-081001 predefined authorized clinical zone, a level of authorization for access to the co-manipulation surgical system based on the identified at least one object or person.

[0046] For example, the controller may be configured to perform at least one of image classification, objection detection, depth estimation, or temporal analysis to identify the at least one object or person within the field of view of the optical sensor based on the image and / or depth data. In some embodiments, the predefined authorized clinical zone may comprise an operating room. Accordingly, the controller may be configured to determine that the comanipulation surgical system is within the operating room if the identified object comprises capital equipment associated with the operating room. Additionally, or alternatively, the predefined authorized clinical zone may comprise a proximity of a pre-authorized user. Accordingly, the controller may be configured to determine that the co-manipulation surgical system is within the proximity of the pre-authorized user if the identified person comprises a user associated with a predefined level of authorization. Moreover, the controller may be configured to automatically load a user profile associated with the identified person, the user profile comprising the predefined level of authorization associated with the identified person, and automatically provide, if the co-manipulation surgical system is within the proximity of the preauthorized user, the predefined level of authorization for access to the co-manipulation surgical system. In addition, the controller may be configured to restrict access to the co-manipulation surgical system if the co-manipulation surgical system is not within the predefined authorized clinical zone. The co-manipulation surgical system further may comprise a graphical user interface operatively coupled to the controller, such that the controller may be configured to cause the graphical user interface to display an alert indicative of the restricted access if the comanipulation surgical system is not within the predefined authorized clinical zone.BRIEF DESCRIPTION OF THE DRAWINGS

[0047] FIGS. 1 A and IB illustrate an exemplary co-manipulation surgical system constructed in accordance with some embodiments.

[0048] FIG. 2A illustrates movement of an exemplary stage assembly of the platform of the co-manipulation surgical system in accordance with some embodiments.713732261 v3 22225887-081001

[0049] FIG. 2B illustrates proximity sensors within a base of a robot arm of the comanipulation surgical system.

[0050] FIG. 3 A illustrates an exemplary robot arm of the co-manipulation surgical system constructed in accordance with some embodiments.

[0051] FIG. 3B illustrates an exemplary setup joint of the robot arm of FIG. 3 A constructed in accordance with some embodiments.

[0052] FIG. 4 illustrates the degrees of freedom of movement of the shoulder portion and the stages of the co-manipulation surgical system for preset configurations of the platform and robot arms in accordance with some embodiments.

[0053] FIGS. 5 A and 5B illustrate an exemplary surgical instrument coupling mechanism at the distal end of the robot arm of FIG. 3 A constructed in accordance with some embodiments.

[0054] FIG. 6A illustrates the robot arms in a sterile-drape ready configuration, and FIG. 6B illustrates the robot arms covered in a sterile drape.

[0055] FIGS. 7A-7D illustrate the setup of the co-manipulation surgical system in accordance with some embodiments.

[0056] FIG. 8 shows some example components that may be included in a co-manipulation robot platform in accordance with some embodiments.

[0057] FIG. 9A illustrates a field of view of the optical scanner during a laparoscopic surgical procedure, and FIG. 9B illustrates a depth map of the field of view of the optical scanner of FIG. 9A.

[0058] FIG. 9C illustrates a 3D point c / loud generated via a depth map of the field of view of the optical scanner.

[0059] FIG. 9D illustrates a 360 degree field of view of an optical scanner of the system in accordance with some embodiments.

[0060] FIG. 10 illustrates a trocar heat map based on various indications.713732261 v3 23225887-081001

[0061] FIGS. 11 A and 1 IB are free-body diagrams illustrating forces applied to the surgical instrument coupled to the robot arm during a surgical procedure.

[0062] FIG. 12 illustrates an exemplary display showing level of force applied forces applied to the surgical instrument coupled to the robot arm during a surgical procedure.

[0063] FIG. 13 illustrates an exemplary virtual overlay of a graphical user interface of the co-manipulation surgical system.

[0064] FIGS. 14A to 14E show exemplary dashboards illustrating block time utilization in accordance with some embodiments.

[0065] FIG. 15 is a flow chart illustrating robot arm trajectory generation for instrument centering in accordance with some embodiments.

[0066] FIG. 16 is a schematic overview of the data flow of the co-manipulation surgical system in accordance with some embodiments.

[0067] FIG. 17A illustrates an exemplary virtual map of the co-manipulation surgical system within an operating room.

[0068] FIG. 17B illustrates an alternative exemplary virtual map of the co-manipulation surgical system within an operating room generated via combined depth and RGB data in accordance with some embodiments.

[0069] FIGS. 18A to 18C illustrate an exemplary graphical user interface of the comanipulation surgical system displaying a virtual map in accordance with some embodiments.

[0070] FIG. 19A illustrates a conventional surgeon’s preference card, and FIGS. 19B and 19C illustrate exemplary graphical reconstructions of an operating room based on a surgeon’s preference in accordance with some embodiments.

[0071] FIG. 19D illustrates an exemplary graphical display for facilitating patient preparation in accordance with some embodiments.713732261 v3 24225887-081001

[0072] FIGS. 20 A and 20B illustrate exemplary 3D reconstructions of the operating room in accordance with some embodiments.

[0073] FIG. 20C illustrates an exemplary display of a timestamped surgical procedure alongside associated force profile data and 3D reconstruction of the robot arm.

[0074] FIG. 21 A illustrates exemplary timestamped force profile data for various surgical procedures.

[0075] FIG. 21 B illustrates an exemplary dashboard for displaying information aggregated in an interactive surgery summary report in accordance with some embodiments.

[0076] FIG. 22A illustrates an exemplary staff mobile application and FIG. 22B illustrates an exemplary surgeon mobile application.

[0077] FIG. 23 is a flow chart illustrating an exemplary surgical workflow supported with data captured by a wearable in accordance with some embodiments.

[0078] FIGS. 24A and 24B illustrate a schematic overview of data flow of an exemplary large language model of the co-manipulation surgical system in accordance with some embodiments.

[0079] FIGS. 25A to 25FF illustrate fingerprint data of the system for predictive maintenance in accordance with some embodiments.

[0080] FIGS. 26A to 26E illustrate an exemplary graphical user interface of the comanipulation surgical system.

[0081] FIG. 27 is a flow chart illustrating an exemplary security workflow for surrounding context-based authorization in accordance with some embodiments.DETAILED DESCRIPTION

[0082] Disclosed herein are co-manipulation surgical robot systems for assisting an operator, e.g., a surgeon, in performing a surgical procedure, e.g., a laparoscopic procedure, and methods of use thereof. The co-manipulation surgical robot systems described herein provide superior713732261 v3 25225887-081001 control and stability such that the surgeon and / or assistant may seamlessly position various off- the-shelf surgical instruments as needed, thus avoiding the workflow limitations inherent to both human and mechanical solutions. For example, the robot arms of the co-manipulation surgical robot system may provide surgical assistance by holding a first surgical instrument, e.g., a scope such as an endoscope / laparoscope, via a first robot arm, and a second surgical instrument, e.g., a retractor, via a second robot arm, stable throughout the procedure to provide an optimum view of the surgical site and reduce the variability of force applied by the surgical instruments to the body wall at the trocar point. As will be understood by a person having ordinary skill in the art, the robots arms of the co-manipulation surgical robot systems described herein may hold any surgical instrument, preferably having a long and thin instrument shaft, used for surgical procedures such as laparoscopic procedures including, e.g., scopes, retractors, graspers, surgical scissors, needle holders, needle drivers, clamps, suturing instruments, cautery tools, staplers, clip appliers, hooks, etc.

[0083] The co-manipulation surgical robot system further allows the surgeon to easily maneuver both tools when necessary, providing superior control and stability over the procedure and overall safety. Any implementations of the systems described herein enable a surgeon to directly co-manipulate instruments while remaining sterile at the patient bedside. For example, the system may include two robot arms that may be used by the surgeon to hold both a scope, e.g., a laparoscope, and a retractor. During a surgical procedure, the system may seamlessly reposition either instrument to provide optimal visualization and exposure of the surgical field. Both instruments may be directly coupled to the robot arms of the system and the system may constantly monitor and record the position of the two instruments and / or the two robot arms throughout the procedure.

[0084] Moreover, the system may record information such as the position and orientation of surgical instruments attached to the robot arms, sensor readings related to force(s) applied at proximal and distal ends of the surgical instruments attached to robot arms, force required to hold each instrument in position, endoscopic video streams, algorithm parameters, operating room 3D stream captured with an optical scanning device, including, e.g., position(s) of surgical entry port(s) / trocar(s), position and movements of the surgeon’s hands, surgical instrument s) position and orientation, whether or not attached to robot arms, patient position, patient table713732261 v3 26225887-081001 orientation and height, sterile drape, as well as other objects in the operating room, and further may generate a virtual reconstruction, e.g., a 3D reconstruction, of the operating room for training purposes and / or to improve surgical procedure efficiency, as well as for guiding setup of the co-manipulation system. Such data may be used to develop a database of historical data and / or identify user preferences that may be used to develop the algorithms used in some implementations to control one or more aspects of an operation of the system. In addition, such data may be used during a procedure to control one or more aspects of an operation of the system per one or more algorithms of the system, e.g., based on stored user preferences associated with unique surgeon profiles. For example, the data may be used to assess a level of fatigue of a user of the system as described in U.S. Patent No. 11,504,197 to Noonan, the entire contents of which is incorporated herein by reference.

[0085] As the operator manipulates a robot arm of the co-manipulation surgical robot system by applying movement to the surgical instrument coupled to the robot arm, the system may automatically transition the robot arm between various operational modes upon determination of predefined conditions. For example, the system may transition the robot arm to a passive mode responsive to determining that movement of the robot arm due to movement at the handle of the surgical instrument is less than a predetermined amount for at least a predetermined dwell time period, such that in the passive mode, the robot arm maintains a static position, e.g., to prevent damage to the equipment and / or injury to the patient. Additionally, the system may transition the robot arm to a co-manipulation mode responsive to determining that force applied at the robot arm due to force applied at the handle of the surgical instrument exceeds a predetermined threshold, such that in the co-manipulation mode, the robot arm is permitted to be freely moveable responsive to movement at the handle of the surgical instrument for performing the surgical procedure using the surgical instrument, while a first impedance is applied to the robot arm in the co-manipulation mode to account for weight of the surgical instrument and the robot arm, e.g., gravity compensation.

[0086] Moreover, the system may transition the robot arm to a haptic mode responsive to determining that at least a portion of the robot arm is outside a predefined haptic barrier, such that in the haptic mode, a second impedance greater than the first impedance is applied to the robot arm, thereby making movement of the robot arm responsive to movement at the handle of713732261v3 27225887-081001 the surgical instrument more viscous in the haptic mode than in the co-manipulation mode. The system further may transition the robot arm to a robotic assist mode responsive to detecting various conditions that warrant automated movement of the robot arm to guide the surgical instrument attached thereto, e.g., along a planned trajectory or to avoid a collision with another object or person in the surgical space. For example, in an instrument centering mode of the robotic assist mode, a robot arm coupled to a scope may automatically move the scope along a planned trajectory to track an identified surgical instrument and maintain the instrument within the field of view (FOV) of the scope to provide assisted instrument centering, as described in U.S. Patent No. 11,844,583 to Ye and WO 2024 / 150115 to Basafa, the entire contents of each of which are incorporated herein by reference.

[0087] As described in further detail below, the system further may transition the robot arm to one or more setup modes for manual and / or automatic reconfiguration of the robot arm to an optimized position for a given surgical procedure. For example, with knowledge of the location of the trocar port(s) within a surgical site, the system may automatically reconfigure the robot arms to an optimized configuration relative to the trocar ports for a given procedure. The various sub-modes of the robotic assist mode described herein, e.g., instrument centering mode, user- guided setup mode, preset configuration mode, etc., may be actuated via a graphical user interface of the system, and / or via one or more actuation modalities such as gesture detection, voice control (e.g., voice commands by a user captured by one or more microphones of the system), and / or actuators within the vicinity of the robot arms that may receive user input without requiring the user to step away from the robot arms / surgical site. For example, to initiate instrument centering mode of robotic assist mode described in further detail below, the user may provide a voice command instructing the system to track a specific surgical instrument, e.g., a hook, such that, if the system detects a hook within the field of view of the scope, e.g., via object segmentation as described herein, the system will transition the robot arm holding the scope to the instrument centering mode to track the hook without requiring manual intervention, while maintaining continuity and efficiency of the surgical workflow. In some embodiments, responsive to user input, the system further may perform additional tasks normally performed by a human assistant, e.g., make calls, communication instructions to other staff members, etc.713732261 v3 28225887-081001

[0088] Referring now to FIGS. 1 A and IB, co-manipulation surgical robot system 100 is provided. As shown in FIGS. 1A and IB, system 100 may include platform 200, e.g., a surgical cart, sized and shaped to support one or more robot arms, e.g., robot arm 300a and robot arm 300b (collectively referred to herein as robot arms 300), each of robot arms 300 having a surgical instrument coupler interface, e.g., coupler interface 400a and coupler interface 400b, for removably coupling to a surgical instrument, and a computing system operatively coupled to platform 200 and robot arms 300. As shown in FIG. 1 A, system 100 further may include one or more optical scanners, e.g., optical scanner 202a and optical scanner 202b, for capturing depth data, and graphical user interface display 210 for displaying operational information as well as receiving user input.

[0089] As shown in FIGS. 1 A and IB, platform 200 may include a stage assembly, e.g., one or more stages coupled to the base portion of one or more robot arms, e.g., base portion 302a of robot arm 300a and base portion 302b of robot arm 300b, for providing movement to the respective robot arm, e.g., in at least the horizontal and vertical directions relative to platform 200. For example, each stage may include vertical extenders, e.g., vertical extender 206a and vertical extender 206b, for independently moving robot arm 300a and robot arm 300b, respectively, vertically relative to platform 200, and horizontal extenders, e.g., horizontal extender 208a and horizontal extender 208b, for independently moving robot arm 300a and robot arm 300b, respectively, horizontally relative to platform 200, e.g., via one or more stage assembly motors operatively coupled to the vertical and horizontal extenders, to thereby permit the operator flexibility in positioning robot arms 300 relative to the patient. Accordingly, platform 200 may independently move each of robot arm 300a and robot arm 300b in any direction, including a first or vertical direction toward and away from the floor (e.g., along the z- axis), and / or a second or horizontal direction toward and away from the patient (e.g., along the x- axis), as shown in FIG. 2A, and / or a third direction or horizontal direction along a length of the patient (e.g., along the y-axis). Moreover, the system may keep track of each time the stage assembly of platform 200 is actuated, which may be indicative of the occurrence of a surgical procedure, and further may be used to improve efficiency of a subsequent similar surgical procedure (or setup) by the surgeon, e.g., by reducing the amount of times the stage assembly is actuated before / during a procedure.713732261 v3 29225887-081001

[0090] Referring again to FIG. 2A, platform 200 may include a plurality of wheels 204, e.g., castor wheels, to provide mobility of platform 200, and accordingly, robot arms 300, within the operating room. Wheels 204 may each include a braking mechanism which may be actuated to prevent movement of platform 200 via wheels 204. Preferably, wheels 204 may be manually actuated by an operator to mechanically engage / disengage the respective braking mechanism. For example, as shown in FIG. IB, platform 200 may include locking pedal 211a configured to be actuated, e.g., stepped on by a user, to engage the braking mechanism, and unlocking pedal 211b configured to be actuated, e.g., stepped on by a user, to disengage the braking mechanism. When ready for operation, platform 200 may be moved to a desired position at the side of the patient bed and locked in place via wheels 204, and the vertical and horizontal positions of robot arms 300a and 300b may be manually or automatically adjusted to an optimum position relative to the patient for the procedure via vertical extenders 206a, 206b and horizontal extenders 208a, 208b, responsive to user input received by graphical user interface display 210, and / or via user guided stage control as described in further detail below. In addition, as shown in FIG. 2A, platform 200 further may include one or more additional actuators, e.g., foot pedal 213, configured to be actuated to provide user input with regard to one or more operational modes, as described in further detail below.

[0091] Referring again to FIG. 1A, system 100 may include a plurality of depth sensors, e.g., proximity sensors 212, disposed on platform 200. Proximity sensors 212 may be, e.g., a depth camera, a stereo RGB camera, a LIDAR device, a 360 camera, and / or an electromagnetic, capacitive, ultrasound, or infrared proximity sensor, etc. As shown in FIG. 1A, a first set of proximity sensors 212 may be positioned on robot arm 300a, e.g., at a lower portion of base portion 302a, and a second set of proximity sensors 212 may be positioned on robot arm 300b, e.g., at a lower portion of base portion 302b, to thereby enhance detection of objects approaching the vicinity of robot arms 300a, 300b. For example, as shown in FIG. 2B, each base portion 302 may include a set of proximity sensors, e.g., forward proximity sensor 212a for detecting and determining the proximity of objects in front of and around base portion 302 and bottom proximity sensor 212b for detecting and determining the proximity of objects in beneath and around base portion 302 (collectively referred to herein as proximity sensors 212). As will be713732261 v3 30225887-081001 understood by a person having ordinary skill in the art, each base portion may have less or more than two proximity sensors.

[0092] Surgical robot system 100 is configured for co-manipulation, such that system 100 may assist the user or operator, e.g., a surgeon and / or surgical assistant, by permitting the user to freely move robot arm 300a and / or robot arm 300b due to manipulation of one or more surgical instruments coupled with the robot arms in response to force applied by the user to the surgical instruments. Accordingly, system 100 may be configured so that it is not controlled remotely during operation, e.g., via a remote surgeon console used in teleoperated robot systems, such that robot arms 300 move directly responsive to movement of the surgical instrument coupled thereto by the operator, while compensating for the mass of the surgical instrument and of the respective robot arm and providing localized impedance along the robot arm, thereby increasing the accuracy of the movements or actions of the operator as the operator manipulates the surgical instrument.

[0093] System 100 may be particularly useful in laparoscopic surgical procedures and / or other surgical procedures that utilize long and thin instruments that may be inserted, e.g., via trocars / cannulas, into the body of a patient to allow surgical intervention. As will be understood by a person having ordinary skill in the art, system 100 may be used for any desired or suitable surgical operation. Moreover, system 100 may be used in conjunction or cooperation with video monitoring provided by one or more cameras and / or one or more endoscopes so that an operator of system 100 may view and monitor the use of the instruments coupled with robot arms 300a, 300b via respective coupler interfaces 400a, 400b. For example, robot arm 300a may be removably coupled with and manipulate an endoscope, while robot arm 300b may be removably coupled with and manipulate a surgical instrument.

[0094] As shown in FIG. 1 A, system 100 may include one or more optical scanners, e.g., optical scanners 202a, 202b (collectively referred to herein as optical scanners 202), e.g., a LiDAR scanner or other suitable optical scanning device such as an RGBD camera or sensor, RGB camera with machine learning, a time-of-flight depth camera, structured light, multiple projection cameras, a stereo camera, ultrasound sensors, laser scanner, other type of coordinate measuring area scanner, a 360 camera, or any combination of the foregoing, for providing a713732261 v3 31225887-081001 video stream of the surgical scene, e.g., via streaming, for monitoring and analysis, as described in U.S. Patent No. 11,980,431 to Alvarez, the entire contents of which are incorporated herein by reference. For example, the LiDAR camera / scanner may be capable of recording both color (RGB) and the Depth (D) of the surgical field, and may include, for example, an Intel RealSense LiDAR Camera L515 or an Intel RealSense Depth Camera D435i (made available by Intel, Santa Clara, California), a Mid-360 LiDAR sensor (made available by Livox, Shenzhen, Guangdong, China), or other LiDAR or depth cameras having similar or suitable specifications. As will be understood by a person having ordinary skill in the art, “image data” as used herein may refer to depth data and / or color RGB data collected by the optical sensors / scanners described herein. The 360 camera may include a plurality of individual RGB cameras spatially distributed relative to each to enable capture of image data in 360 degrees about the 360 camera, thereby obtaining a comprehensive scan of the operating room. For example, the 360 camera may include, e.g., GMSL RGB cameras (made available by Leopard Imaging Inc., Fremont, California).

[0095] Optical scanners 202, and any other electronics, wiring, or other components of the system, may be supported via platform 200 such that optical scanners 202 are mounted in a fixed location relative to the other objects in the surgical space, and the position and orientation of optical scanners 202 are known or may be determined with respect to the global coordinate system of the system, and accordingly, the robot arms. This allows all data streams to be transformed into a single coordinate system for development purposes. Moreover, telemetry data captured by optical scanners 202, e.g., indicative of the movements of the surgeon’s hands, other body parts, the patient bed, the cut-out in a sterile drape over the patient on the surgical bed, the exposed skin through the cut-out in the sterile drape, the trocar(s), the surgical instruments, and other components of the system, may be selectively recorded to provide a rich and detailed dataset describing the precise movements and forces applied by the surgeon throughout the procedure, along with corresponding timestamps. As described in further detail below, the telemetry data further may be used to generate a 3D reconstruction of the operating room to provide an interactive review of the recorded surgical procedure, e.g., for setup and / or training purposes.

[0096] As shown in FIG. 1 A, a first optical scanner, e.g., optical scanner 202a, may be supported on an upper portion of platform 200, e.g., via lighthouse 203, and may be adjusted,713732261 v3 32225887-081001 e.g., up / down, in / out, right / left, to adjust the FOV of optical scanner 202a to allow optical scanner 202a to gain an optimum FOV or position relative to the other components of the system and / or other persons or objects in the operating room, for example, robot arms 300a, 300b, the surgical instruments attached thereto, the surgeon, one or more surgical assistants, the surgical bed, the sterile drape, etc. For example, optical scanner 202a may collect depth data indicative of, e.g., the height of the surgical bed, the angle of the surgical bed (cranial to caudal, and medial to lateral), the plane of the surgical bed, the cranial end of the surgical bed, the position and orientation of the surgical bed, the location of one or more trocar ports, movement of a surgical instrument coupled to the distal end of the robot arm, movement of a handheld surgical instrument not coupled to the robot arm, e.g., held by a user, attachment and detachment of a surgical instrument to the distal end of the robot arm, etc. In some embodiments, optical scanner 202a may be mounted on lighthouse 203 via a periscope configured to adjust a height of the optical scanner relative to platform 200, and / or may be rotatable in 360 degrees relative to platform 200, to thereby maximize the FOV of the optical scanner within the operating room.

[0097] As shown in FIG. 1 A, a second optical scanner, e.g., optical scanner 202b, may be supported on a lower portion of platform 200 to allow optical scanner 202b to provide the system a more complete FOV of the operating room that may not be captured by first optical scanner 202a, e.g., the patient table and objects on the floor of the operating room such as electrical cables. For example, optical scanner 202b may collect depth data indicative of, e.g., the distance / proximity between system 100 and a surgical bed, the relative angle between system 100 and the surgical bed, closest feature on an edge of the surgical bed, the cranial and caudal ends of the surgical bed, one or more objects / persons between system 100 and the surgical bed, one or more objects / persons on the other side of the surgical bed, surgical drapes, etc. As will be understood by a person having ordinary skill in the art, more than two optical scanners may be used to further enhance the FOV of the system.

[0098] The data obtained by the optical scanners may be used to optimize the procedures performed by the system including, e.g., automatic servoing (i.e., moving) of one or more portions of robot arms 300. By tracking the tendency of the surgeon to keep the tools in a particular region of interest and / or the tendency of the surgeon to avoid moving the tools into a particular region of interest, the system may optimize the automatic servoing algorithm to713732261 v3 33225887-081001 provide more stability in the particular region of interest. In addition, the data obtained may be used to optimize the procedures performed by the system including, e.g., automatic re-centering of the FOV of the optical scanning devices of the system. For example, if the system detects that the surgeon has moved or predicts that the surgeon might move out of the FOV, the system may cause the robot arm supporting the optical scanning device, e.g., a scope, to automatically adjust the scope to track the desired location of the image as the surgeon performs the desired procedure, as described in further detail below. This behavior may be surgeon-specific and may require an understanding of a particular surgeon’s preference for an operating region of interest. Additionally, or alternatively, this behavior may be procedure-specific. Thus, the system may control the robot arms pursuant to specific operating requirements and / or preferences of a particular surgeon.

[0099] As described above, each surgeon’s preferences may be stored in a surgeon profile associate with the surgeon, such that upon execution of the surgeon profile, the system may implement the surgeon’s preferred settings when using, e.g., instrument centering mode. For example, the surgeon’s preferred settings may include camera pivoting or zooming speed adjustment, whether to use constant or distance-based adaptive scope motion during instrument centering mode, the surgeon’s dominant hand and whether to automatically pair an instrument to be followed based on the direction of the introduction of the instrument within the FOV of the scope, the force required to perform user override during instrument centering mode, e.g., transition the robot arm holding the scope from passive mode to co-manipulation mode during instrument centering mode, the size of the predefined center portion of the FOV of the scope where a surgical instrument will be paired to be followed when held therein for at least a predetermined hold period, the size of the predefined boundary region within the field of the view of the scope, etc. Moreover, the data obtained by the optical scanners may be used to keep track of predefined phases of a surgical procedure, e.g., draping / undraping, instrument coupling / decoupling, etc., and further to improve language learning models described herein by corroborating actions of the robot arms / system in response to user voice commands, as described in further detail below.

[0100] Referring now to FIG. 3A, a surgical support arm is provided. As described above, system 100 may include a plurality of robot arms, e.g., robot arm 300a and robot arm 300b;713732261 v3 34225887-081001 however, as each robot arm may be constructed identically, only a single robot arm, e.g., robot arm 300, is described with regard to FIG. 3 A for brevity. Aspects of the robot arms described herein may utilize structures and software from U.S. Patent No. 10,118,289 to Louveau, U.S. Patent No. 11,504,197 to Noonan, U.S. Patent No. 11,622,826 to Basafa, U.S. Patent No. 11,812,938 to Wu, U.S. Patent No. 11,844,583 to Ye, U.S. Patent No. 12,042,241 to Wu, U.S. Patent No. 12,370,001 to Basafa, WO 2023 / 203491 to Gayet, and WO 2024 / 150115 to Basafa, the entire contents of each of which are incorporated herein by reference. Robot arm 300 may include a plurality of arm segments / links and a plurality of articulation joints extending from a base portion. For example, robot arm 300 may include a base portion, a shoulder portion, an elbow portion, and a wrist portion, thereby mimicking the kinematics of a human arm. As shown in FIG. 3 A, robot arm 300 may include a base, which includes base portion 302 rotatably coupled to shoulder portion 304 at base joint 303. For example, shoulder portion 304 may sit on top of base portion 302, and may be rotated relative to base portion 302 about axis QI at base joint 303. In some embodiments, robot arm 300 may be interchanged, swapped, or coupled with the base in any desired arrangement. Moreover, as described above, the base of robot arm 300 may be mounted on platform 200, and selectively moved relative to platform 200 via the stage assembly of platform 200.

[0101] Robot arm 300 further may include shoulder link 305, which includes proximal shoulder link 306 rotatably coupled to distal shoulder link 308. A proximal end of proximal shoulder link 306 may be rotatably coupled to shoulder portion 304 of the base at shoulder joint 318, such that proximal shoulder link 306 may be rotated relative to shoulder portion 304 about axis Q2 at shoulder joint 318. As shown in FIG. 3A, axis Q2 may be perpendicular to axis QI. The distal end of proximal shoulder link 306 may be rotatably coupled to the proximal end of distal shoulder link 308 at joint 320, such that distal shoulder link 308 may be rotated relative to proximal shoulder link 306 about axis Q3 at joint 320. As shown in FIG. 3 A, axis Q3 may be parallel to the longitudinal axis of shoulder link 305.

[0102] In addition, robot arm 300 may include actuator 330, e.g., a collar, lever, button, or switch, operatively coupled to a motor operatively coupled to distal shoulder link 308 and / or proximal shoulder link 306 at joint 320, such that distal shoulder link 308 may only be rotated relative to proximal should link 306 upon actuation of actuator 330. Actuator 330 may be713732261 v3 35225887-081001 configured to permit dual actuation. For example, as shown in FIG. 3A, actuator 330 may be a collar rotatably coupled to a link of robot arm 300, e.g., elbow link 310 described below, such that rotation of collar 330 in a first direction about the longitudinal axis of link 310 may cause distal shoulder link 308 to rotate in a corresponding first direction relative to proximal shoulder link 306, and rotation of collar 330 in a second direction about the longitudinal axis of link 310 opposite to the first direction may cause distal shoulder link 308 to rotate in a corresponding second direction relative to proximal shoulder link 306 opposite to the corresponding first direction.

[0103] As shown in FIG. 3 A, collar 330 may include setup mode actuator 336 disposed thereon, e.g., a button, which the system may require to be actuated to permit a rotation of collar 330 to cause a corresponding rotation of distal shoulder link 308 relative to proximal shoulder link 306. Preferably, a single setup mode actuator 336 may include two underlying buttons for redundancy, for example, to ensure functionality of setup mode actuator 336 and facilitate detection of a fault condition, e.g., when the readings between the two underlying buttons differ. Accordingly, actuation of setup mode actuator 336 by the user results in actuation of both underlying buttons under normal operating conditions. For example, the user may be required to actuate setup actuator 336 to switch the system to the user-guided setup mode, and maintain setup actuator 336 in an actuated state while collar 330 is rotated to cause a corresponding rotation of distal shoulder link 308 relative to proximal shoulder link 306. In addition to actuating setup actuator 336 to permit rotation of collar 330 to cause rotation of distal shoulder link 308 relative to proximal shoulder link 306, in some embodiments, actuation of setup actuator 336 in a predefined manner / pattern may serve as an input to the system for enabling additional functionalities without requiring the user to move away from robot arm 300 during a surgical procedure. For example, actuating setup actuator 336 in a predefined pattern, e.g., twice quickly, may initiate a predetermined sub-mode / function of the system, e.g., instrument centering mode. Moreover, actuating setup actuator 336 in another distinct predefined pattern may initiate another predetermined sub-mode / function of the system, e.g., recording of video data captured by the one or more optical scanners of the system, and actuating setup actuator 336 in another distinct predefined pattern may stop the video recording.713732261 v3 36225887-081001

[0104] Alternatively, or additionally, instead of actuating collar 330 to cause rotation of distal shoulder link 308 relative to proximal shoulder link 306, in some embodiments, actuation of setup actuator 336 may initiate a user-guided setup mode where application of a force to the distal end of the robot arm may cause a corresponding movement of robot arm 300, and / or the stage assembly of platform 200, as described in further detail below. For example, in the user- guided setup mode, application of a force to the distal end of the robot arm, e.g., a left / right force, may cause distal shoulder link 308 to rotate in a corresponding direction relative to proximal shoulder link 306. Distal shoulder link 308 may continue to be rotated relative to proximal shoulder link 306 until the applied force is released and / or a counter force in an opposite direction is applied to the distal end of the robot arm, and / or until a maximum rotation is reached.

[0105] Accordingly, motorized axis Q3 may be a “setup” axis, such that distal shoulder link 308 may be automatically rotated and fixed relative to proximal shoulder link 306 upon actuation of actuator 330 and / or setup actuator 336, e.g., during a setup stage of robot arm 300, prior to operation of robot arm 300 in a surgical procedure. As described in further detail below, M4 operatively coupled to joint 320, e.g., a setup joint, must be actuated, e.g., via actuator 330 and / or GUI 210, to automatically rotate distal shoulder link 308 relative to proximal shoulder link 306 at joint 320. As shown in FIG. 3B, motor M4 may be operatively coupled to a motion transmission mechanism coupled to distal shoulder link 308, e.g., worm gear 323, via gear 321, such that actuation of motor M4 causes rotation of distal shoulder link 308 relative to proximal shoulder link 306 via engagement between gear 321 and worm gear 323. In addition, the system may switch between the operating stage and the setup stage during a surgical procedure to permit reconfiguration of the robot arm via the setup joints as needed. For example, the setup stage may occur prior to the operating stage, and the system may switch back to the setup stage during the operating stage to permit reconfiguration of robot arm 300 if necessary.

[0106] Referring again to FIG. 3 A, robot arm 300 further may include elbow link 310. A proximal end of elbow link 310 may be rotatably coupled to a distal end of distal shoulder link 308 at elbow joint 322, such that elbow link 310 may be rotated relative to distal shoulder link 308 about axis Q4 at elbow joint 322. Robot arm 300 further may include wrist portion 311, which may include proximal wrist link 312 rotatably coupled to the distal end of elbow link 310713732261 v3 37225887-081001 at wrist joint 324, middle wrist link 314 rotatably coupled to proximal wrist link 312 at joint 326, and distal wrist link 316 coupled to / extending from middle wrist link 314, which may be rotatably coupled to surgical instrument coupler interface 400 (not shown) at joint 328, as further described in further detail with regard to FIGS. 5 A and 5B. Accordingly, wrist portion 311 may be rotated relative to elbow link 310 about axis Q5 at wrist joint 324, middle wrist portion 314 may be rotated relative to proximal wrist link 312 about axis Q6 at joint 326, and surgical instrument coupler interface 400 may be rotated relative to distal wrist link 316, and accordingly middle wrist link 314, about axis Q7 at joint 328.

[0107] Referring again to FIG. 3A, robot arm 300 may include actuator 332, e.g., a clutch lever, button, or switch, operatively coupled to elbow link 310 and / or proximal wrist link 312 at joint 324, e.g., a setup joint, such that proximal wrist link 312 may only be rotated relative to elbow link 310 upon actuation of actuator 332. Accordingly, axis Q5 may be a “setup” axis, such that proximal wrist link 312 may be rotated and fixed relative to elbow link 310 during the setup stage, upon actuation of actuator 332. When actuator 332 is in an unactuated state, setup joint 324 prevents relative movement between proximal wrist link 312 and elbow link 310, such that proximal wrist link 312 is fixed relative to elbow link 310. In some preferred embodiments, upon actuation of actuator 332, proximal wrist link 312 may be manually rotated in predefined increments relative to elbow link 310, thereby removing the necessity of having additional motors and / or electronics at the distal region of robot arm 300.

[0108] As shown in FIG. 3A, robot arm 300 may include a plurality of motors, e.g., motors Ml, M2, M3, which may all be disposed within the base of robot arm 300, and M4, which preferably may be disposed adjacent to joint 320. Alternatively, motor M4 also may be disposed within the base of robot arm 300. In addition, as described above, system 100 may include one or more stage assembly motors operatively coupled to the stage assembly of platform 200, preferably disposed within platform 200. Each of motors Ml, M2, M3, may be operatively coupled to a respective motorized joint of robot arm 300, e.g., base joint 303, shoulder joint 318, and elbow joint 322, to thereby apply a localized impedance at the respective joint. For example, motors Ml, M2, M3 may produce an impedance / torque at any of base joint 303, shoulder joint 318, and elbow joint 322, respectively, to thereby effectively apply a desired impedance at the distal end of robot arm, e.g., at the attachment point with the surgical instrument, to improve the713732261 v3 38225887-081001 sensations experienced by the operator during manipulation of the surgical instrument as well as the actions of the operator during surgical procedures. For example, impedance may be applied to the distal end of robot arm 300, and accordingly the surgical instrument coupled thereto, to provide a sensation of a viscosity, a stiffness, and / or an inertia to the operator manipulating the surgical instrument.

[0109] Moreover, applied impedances may simulate a tissue density or stiffness, communicate surgical boundaries to the operator, and may be used to direct a surgical instrument along a desired path, or otherwise. In some embodiments, the motors may actuate the respective joints to thereby cause movement of robot arm 300 about the respective joints. Accordingly, axis QI, axis Q2, and axis Q4 may each be a “motorized” axis, such that motors Ml, M2, M3 may apply an impedance / torque to base joint 303, shoulder joint 318, and elbow joint 322, respectively, to inhibit or actuate rotation about the respective axis. As described in further detail below, motors Ml, M2, M3 may be controlled by a processor of the co-manipulation robot platform. With three motorized axes, some implementations of robot arm 300 may apply force / torque at the distal end of robot arm 300 in three directions to thereby move the surgical instrument coupled to the distal end of robot arm 300 in three degrees of freedom.

[0110] As described above, motor M4 may be operatively coupled to setup joint 320 to thereby apply a torque to joint 320 to actuate rotation of distal shoulder link 308 relative to proximal shoulder link 306 about axis Q3. Unlike the other motorized joints described herein, e.g., base joint 303, shoulder joint 318, and elbow joint 322, motorized joint 320 is preferably not “back-drivable,” in that the user cannot actuate motorized joint 320, e.g., via movement of the surgical instrument coupled to the robot arm when the system is in co-manipulation mode. Instead, as described above, actuation of motorized joint 320 may be conducted via one or more actuators, e.g., actuator 330, setup actuator 336, and / or an actuator displayed on GUI 210, that may be actuated to automatically cause rotation of distal shoulder link 308 relative to proximal shoulder link 306. Moreover, the system may keep track of each time motorized joint 320 is actuated, which may be indicative of the occurrence of a surgical procedure, and further may be used to improve efficiency of a subsequent similar surgical procedure (or setup) by the surgeon, e.g., by reducing the amount of times motorized joint 320 is actuated before / during a procedure.713732261 v3 39225887-081001

[0111] Axis Q6 and axis Q7 may each be a “passive” axis, such that middle wrist link 314 may be rotated relative to proximal wrist link 312 at passive joint 326 without any applied impedance from system 100, and surgical instrument coupler interface 400 may be rotated relative to distal wrist link 316 at passive joint 328 without any applied impedance from system 100. The distal end of distal wrist link 316 may be rotatably coupled to surgical instrument coupler interface 400, e.g., at a passive joint, for removably coupling with a surgical instrument, e.g., via coupler body 500 as shown in FIGS. 5A and 5B, which may be removably coupled to the surgical instrument and to coupler interface 400, as described in further detail below. Alternatively, wrist portion 311 may include a passive ball joint at the attachment point with the surgical instrument, as described in U.S. Patent No. 10,582,977, the entire content of which is incorporated herein by reference.

[0112] Referring again to FIG. 3A, robot arm 300 further may include a plurality of encoders, e.g., encoders E1-E7, disposed on at least some of the plurality of joints of robot arm 300 for measuring angulation and / or angular rotation between adjacent links of robot arm 300. The encoders may be absolute encoders or other position / angulation sensors configured to generate data for accurately determining the position and / or angulation of corresponding links at the respective joint and / or the exact position of the surgical instrument coupled to the distal end of robot arm 300. Accordingly, the exact position of each link, joint, and the distal end of robot 300 may be determined based on measurements obtained from the plurality of encoders. Preferably, a redundant encoder is disposed at each location along robot arm 300 where an encoder is placed, to provide more accurate position data, as well as, to facilitate detection of a fault condition, e.g., when the readings between an encoder and a redundant encoder differs.

[0113] Prior to attachment with a surgical instrument, robot arm 300 may be manually and / or automatically manipulated by a user, e.g., via actuation of setup joints 320, 324 and / or the stage assembly of platform 200 to position robot arm 300 in a desired position for coupling with the surgical instrument. For example, as described above, upon actuation of actuator 330 and / or actuator 336, the user may automatically rotate distal shoulder link 308 relative to proximal shoulder link 306, and upon actuation of actuator 332, the user may manually manipulate proximal wrist portion 312 relative to distal shoulder link 308. Moreover, robot arm 300 may713732261 v3 40225887-081001 further be manually moved by application of a force directly on the other links and / or joints of robot arm 300.

[0114] Upon attachment to the surgical instrument, robot arm 300 may still be manipulated manually by the user exerting force, e.g., one or more linear forces and / or one or more torques, directly to robot arm 300; however, in the operating stage, the operator preferably manipulates robot arm 300 only via the handle of the surgical instrument, which applies force / torque to the distal end of the robot arm 300, and accordingly the links and joints of robot arm 300. As the operator applies a force to the surgical instrument attached to robot arm 300, thereby causing movement of the surgical instrument, robot arm 300 will move responsive to the movement of the surgical instrument to provide the operator the ability to freely move surgical instrument relative to the patient. As will be understood by a person having ordinary skill in the art, robot arm 300 may include less or more articulation joints than is shown in FIG. 3A, as well as a corresponding number of motors and encoders / sensors.

[0115] In addition, each of robot arms 300 further may include indicators 334 for visually indicating the operational mode associated with the respective robot arm in real-time. For example, indicators 334 may be positioned on at least one of shoulder portion 304, shoulder link 305, elbow link 310, platform 200, lighthouse 203, display 210, etc. Indicators 334 may include lights, e.g., LED lights, and may be programmed to illuminate in a predetermined amount of colors and in distinct patterns, e.g., solid on or blinking, to thereby convey predetermined statuses of the overall system and / or specific statuses of the system and the robot arm, e.g., when a coupler is mounted on the robot arm, the current operational mode of the robot arm, etc., as described in, for example, U.S. Patent No. 11,504,197. The specific color associated with a predetermined status may be customized, e.g., surgeon specific, and stored in a surgeon profile associated with the user for subsequent procedure performed by the surgeon, as described in further detail below.

[0116] Moreover, system 100 may automatically move each of robot arms 300 to one or more preset configurations, e.g., via the motorized joints of the robot arms, upon selection of the preset configuration, e.g., via GUI 210, during the setup stage. The preset configurations may be user specific based on user preference for a given surgical procedure and may be stored in a713732261 v3 41225887-081001 surgeon profile associated with the user, as described in U.S. Patent No. 12,042,241 to Wu. For example, a user’s surgeon profile may contain the user’s specific settings and information, including, e.g., username; level of expertise; different procedures performed, and / or region of clinical practice. In addition, the clinical procedure may require a user to store specific settings such as clinical procedure (e.g., cholecystectomy, hernia, etc.), table orientation and height, preferred port placement, settings per assistant arm for each algorithm, patient characteristics (e.g., BMI, age, sex), and / or surgical tools characteristics and specifications (e.g., weights, length, center of gravity, etc.). The user may be able to enable his own profile, and optionally may enable another user’s profile, such as the profile of a peer, the most representative profile of a surgeon of the user’s area of practice, the most representative profile of a surgeon with a specific level of expertise, and / or the recommended profile according to patient characteristics. The identification of a user may be performed via password, RFID key, facial recognition, etc.

[0117] Moreover, the preset configuration for a given surgical procedure may be specific to which side of the patient table the system is positioned, the location of the trocar port(s), which quadrant of the patient body the surgical procedure will take place on, and / or the anticipated configuration of the surgical bed. For example, the quadrant of the patient body and the anticipated surgical bed configuration may drive the preset configurations as they both dictate in which direction, e.g., up or down relative to the horizontal plane, a scope may need to be pointed for a given surgical procedure. Moreover, based on the identified location of the trocar(s), as described in further detail below, the preset configurations may be adjusted by the system in realtime to maintain a predefined distance / orientation relative to the trocar(s), thereby further optimizing the configuration of robot arms 300 for a given surgical procedure.

[0118] The system may cause the platform and / or robot arms to move to each preset configuration by causing movement of the platform and / or robot arms in a limited number of degrees of freedom. For example, as shown in FIG. 4, the shoulder portion of each robot arm, 300a, 300b may be rotated about the respective Q3 axis, e.g., in a left or right direction from a neutral configuration, and each of base 302a, 302b may be moved via the stage assembly of the platform, e.g., in / out along the horizontal plane and up / down along the vertical plane. Accordingly, based on the limited degrees of freedom of movement of the shoulder portions of the robot arm and stage assembly of the platform, the system may cause the platform and / or713732261 v3 42225887-081001 robot arms to move to any one of a plurality of preset configurations based on the selected surgical procedure. Moreover, the system may keep track of each time the system is deployed to a preset configuration, which may be indicative of the occurrence of a surgical procedure.

[0119] Referring now to FIGS. 5A and 5B, a close-up view of the coupling mechanism of coupler interface 400 and coupler body 500 is provided. The coupling mechanism may be constructed as described in U.S. Patent No. 11,812,938 or PCT / IB2025 / 057828, the entire contents of each of which are incorporated herein by reference. For example, the coupling mechanism may include coupler interface 400 at the distal end of the distal-most link of the robot arm (illustratively, link 316), and coupler body 500, which may be configured to be removably coupled to a surgical instrument and to coupler interface 400, such that a sterile drape may be placed between coupler interface 400 and coupler body 500. Accordingly, coupler body 500 may be disposable, or alternatively, sterilizable between surgical procedures. Moreover, the coupling mechanism may be operatively coupled to one or more sensors for detecting when coupler body 500 is coupled to coupler interface 400, and when a surgical instrument is coupled to coupler body 500 when coupler body 500 is coupled to coupler interface 400, as well as the type / size / make of the surgical instrument coupled to coupler body 500.

[0120] Coupler interface 400 may be rotatably coupled to the distal end of distal wrist link 316 using any suitable fasteners or connectors, e.g., magnets, screws, pins, clamps, welds, adhesive, rivets, and / or any other suitable faster or any combination of the foregoing. In addition, as described in U.S. Patent No. 11,812,938, coupler interface 400 may include a repulsion magnet disposed therein. The repulsion magnet is configured to apply a magnetic force to a magnet slidably disposed within coupler body 500 to facilitate determination of when coupler body 500 is coupled to coupler interface 400 and no surgical instrument is coupled to coupler body 500, and / or to facilitate coupling of the surgical instrument to coupler body 500. Moreover, coupler interface 400 may include a metal rod, e.g., a ferrous rod, extending therethrough, and link 316 may include one or more sensors, e.g., one or more Hall effect sensors positioned adjacent to a proximal end of the ferrous rod, configured to detect a magnetic field induced in the ferrous rod. Preferably, link 316 includes at least two Hall effect sensors to provide redundancy for more accurate magnetic field measurements. Moreover, as described above, robot arm 300 may include one or more encoders E7 for measuring angulation of between713732261 v3 43225887-081001 distal wrist link 316 and surgical instrument coupler interface 400 may be disposed on or adjacent to joint 328, e.g., within link 316. For example, encoders E7 may include two or more encoders positioned circumferentially around the extended portion of coupler interface 400.

[0121] Coupler body 500 may be configured to be removably coupled to a surgical instrument having a predefined shaft diameter, e.g., a 5 mm or 10 mm surgical instrument. Coupler body 500 is preferably designed to be locked to the distal end of the robot arm with a sterile drape therebetween such that the robot arm remains covered and sterile throughout a procedure. Accordingly, prior to coupling coupler body 500 to coupler interface 400, a sterile drape may be positioned between coupler body 500 and coupler interface 400, such that the sterile drape may be draped over robot arm 300. Further, coupler body 500 also has a separate portion for locking to a surgical instrument (e.g., a commercially available surgical instrument) to permit the clinician to perform the surgeries with the robot arm(s) as described herein. For example, in its locked state, coupler body 500 may apply a friction force against the surgical instrument that prevents longitudinal movement of the instrument relative to coupler body 500, while permitting rotational movement of the instrument within coupler body 500.

[0122] As described in U.S. Patent No. 11,812,938, coupler body 500 may include a magnet slidably disposed within coupler body 500. The magnet may have a magnetic force such that when coupler body 500 is coupled to coupler interface 400, the magnet induces a magnetic field, which may be detected by one or more magnetic field sensors, e.g., disposed within link 316, as described above. Accordingly, the strength of the induced magnetic field will be proportional to the distance between the magnet and coupler interface 400 such that the magnetic field detected by the magnetic field sensors may be indicative of the position of the magnet within coupler body 500. Similarly, when no magnetic field is induced via the magnet, the magnetic field sensors may detect that coupler body 500 is not coupled to coupler interface 400. Moreover, the repulsion magnet of coupler interface 400 may have a magnetic force such that, when coupler body 500 is coupled to coupler interface 400, the repulsion magnet applies a magnetic force to the magnet of coupler body 500 to thereby cause the magnet to move away from coupler interface 400. Accordingly, the position of the magnet within coupler body 500 may be indicative of whether a surgical instrument is or is not coupled to coupler body 500 when coupler body 500 is coupled to coupler interface 400. For example, the system may determine that713732261 v3 44225887-081001 coupler body 500 is coupled to coupler interface 400 and that a surgical instrument is coupled to coupler body 500, based on the strength of the magnetic field induced by the magnet. Moreover, the system may keep track of each time a coupler body is coupled to and removed from the coupled interface at the distal end of the robot arm and / or each time a surgical instrument is coupled to and removed from coupler body 500, e.g., based on the signals observed by the magnetic field sensors, which may be indicative of the occurrence of a surgical procedure.

[0123] Referring now to FIG. 6A, robot arms 300 may be positioned in a surgical drapeready configuration (e.g., preset “drape mode”). For example, upon actuation of the drape mode, e.g., via GUI 210, the system may automatically cause robot arms 300a, 300a to be extended such that wrist portions 311a, 311b, elbow links 310a, 310b, and shoulder links 305a, 305b extend away from shoulder portions 304a, 304b of the respective base, as shown in FIG. 6 A, to permit a surgical / sterile drape to be draped over each component of robot arms 300a, 300b, as shown in FIG. 6B. This configuration permits efficient and accessible draping of the respective robot arms with a single surgical / sterile drape. Preferably, robot arms 300a, 300b extend away from platform 200 such that shoulder links 305a, 305b extend at an angle away from shoulder portions 304a, 340b, and wrist portions 311a, 311b, and elbow links 310a, 310b are substantially parallel to each other and to the ground, as shown in FIG. 6A. Moreover, the system may keep track of each time the drape mode is actuated, which may be indicative of the occurrence of a surgical procedure.

[0124] As shown in FIG. 6B, a single sterile drape, e.g., sterile drape 800 having first drape portion 801a sized and shaped for draping robot arm 300a and second drape portion 801b sized and shaped for draping robot arm 300b, may be used to drape both robot arms 300a, 300b and at least the front side of platform 200, as described in U.S. Patent No. 12,167,900 to Wu, the entire contents of which are incorporated herein by reference. Drapes having a sealed end portion without any openings, and being sealed along a length thereof may provide a better sterile barrier for system 100. Accordingly, all of robot arms 300a, 300b may be located inside sterile drape 800 and / or be fully enclosed within sterile drape 800, except at an opening at a proximal end of sterile drape 800, e.g., near the base of robot arms 300. The presence of sterile drape 800 may be detected via, e.g., the optical scanners of system 100, which may be indicative of the occurrence of a surgical procedure.713732261 v3 45225887-081001

[0125] Referring now to FIGS. 7A to 7D, setup of the co-manipulation surgical system is provided. As shown in FIG. 7A, platform 200 may be moved to a desirable position relative to patient table PT by a user, e.g., via wheels 204, while robot arms 300a, 300b are in their respective stowed configurations. As shown in FIG. 7B, when platform 200 is in its desired position relative to patient table PT, such that wheels 204 are locked, robot arms 300a, 300b may be extended away from their respective stowed configurations. As shown in FIG. 7C, the vertical position of the robot arms relative to platform 200 may be adjusted to the desired position, and as shown in FIG. 7D, the horizontal position of the robot arms relative to platform 200 may be adjusted to the desired position. As described above, the desired positions of the robot arms may be stored as an “operation-ready” preset configuration, which may be specific to the operation being performed, as well as the surgeon’s preferences. Accordingly, when platform 200 is in the desired position relative to patient table PT, as shown in FIG. 7A, the preset configuration may be actuated, e.g., via GUI 210 or voice control, etc., to automatically move platform 200 and robot arms 300a, 300b to the preset configuration relative to patient table PT, while avoiding collisions between platform 200 and robot arms 300a, 300b and other objects in the room based on the depth and proximity data observed and generated by optical sensors 202 and proximity sensors 212, and while taking into account the location of one or more trocars at the surgical site to thereby position the distal ends of robot arms 300a, 300b, in an optimal position relative to the trocar(s).

[0126] As platform 200 is being moved toward the patient, the scene may be directly observed by one or more optical scanners 202 and one or more proximity sensors 212, and further graphically displayed to guide movement of platform 200 by the user within the operating room, as described in further detail below with regard to FIGS. 16-17C. From the depth maps observed and generated by optical scanners 202 and the proximity data observed and generated by proximity sensors 212, key features may be identified such as, for example, the height and / or location of patient table PT, the surface of the patient’s abdomen, the position and other characteristics of the surgeon, including the surgeon’s height, the trocar port(s), the distance between the base of robot arms 300a, 300b, e.g., base portions 302a, 302b and shoulder portions 304a, 304b, and other objects in the room such as the patient table PT, robot arms 300a, 300b, one or more surgical instruments coupled with the robot arms, and / or the distance between713732261 v3 46225887-081001 platform 200 and robot arms 300a, 300b and other objects in the room such as the patient table PT. Identification of such key features may be carried out using standard computer vision techniques such as template matching, feature tracking, edge detection, etc.

[0127] As each feature is registered, its position and orientation may be assigned a local coordinate system and transformed into the global co-ordinate system using standard transformation matrices. Once all features are transformed into a single global co-ordinate system, an optimization algorithm, e.g., least squares and gradient descent, may be used to identify the most appropriate vertical and horizontal positions of robot arms 300a, 300b, which may be adjusted via platform 200, as well as the most appropriate position of the distal ends of the robot arms relative to the trocar ports, to maximize the workspace of the robot arms with respect to the insertion point on the patient. The optimal workspace may be dependent on the surgical operation to be performed and / or the surgeon’s preferred position.

[0128] Moreover, the system may process color and / or depth data obtained from optical scanners 202 and proximity sensors 212 to identify objects within the operating room, e.g., the patient bed or the trocar, as well as the planes associated with the identified objects. With knowledge of the location platform 200 and robot arms 300a, 300b relative to the identified objects, including the location of the base of robot arms 300a, 300b, the system may cause the stage assembly to automatically move (or stop movement of) robot arms 300a, 300b to avoid collision with the identified objects during setup, e.g., when robot arms 300a, 300b approaches a predetermined distance threshold relative to the identified objects. In addition, the system may generate and emit, e.g., an audible alert indicative of the proximity of the stages of platform 200 and / or robot arms 300a, 300b relative to the identified objects. For example, the audible alert may change in amplitude and / or frequency as the distance between the stages of platform 200 and / or robot arms 300a, 300b and the identified objects decreases, as perceived by the system based on the depth data. In addition, with knowledge of the location platform 200 and robot arms 300a, 300b relative to the trocar, if the system detects that the position of the patient bed, and accordingly the trocar, is changing, e.g., via adjustment by a user, the system may automatically adjust the arrangement of the robot arms to accommodate the movement of the patient bed and maintain relative position between the distal end of the robot arms and the trocar.713732261 v3 47225887-081001

[0129] Referring now to FIG. 8, components that may be included in co-manipulation robot platform 800 are described. Platform 800 may include one or more processors 802, communication circuitry 804, power supply 806, user interface 808, and / or memory 810. One or more electrical components and / or circuits may perform some of or all the roles of the various components described herein. Although described separately, it is to be appreciated that electrical components need not be separate structural elements. For example, platform 800 and communication circuitry 804 may be embodied in a single chip. In addition, while platform 800 is described as having memory 810, a memory chip(s) may be separately provided.

[0130] Platform 800 may contain memory and / or be coupled, via one or more buses, to read information from, or write information to, memory. Memory 810 may include processor cache, including a multi-level hierarchical cache in which different levels have different capacities and access speeds. The memory also may include random access memory (RAM), other volatile storage devices, or non-volatile storage devices. Memory 810 may be RAM, ROM, Flash, other volatile storage devices or non-volatile storage devices, or other known memory, or some combination thereof, and preferably includes storage in which data may be selectively saved. For example, the storage devices can include, for example, hard drives, optical discs, flash memory, and Zip drives. Programmable instructions may be stored on memory 810 to execute algorithms for, e.g., calculating desired forces to be applied along robot arm 300 and / or the surgical instrument coupled thereto and applying impedances at respective joints of robot arm 300 to effect the desired forces.

[0131] Platform 800 may incorporate processor 802, which may consist of one or more processors and may be a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any suitable combination thereof designed to perform the functions described herein. Platform 800 also may be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Platform 800, in conjunction with firmware / software stored in the memory may execute an operating system (e.g., operating system 850), such as, for example, Windows, Mac OS, QNX, Unix or Solaris 5.10. Platform 800713732261 v3 48225887-081001 also executes software applications stored in the memory. For example, the software may be programs in any suitable programming language known to those skilled in the art, including, for example, C++, PHP, or Java.

[0132] Communication circuitry 804 may include circuitry that allows platform 800 to communicate with image capture device(s) such as optical scanner and / or endoscope. Communication circuitry 804 may be configured for wired and / or wireless communication over a network such as the Internet, a telephone network, a Bluetooth network, and / or a WiFi network using techniques known in the art. Communication circuitry 804 may be a communication chip known in the art such as a Bluetooth chip and / or a WiFi chip. Communication circuitry 804 permits platform 800 to transfer information, such as force measurements on the body wall at the trocar insertion point locally and / or to a remote location such as a server, e.g., a Cloud-based server.

[0133] Power supply 806 may supply alternating current or direct current. Power supply 806 may be a port to allow platform 800 to be plugged into a conventional wall socket, e.g., via a cord with an AC to DC power converter and / or a USB port, for powering components within platform 800. Power supply 806 may be operatively coupled to an emergency switch, such that upon actuation of the emergency switch, power stops being supplied to the components within platform 800 including, for example, the braking mechanism disposed on at least some joints of the plurality of joints of robot arm 300. In direct current embodiments, power supply 806 may include a suitable battery such as a replaceable battery or rechargeable battery and apparatus may include circuitry for charging the rechargeable battery, and a detachable power cord. For example, the battery may be an uninterruptable power supply (UPS) that may be charged when the system is plugged in, and which is only operatively coupled to certain computing components of the system, e.g., processor 802, such that the battery may automatically provide power to the computing components when the system is temporarily unplugged from the electrical power source, e.g., to move the system to another side of a patient table during a multiquadrant procedure.

[0134] User interface 808 may be used to receive inputs from, and / or provide outputs to, a user. For example, user interface 808 may include a touchscreen display (e.g., GUI 210),713732261 v3 49225887-081001 switches, dials, lights, etc. Accordingly, user interface 808 may display information such as selected surgical instrument identity and force measurements observed during operation of robot arm 300. Moreover, user interface 808 may receive user input including actuation of one or more operational modes / sub-modes, and / or adjustments to the predetermined amount of movement at the handle of the surgical instrument or the predetermined dwell time period to cause the robot arm to automatically switch to the passive mode, the predetermined threshold of force applied at the handle of the surgical instrument to cause the robot arm to automatically switch to the co-manipulation mode, a position of the predefined haptic barrier, an identity of the surgical instrument coupled to the distal end of the robot arm, a vertical height of the robot arm, a horizontal position of the robot arm, etc., such that platform 800 may adjust the information / parameters accordingly. In some embodiments, user interface 808 is not present on platform 800, but is instead provided on a remote, external computing device communicatively connected to platform 800 via communication circuitry 804.

[0135] Memory 810, which is one example of a non-transitory computer- readable medium, may be used to store operating system (OS) 850, optical scanner interface module 812, encoder interface module 814, robot arm position determination module 816, trocar position detection module 818, force detection module 820, impedance calculation module 822, motor interface module 824, gesture detection module 826, operational mode determination module 828, phase determination module 830, trajectory generation module 832, surgeon personalization module 340, 3D reconstruction generation module 836, case counting module 838, surgical item retention module 840, report generation module 842, wearables interface module 844, language model module 846, and predictive maintenance module 848. The modules are provided in the form of computer-executable instructions / algorithms that may be executed by processor 802 for performing various operations in accordance with the disclosure.

[0136] For example, during a procedure, the system may continuously run the algorithms described herein based on the data collected by the system. That data may be collected and / or recorded using any of the components and methods disclosed herein, including, e.g., from sensors / encoders within the robots, from optical scanning devices in communication with the other components of the robotic system, and / or from manual inputs by an operator of the system. Accordingly, the algorithms, the data, and the configuration of the system may enable the user to713732261 v3 50225887-081001 co-manipulate the robot arms with minimal impact and influence from the weight of the robot arms and / or surgical instruments coupled thereto, force of gravity, and other forces that traditional robot arms fail to compensate for.

[0137] Optical scanner interface module 812 may be executed by processor 802 for receiving image / depth data obtained by an optical scanning device, e.g., optical scanners 202, and processing the image / depth data to detect, e.g., predefined conditions therein. Moreover, optical scanner interface module 812 may generate depth maps indicative of the received image / depth data, which may be displayed to the operator, e.g., via a monitor, as shown in FIGS. 9A and 9B. FIG. 9A illustrates image data captured by optical scanner 202, and FIG. 9B illustrates a depth map of at least some objects within the surgical space generated from the data captured by optical scanner 202. Specifically, optical scanner interface module 812 may create a depth map, e.g., point clouds, where each pixel’s value is related to the distance from optical scanner 202, as shown in FIG. 9C, which illustrates a depth map of an operating room including, for example, operator O, person Pl, platform 200 and robot arms 300, patient table PT, laparoscopy tower LT, etc. For example, the difference between pixels for a first object (such as a first surgical instrument) and a second object (for example, a trocar) will enable the system to calculate the distance between the surgical instrument and the trocar. Moreover, the difference between pixels for a first object (such as a first surgical instrument) at a first point in time and the first object at a second point in time will enable the system to calculate whether the first object has moved, the trajectory of movement, the speed of movement, and / or other parameters associated with the changing position of the first object.

[0138] As described above, optical scanner 202 further may include a 360 camera, which may include a plurality of RGB cameras configured to collectively capture a 360 degrees field of view about the 360 camera. For example, the 360 camera may include 4 to 8 RGB cameras, or preferably 6 cameras, equally spatially positioned relative to each other to capture a comprehensive 360 field of view about the 360 camera. Accordingly, optical scanner interface module 812 may receive RGB image data from the 360 camera, as shown in FIG. 9D, which illustrates RGB image data of the operating room captured in 360 degrees about the 360 camera. As shown in FIG. 9C (right), the 360 degrees RGB image data may be displayed in a “fisheye”713732261 v3 51225887-081001 view. As will be understood by a person having ordinary skill in the art, the 360 camera may include more or less than 4 to 8 RGB cameras.

[0139] Based on the depth map generated by the optical scanning devices, optical scanner interface module 812 may cluster different groups of (depth) pixels into unique objects, a process which is referred to as object segmentation. Examples of such algorithms for segmentation may include: matching acquired depth map data to a known template of an object to segment; using a combination of depth and RGB color image to identify and isolate relevant pixels for the object; and / or machine learning algorithms trained on a real or synthetic dataset to objects to identify and segment. Examples of such segmentation on a depth map may include: locating the robot arms or determining the position of the robot arms; identifying patient ports (e.g., trocar ports) in 3D space and determining a distance from the instruments to the trocar ports; determining the relative distances between, e.g., the stages of platform 200, robot arm 300, any surgical instruments attached thereto, and objects / persons in the operating room such as the surgical table, drapes, etc.; identifying the surgeon and distinguishing the surgeon from other operators in the room; identifying the surgeon in the sensor’s FOV; determining that the co-manipulation robot system is in an authorization zone, e.g., an operating room, based on the identification of capital equipment in the sensor’s FOV; and / or identifying authorized persons, e.g., a user preauthorized to use the co-manipulation robot system, and optionally, the level of authorization associated with the identified person. Moreover, optical scanner interface module 812 may execute object segmentation algorithms to uniquely identify the surgeon and track the surgeon with respect to, for example, a surgical table, a patient, one or more robot arms, etc. In addition, optical scanner interface module 812 may execute object segmentation algorithms to determine if a surgeon is touching or handling either of the robot arms and, if so, identify which robot arm is being touched or handled by the surgeon.

[0140] For example, optical scanner interface module 812 may measure and record any of the following within the coordinate space of the system: motion of the handheld surgical instruments manipulated by the surgeon (attached to or apart from a robot arm); the presence / absence of other surgical staff (e.g., scrub nurse, circulating nurse, anesthesiologist, etc.); the height and angular orientation of the surgical table; patient position and volume on the surgical table; presence / absence of the drape on the patient; presence / absence of trocar ports, and713732261 v3 52225887-081001 if present, their position and orientation; gestures made by the surgical staff; tasks being performed by the surgical staff; interaction of the surgical staff with the system; surgical instrument identification; attachment or detachment “action” of surgical instruments to the system; position and orientation tracking of specific features of the surgical instruments relative to the system (e.g., camera head, coupler, fiducial marker(s), etc.); measurement of motion profiles or specific features in the scene that allow for the phase of the surgery to be identified; position, orientation, identity, and / or movement of any other instruments, features, and / or components of the system or being used by the surgical team.

[0141] Optical scanner interface module 812 may combine measurements and / or other data described above with any other telemetry data from the system and / or video data from the scope to provide a comprehensive dataset with which to improve the overall usability, functionality, and safety of the co-manipulation robot-assisted surgical systems described herein. For example, as the system is being set up to start a procedure, optical scanner interface module 812 may detect the height and orientation of the surgical table and / or the location of one or more trocars based on the image obtained by optical scanner 202. This information may allow the system to automatically configure the degrees of freedom of platform 200 supporting robot arms 300 to the desired or correct positions relative to the surgical table and / or the one or more trocars. Specifically, based on image data received by optical scanner interface module 812, the system may be used to ensure that the height of platform 200 is optimally positioned to ensure that robot arms 300 overlap with the intended surgical workspace, as indicated by the location of the trocar(s). In addition, the system may automatically reconfigure the degrees of freedom of platform 200 as well as the arrangement of robot arms 300 responsive to movement of the surgical table, and accordingly the trocar(s), and / or responsive to the position of the trocar(s) relative to platform 200, to maintain relative position between the distal end of the robot arms and the trocar(s).

[0142] Optical scanner interface module 812 further may identify the specific surgeon carrying out the procedure, such that the system may use the surgeon’s identity to load a surgeon profile associated with the particular user into the system. The surgeon profile may include information related to a surgeon’s operating parameter and / or preferences, a surgeon’s patient list having parameters for each patient, the desired or required algorithm sensitivity for the713732261 v3 53225887-081001 surgeon, the degree of freedom positioning of the support platform, etc. Examples of algorithm sensitivities that may be surgeon-specific include: adapting / adjusting the force required to transition from passive mode to co-manipulation mode (e.g., from low force to high force), adapting / adjusting the viscosity felt by the surgeon when co-manipulating the robot arm (e.g., from low viscosity to high viscosity), preferred surgical instrument trajectories when performing specific surgical procedures, etc. Moreover, the surgeon’s preferences may include preferred arrangements of robot arm 300, e.g., the positioning of the links and joints of robot arm 300 relative to the patient, with regard to specific surgical instruments, e.g., the preferred arrangement may be different between a scope and a retractor. Accordingly, overriding maneuvers performed by a specific surgeon during a surgical procedure may be recorded and stored within the surgeon’s profile. In addition, video data recorded during a surgical procedure performed by a specific surgeon may be at least partially stored within memory associated with the surgeon’s profile. Loading of a surgeon profile and / or procedure type may be indicative of the occurrence of a surgical procedure.

[0143] Moreover, optical scanner interface module 812 may track the motion of the handheld surgical instruments that are directly and independently controlled by the surgeon, that are not coupled with the robot arm, e.g., via optical scanners 202 and / or the scope’s video feed. For example, optical scanner interface module 812 may track a clearly defined feature of the instrument, a fiducial marker attached to the instrument or to the gloves (e.g., the sterile gloves) of the surgeon, the coupler between the robot arm and the instrument, a distal tip of the instrument, and / or any other defined location on the instrument. The following are examples of uses and purposes of the motion data: (i) closing a control loop between a handheld instrument and the robot arm holding the camera, thus allowing the surgeon to servo (i.e., move) the camera by “pointing” with a handheld instrument; (ii) tracking information that may be used independently or in combination with other data streams to identify the phase of the surgical procedure; (iii) to identify the dominant hand of the surgeon; (iv) to monitor metrics associated with the experience of the surgeon; (v) to identify which tools the surgeon is using and when to change them for other tools; and / or (vi) tracking of the skin surface of the patient, as well as the number, position and orientation of the trocar ports. This data and information also may be used and computed by the system as part of the co-manipulation control paradigm. As will be713732261 v3 54225887-081001 understood by a person having ordinary skill in the art, the location / movement of a surgical instrument coupled to a robot arm of the system will be known by the system based on the known robot telemetry and current kinematics of the robot arm, without the need of data captured by optical scanner 202.

[0144] Optical scanner interface module 812 further may receive image data from additional optical scanning devices as defined herein, including for example, a scope operatively coupled to the system. Accordingly, optical scanner interface module 812 further may use object segmentation algorithms to analyze image data obtained from the scope to locate and track one or more surgical instruments and / or anatomical structures and distinguish the tracked surgical instrument(s) and / or anatomical structure(s) from other objects and structures within the FOV of the scope, as described in U.S. Patent No. 11,844,583. For example, the object segmentation algorithms may include deep learning approaches. Specifically, a neural network may be trained for instrument / anatomical structure detection via a manually annotated video dataset sampled from multiple laparoscopic surgeries includes various surgical instruments and anatomical environments. Labeled training data including manual annotations indicative of surgical instrument / anatomical structure locations as well as class labels indicative of surgical instrument / anatomical structure type may be fed through a feature extractor to train the neural network to generate class labels and identify surgical instrument / anatomical structure location within an image dataset.

[0145] In addition, optical scanner interface module 812 may receive depth data obtained by proximity sensors 212 coupled to platform 200 and process the depth data to generate a virtual map of the area surrounding platform 200, as described in further detail below. Accordingly, the data streams from the robot arms, the camera feed from the scope, the data acquired from optical scanner 202 and / or proximity sensors 212, as well as data optionally captured from one or more imaging devices disposed on a structure adjacent to the robot arms, the walls, ceiling, or other structures within the operating room, may be recorded, stored, and used individually or in combination by optical scanner interface module 812 to understand and control the surgical system and procedures of the surgical system. The foregoing components, devices, and combinations thereof are collectively referred to herein as optical sensors, optical scanners, or optical scanning devices.713732261 v3 55225887-081001

[0146] Optical scanner interface module 812 further may perform surrounding context-based authorization of the co-manipulation robot system as part of the security workflow by determining if the co-manipulation robot system, e.g., platform 200 and robot arms 300, is within an “authorized clinical zone,” e.g., a predefined area such as an operating room or within the vicinity of at least one pre-authorized user, based on image / depth data obtained by one or more optical scanning devices and / or proximity sensors, as described above, and automatically providing a corresponding predefined level of authorization for access to the system, as shown in FIG. 27, thereby bypassing the otherwise mandatory step of a user performing a user action, e.g., password, two-factor authentication, RFID key, biometrics such as fingerprint, facial recognition, etc., before the system is authorized for use and, thus, avoiding interruption of the user’s journey and promoting system use / adoption.

[0147] For example, as shown in FIG. 27, upon powering on of the co-manipulation robot system and upon receipt of video input, e.g., image / depth data, from one or more optical scanners and / or proximity sensors of the system, e.g., optical scanner 202, as described above, optical scanner interface module 812 may execute one or more object segmentation algorithms to identify objects (e.g., capital equipment such as the surgical table, a Mayo stand, a laparoscopic tower, camera booms, etc.), and / or persons (e.g., the surgeon, surgical staff, field service engineer, etc.) within the obtained image / depth data, e.g., via image classification and / or object detection to thereby balance accuracy, speed, and robustness across varied operating rooms, depth estimation to thereby prevent spoofing, and / or temporal analysis to thereby improve the reliability of detection, to thereby determine if the co-manipulation robot system is within an authorized clinical zone. For example, upon detection / identifi cation of one or more predetermined objects, e.g., capital equipment, within the FOV of the optical scanning devices and / or proximity sensors of the co-manipulation robot system, optical scanner interface module 812 may determine that the co-manipulation robot system is within a predefined authorized clinical zone, e.g., an operating room, and automatically grant a predefined level of authorization for access to the system by any user, e.g., via GUI 210.

[0148] Moreover, upon detection / identification of one or more persons, e.g., the surgeon, a member of the surgical staff, a field service engineer, etc., within the FOV of the optical scanning devices and / or proximity sensors of the co-manipulation robot system, optical scanner713732261 v3 56225887-081001 interface module 812 may determine that the co-manipulation robot system is within an “authorized clinical zone,” e.g., within the proximity of a pre-authorized user, and automatically grant a predefined level of authorization associated with the identified pre-authorized user for access to the system. For example, upon detection of a pre-authorized user, optical scanner interface module 812 may automatically load the user profile associated with the pre-authorized user and provide a level of access to the system corresponding to the authorization level stored in the user profile. As will be understood by a person having ordinary skill in the art, the predefined levels of authorization may range between limited access, e.g., for access to the backend of the system for performing different levels of troubleshooting, for maintenance, or for operational use of the co-manipulation robot system for performing a surgical procedure, etc., to full access. As shown in FIG. 27, if optical scanner interface module 812 does not determine that the co-manipulation robot system is within an authorized clinical zone, access to the system may be restricted / disabled and, optionally, cause GUI 210 to display an error message.

[0149] Encoder interface module 814 may be executed by processor 802 for receiving and processing angulation measurement data from the plurality of encoders of robot arm 300, e.g., encoders E1-E7, in real-time. For example, encoder interface module 814 may calculate the change in angulation over time of the links of robot arm 300 rotatably coupled to a given joint associated with the encoder. As described above, the system may include redundant encoders at each joint of robot arm 300, to thereby ensure safe operation of robot arm 300. Moreover, additional encoders may be disposed on platform 200 to measure angulation / position of each robot arm relative to platform 200, e.g., the vertical and horizontal position of the robot arms relative to platform 200. Accordingly, an encoder may be disposed on platform 200 to measure movement of the robot arms along the vertical axis of platform 200 and another encoder may be disposed on platform 200 to measure movement of the robot arms along the horizontal axis of platform 200.

[0150] Robot arm position determination module 816 may be executed by processor 802 for determining the position of robot arm 300 and the surgical instrument attached thereto, if any, in 3D space in real-time based on the angulation measurement data generated by encoder interface module 814. For example, robot arm position determination module 816 may determine the position of various links and joints of robot arm 300 as well as positions along the surgical713732261 v3 57225887-081001 instrument coupled to robot arm 300. Based on the position data of robot arm 300 and / or the surgical instrument, robot arm position determination module 816 may calculate the velocity and / or acceleration of movement of robot arm 300 and the surgical instrument attached thereto in real-time. For example, by determining the individual velocities of various joints of robot arm 300, e.g., via the encoders associated with each joint of the various joints, robot arm position determination module 816 may determine the resultant velocity of the distal end of robot arm 300, which may be used by a passive mode determination module of operational mode determination module 828 to determine whether movement of the distal end of robot arm 300 is within a predetermined threshold for purposes of transitioning system 100 to passive mode, as described in further detail below. Moreover, robot arm position determination module 816 may record the various positions of the robot arms along with associated timestamps throughout a surgical procedure (including setup), which may be used in 3D reconstruction generation for training purposes, as described in further detail below.

[0151] Trocar position detection module 818 may be executed by processor 802 for determining the position and / or orientation of one or more trocar ports inserted within the patient. The position and / or orientation of a trocar port may be derived based on data obtained from, e.g., inertial measurement units and / or accelerometers, optical scanners, electromechanical tracking instruments, linear encoders, the sensors, and data as described above. For example, the position of the trocar ports on the patient may be determined using a laser pointing system that may be mounted on one or more of the components of the system, e.g., wrist portion 311 of the robot arm, and may be controlled by the system to point to the optimal or determined position on the patient’s body to insert the trocar. Moreover, upon insertion of the surgical instrument that is attached to robot arm 300 through a trocar, virtual lines may continuously be established along the longitudinal axis of the surgical instrument, the alignment / orientation of which may be automatically determined upon attachment of the surgical instrument to coupler interface 400 via the coupler body via the magnetic connection as described above, in real-time as the surgical instrument moves about the trocar point. For example, when the surgical instrument is inserted within the trocar port, it will be pointing toward the trocar point, and accordingly, distal wrist link 316 will also point toward the trocar point, the angle of which may be measured by an encoder associated therewith. Accordingly, the trocar point may be calculated as the intersection713732261 v3 58225887-081001 of the plurality of virtual lines continuously established along the longitudinal axis of the surgical instrument. In this manner, the calculated trocar point will remain fixed relative to the patient as the surgical instrument is maneuvered about the trocar port, e.g., rotated or moved in or out of the patient.

[0152] Moreover, knowledge of the location of one or more trocar ports within a surgical site provides a comprehensive surgical scene understanding that enhances the efficacy of the system. For example, fine-tuning the poses / configuration of the robot arms during initial setup is one of the most pivotal stages in a robot-assisted surgery, and proper alignment ensures minimized risk to the patient and reduces the necessity for major adjustments during the operation. Thus, by leveraging the capabilities of 3D surgical site and trocar detection, the system may automatically configure the robot arms, optimizing them for specific surgical procedures. For example, based on the image data captured by optical scanner interface module 812, trocar position detection module 818 further may identify and track the location of one or more trocar ports inserted within the patient, e.g., to optimize the configuration of the robot arms relative to the trocar ports, which instrument is being used in a respective port, how often instruments are swapped between ports, which ports have manually held instruments versus instruments coupled to the robot arm, to monitor and determine if additional trocar ports are added, if the system is holding the instruments in place while the patient or surgical table is moving (in which case, the system may change the operational mode of the robot arms to a passive mode and accommodate the movement by repositioning robot arm 300 and / or platform 200), and / or other conditions or parameters of the operating room or the system. For example, the orientation of the trocar port and / or its position relative to robot arm 300 may be determined based on image data received from one or more optical scanners, e.g., optical scanners 200 and / or video feed obtained by a scope. By measuring the true position and orientation of the trocar ports, the system may be provided an additional safety check to ensure that the system level computations are correct, e.g., to ensure that the actual motion of the robot arms or instrument matches a commanded motion of the robot arms or instrument in robotic assist mode. Accordingly, trocar position detection module 818 may automatically adjust the configuration of a robot arm relative to a specific trocar port, e.g., during system setup based on the surgical procedure to be performed, based on the known position / orientation of one or more other trocar ports, as well as the known713732261 v3 59225887-081001 configuration of the other robot arm, e.g., based on the known robot telemetry and current kinematics of the robot arms.

[0153] Trocar position detection module 818 may acquire and process depth data received by optical scanner interface module 812 to identify a trocar port for setting up the robot arm for a procedure, e.g., move the robot arms to a desired vicinity of the patient / trocar port. For example, trocar position detection module 818 may perform specular noise filtering, object segmentation to segment and identify the patient / trocar ports, and transform tracked trocar port coordinates to the robot coordinate space. For example, based on depth data received by optical scanner interface module 812, trocar position detection module 818 may first identify the patient bed as a reference point, and then execute one or more algorithms to delineate the surgical site, focusing predominantly on the patient’s abdomen. To enhance precision, trocar position detection module 818 may deploy a deep learning model trained on vast datasets, enabling it to recognize and differentiate trocars from other elements within the surgical scene. Alongside this, regions may be filtered based on their distinct 3D geometrical properties and color contrasts relative to the patient’s skin. Through processing these 3D point clouds, trocar position detection module 818 may accurately identify the trocar incision points, and formulate a comprehensive surgical site descriptor that captures essential metrics such as the surgical site boundaries, which are crucial for optimal robot arm positioning. In addition, post identification of the trocar sites and the surgical site descriptor metrics, trocar position detection module 818 may cross-reference this data with predefined surgical procedure knowledge, e.g., to discern which trocar locations will accommodate the robot’s instruments. Trocar position detection module 818 further may calculate 3D positions for the robot arm’s terminal joints, strategically offset from the trocar’s 3D coordinates to maintain safe surgical distances. The positioning of the remaining joints may be resolved using inverse kinematics and further fine-tuned based on preset poses specific to the intended surgical procedure.

[0154] For example, the system may prompt the user with an audio cue to remove the scope from the patient’s body and capture a top-view recording, e.g., via a scope. By utilizing computer vision, the system will be able to identify whether the scope is outside of the abdomen and has a stable view of the surgical site. If so, the audio prompt to the user may be skipped and the system may trigger image capture by the scope automatically. This image data may be713732261 v3 60225887-081001 received by optical scanner interface module 812, and stored in memory associated with the surgeon’s profile, along with the rest of the case data. Trocar position detection module 818 may then perform analysis on collected image data as described above to identify trocar port placements on the patient’s abdomen, as well as surgeon position, bed angle, and other scene metadata. The data for the same type of procedures may be accumulated for each surgeon profile, which may be used to determine the surgeon’s preferred port placement for a given procedure, e.g., based on an average center of the previous port placements. Accordingly, the surgeon’s preferred port placements may be used to inform the system of the optimal robot arm configuration for a specific surgeon performing a specific surgical procedure. Moreover, trocar position detection module 818 may generate a trocar heat map of the surgical site from this data to show the variations of trocar port placements for each surgeon for various surgical procedures over a time period, as shown in FIG. 10. Accordingly, surgeons may have access to a diagram displaying the area of port positioning for each type of procedure, as well as additional information such as the surgeon’s standing position and / or the patient bed angle.

[0155] Accordingly, the automated optimal robot arm setup, based on precise surgical site and trocar detection, markedly enhances patient safety by minimizing risks associated with manual calibrations, and / or by maximizing the workspace of the associated robot arm relative to the surgical site, e.g., to thereby reduce / prevent collision with the robot arms. This system promotes efficiency, reducing the time traditionally needed for manual adjustments, and ensures consistency in robot-assisted surgeries. Further, its adaptability across various surgical scenarios underscores its versatility, streamlining the surgical workflow and potentially reducing overall procedural costs. As an added benefit during the procedure, trocar positions computed via computer vision also may be used as a redundant check for trocar positions computed from the instrument and robot arm poses computed via the robot arms’ kinematics and telemetry. In addition, the trocar positions also may be used to “memorize” trocar positions when instruments are momentarily taken out, e.g., when cleaning the scope, in which case trocar position computation is not possible using the robot arms’ kinematics.

[0156] Force detection module 820 may be executed by processor 802 for detecting forces applied on robot arm 300, e.g., at the joints or links of robot arm 300 or along the surgical instrument, as well as applied on the trocar, e.g., body wall forces. For example, force detection713732261 v3 61225887-081001 module 820 may receive motor current measurements in real-time at each motor, e.g., Ml, M2, M3, disposed within the base of robot arm 300, which are each operatively coupled to a joint of robot arm 300, e.g., base joint 303, shoulder joint 318, elbow joint 322, wrist joint 332. The motor current measurements are indicative of the amount of force applied to the associated joint. Accordingly, the force applied to each joint of robot arm 300 as well as to the surgical instrument attached thereto may be calculated based on the motor current measurements and the position data generated by robot arm position determination module 816 and / or trocar position detection module 818. The calculated forces may be recorded and used by the system to determine and apply a predetermined amount of impedance to the robot arm to thereby reduce the force applied to the patient’s body by the surgical instrument, e.g., during trocar placement to prevent bowel and vascular injuries.

[0157] FIGS. 11 A and 1 IB illustrate exemplary force measurements of the system during operation of robot arm 300. As described above, the calibration file of the surgical instrument coupled to robot arm 300 loaded on the system may include information of the surgical instrument including, e.g., the mass of the surgical instrument, the center of mass of the surgical instrument, and the length of the surgical instrument, such that distance D3 between the center of mass and the instrument tip may be derived. In addition, as described above, the position of the surgical instrument at the trocar, e.g., where the surgical instrument enters the patient’s body, may be calculated in real-time, such that distance D2 between the center of mass of the surgical instrument and the trocar may be derived in real-time. Additionally, as described above, the coupler body is preferably coupled to the surgical instrument at a fixed, known position along the elongated shaft of the surgical instrument (which may be included in the calibration file), e.g., adjacent to the proximal portion of the surgical instrument, and thus distance DI between the center of mass of the surgical instrument and the coupler body, e.g., the point of attachment to the distal end of robot arm 300, may be derived. Alternatively, or additionally, as described above, optical scanning devices may be used to determine any one of DI, D2, or D3.

[0158] As shown in FIG. 11 A, when the surgical instrument is positioned through trocar Tr, without any additional external forces acting on the surgical instrument other than at trocar Tr, e.g., the surgical instrument is not lifting or retracting tissue within the patient, the force applied713732261 v3 62225887-081001 to the surgical instrument at trocar Tr by the body wall (e.g., the “body wall force” or the “trocar force”) may be calculated with the following equation:Where Feff is the force at the distal end of robot arm 300 (e.g., the “end-effector force” of robot arm 300), W is the weight vector of the surgical instrument (=-mgz), and Ftr is the trocar force. Accordingly, Feff is the desired force sent to the system, which is the sum of all the forces generated in the algorithm pipeline including, e.g., gravity compensation, hold, etc.

[0159] As shown in FIG. 1 IB, when the surgical instrument is positioned through trocar Tr and holding / retracting tissue, such that an external force is applied to the tip of the surgical instrument, there are two forces to resolve: Ftr and Ftt. Accordingly, two equations are needed to solve for the two unknown vectors, which may be the balances of forces and also the balance of moments around the center of mass of the surgical instrument, e.g., Lcg.Feff D T Ftrx £)2 T F^ x D3=0

[0160] Here, distances DI and D3 are known as described above, and D2 may be derived based on the known position of the distal end of robot arm 300 and the calculated position of trocar Tr. As shown in FIG. 1 IB, the center of mass Lcgof the surgical instrument is behind the point of attachment of the coupler body to the distal end of robot arm 300.

[0161] In addition, force detection module 820 may calculate the amount of force applied by the surgical instrument to the patient at the point of entry, e.g., at the trocar, as well as the amount of force applied to the operating end of the surgical instrument, e.g., the grasper end of a surgical instrument, based on the compensation force, the hold force, one or more parameters of the surgical instrument such as the mass, center of mass, and length of the surgical instrument, and the distance from the center of mass to the point of entry. As described above, the system may alert the operator if the forces, e.g., force Ftt applied to the tip of the instrument and / or force Ftr applied by the instrument at trocar Tr, are greater than the respective threshold forces, and accordingly freeze the system if the calculated force is greater than the threshold force, and / or713732261 v3 63225887-081001 reduce the force exerted at the trocar point at the body wall or at the tip of the instrument by automatically applying brakes or stopping forces to robot arm 300, by slowing or impeding further movement of the instrument in the direction that would increase forces applied at the tip of the instrument or the trocar, and / or automatically moving the robotic arm in a direction that reduces the force being exerted at the instrument tip and / or at the trocar point at the body wall. Accordingly, with knowledge of, e.g., the position of the trocar, the distance from the coupler body to the instrument tip, the current position of the distal end of the robot arm, the current position of the instrument tip, the desired position of the distal end of the robot arm that provides the desired position of the instrument tip, force detection module 820 and / or impedance calculation module 822 described in further detail below may calculate the force required to apply to the distal end of the robot arm to move it from its current position to its desired position to thereby move the instrument tip from its current position to its desired position.

[0162] Due to the passive axes at the distal end of robot arm 300, the force applied by the instrument coupled with the robot arm on the trocar may remain generally consistent throughout the workspace of the robot arm. The force on the trocar may be affected by the interaction of the distal tip of the instrument with tissue within the body. For example, if a tissue retractor advanced through the trocar is engaged with (e.g., grasping) bodily tissue or another object inside the body, the force exerted on the end of the instrument from the bodily tissue or other object may cause a change in the force applied to the trocar. In some aspects, the force on the trocar may be a function of how much weight is being lifted by the instrument being used. In addition, the forces applied to the trocar and / or other bodily tissues, e.g., during trocar placement as well as during the procedure, may be recorded along with associated timestamps for use in generating operation summary reports for a specific surgeon to inform force reduction in subsequent procedures, and / or for generating an alert to inform force reduction in real-time.

[0163] For example, force detection module 820 further may record the calculated forces applied to the system, and accordingly, the forces applied to the patient’s body, e.g., at the trocar insertion site, by the surgical instrument coupled to the robot arm, at various times throughout the procedure as well as the associated timestamps. Accordingly, the data generated by force detection module 820 may be used by report generation module 842 to generate a report for the surgeon with information including the force profile of the surgical instrument throughout a713732261 v3 64225887-081001 surgical procedure, as described in further detail below. Force detection module 820 may execute one or more image segmentation algorithms with image data received / generated by optical scanner interface module 812, e.g., via a scope coupled to a first robot arm of the system, as an input to classify soft tissues within the image data in which forces are / will be applied to by the surgical instrument coupled to a second robot arm of the system. For example, force detection module 820 may classify adipose tissue, different visceral organs, ligaments, fascia, fibrous bands of scar tissue, etc.

[0164] In addition, force detection module 820 may generate a graphical representation of the amount of force being applied to the patient’s body by the surgical instrument coupled to the robot arm in real-time during a procedure, as shown in FIG. 12, such that the surgeon may be able to understand how much force is being applied by the surgeon via the surgical instrument in real-time. Based on information from pre-clinical studies and data collected from skilled surgeons during past clinical procedure using the co-manipulation robot system described herein, e.g., stored in a database accessible by the system, force detection module 820 may establish safety force thresholds based on, for example, the tissue classification discussed above, the type of instrument (e.g., hook, grasper, retractor, etc.) which may be automatically identified by the system as described above or manually inputted by the user, and / or the type of instrument manipulation performed by the user (e.g., pulling, pushing, suturing, etc.). Accordingly, different soft tissues may have different associated safety force thresholds, such that, upon determination by force detection module 820 that a force being applied to a given soft tissue by a known surgical instrument during a specific instrument manipulation reaches and / or exceeds the associated predetermined safety force threshold for that given soft tissue / surgical instrument / instrument manipulation, the system may be programmed to perform one or more actions to thereby alert the user and / or prevent injury to the patient. For example, force detection module 820 may generate a graphical representation of an indicator, e.g., indicator 1200, that informs the user of the amount of force being applied to the bodily tissue by the surgical instrument, which may be overlaid on the scope video feed, as shown in FIG. 12, and / or displayed on a separate monitor. Graphical indicator 1200 may illuminate in a variety of colors indicative of the amount of force applied, e.g., green for low force, yellow for medium force, and red for high force, to thereby inform the user regarding whether to reduce the amount of force713732261 v3 65225887-081001 being applied by the user. With knowledge of the force profile during a surgical procedure in real-time, the surgeon may observe when excessive force is being applied to the patient’s body, e.g., at the trocar incision sites, and reduce such forces to thereby reduce the risk of and / or prevent postoperative hernia due to excessive body wall forces.

[0165] Additionally, or alternatively, upon determination by force detection module 820 that the force applied by the user reaches / exceeds the associated predetermined safety force threshold, the system may generate / provide audio and / or haptic feedback. For example, the system may apply impedance to the robot arm coupled to the surgical instrument operated by the user (e.g., via motor interface module 824) in a manner that causes a slight vibration perceivable by the user, but not strong enough to disrupt the operation. Moreover, the system may apply increased impedance to the robot arm coupled to the surgical instrument to thereby increase the resistance (e.g., viscosity) of the robot arm to the movement applied to the surgical instrument, and accordingly the bodily tissue, by the user, which may reduce the velocity and travel of the surgical instrument. In some embodiments, the system may apply a sufficient amount of impedance to the robot arm to momentarily freeze / stop movement of the robot arm, and accordingly the surgical instrument coupled thereto, relative to the bodily tissue. In addition, as accessing the abdominal wall through a trocar requires high force, which may risk injury adjacent organs, and resistance to the surgical instrument abruptly decreases and / or is eliminated once the surgical instrument crosses the abdominal wall, the system may apply increased impedance to the robot arm once force detection module 820 detects that force applied to the surgical instrument by the abdominal wall abruptly decreases to thereby increase the resistance (e.g., viscosity) of the robot arm and prevent uncontrolled motion of the surgical instrument by the user beyond the abdominal wall.

[0166] Impedance calculation module 822 may be executed by processor 802 for determining the amount of impedance / torque needed to be applied to respective joints of robot arm 300 to achieve the desired effect, e.g., holding robot arm 300 in a static position in the passive mode, permitting robot arm 300 to move freely while compensating for gravity of robot arm 300 and the surgical instrument attached thereto in the co-manipulation mode, applying increased impedance to robot arm 300 when robot arm 300 and / or the surgical instrument attached thereto is within a predefined virtual haptic barrier in the haptic mode, etc. For713732261 v3 66225887-081001 example, impedance calculation module 822 may determine the amount of force required by robot arm 300 to achieve the desired effect based on position data of robot arm 300 generated by robot arm position determination module 818 and the position data of the trocar generated by trocar position detection module 820. By determining the forces applied on robot arm 300 via force detection module 822, as well as the position / velocity / acceleration of the distal end of robot arm 300 in 3D space via robot arm position determination module 818, the desired force / impedance to be applied to robot arm 300 to compensate for the applied forces may be calculated, e.g., for gravity compensation or to hold robot arm 300 in a static position in the passive mode.

[0167] For example, by determining the position of the distal end of robot arm 300, as well as the point of entry of the surgical instrument into the patient, e.g., the trocar position, and with knowledge of one or more instrument parameters, e.g., mass and center of mass of the surgical instrument stored in memory 810, impedance calculation module 822 may calculate the amount of force required to compensate for gravity of the surgical instrument (compensation force), as described above with regard to FIG. 11 A. Accordingly, the amount of compensation force required to compensate for the gravity of the surgical instrument may be converted to torque to be applied at the joints of robot arm 300. For example, the robot Jacobian may be used for this purpose.

[0168] Moreover, by determining the position of the distal end of robot arm 300, and accordingly, a change in position of the distal end of robot arm 300 over time, for example, due to an external force applied to the distal end of robot arm 300, e.g., by tissue held by the operating end of the surgical instrument, and with knowledge of one or more instrument parameters, e.g., mass, center of mass, and length of the surgical instrument, impedance calculation module 822 may calculate the amount of force required to maintain the surgical instrument in a static position (hold force), as described above with regard to FIG. 1 IB. Accordingly, the amount of hold force required to resist the change in position of the distal end of robot arm 300, in addition to the amount of compensation force required to compensate for the gravity of the surgical instrument, may be converted to torque to be applied at the joints of robot arm 300 to maintain robot arm 300 in a static position.713732261 v3 67225887-081001

[0169] Motor interface module 824 may be executed by processor 802 for receiving motor current readings at each motor, e.g., Ml, M2, M3, M4, disposed within the base of robot arm 300, and for actuating the respective motors, e.g., by applying a predetermined impedance / torque to achieved the desired outcome as described herein and / or to cause the joints operatively coupled to the respective motors to move, such as in the robotic assist mode. For example, motor interface module 824 may actuate M4 to cause rotation of distal shoulder link 308 relative to proximal shoulder link 306. Additionally, motor interface module 824 may actuate the one or more motors operatively coupled to the stage assembly of platform 200 to cause the stage assembly to move the robot arms relative to platform 200, e.g., during user guide setup mode, as described above. As described above, motor interface module 824 may keep track of each time the one or more motors are actuated, which may be indicative of the occurrence of a surgical procedure, and further may be used to improve efficiency of a subsequent similar surgical procedure (or setup) by the surgeon, e.g., by reducing the amount of times the one or more motors are actuated before / during a procedure.

[0170] Gesture detection module 826 may be executed by processor 802 for detecting predefined gestural patterns as user input, and executing an action associated with the user input. The predefined gestural patterns may include, for example, movement of a surgical instrument (whether or not attached to robot arm 300), movement of robot arm 300 or other components of the system, e.g., foot pedal, buttons, etc., and / or movement of the operator in a predefined pattern. For example, movement of the surgical instrument back and forth in a first direction (e.g.., left / right, up / down, forward / backward, in a circle) may be associated with a first user input requiring a first action by the system and / or back and forth in a second direction (e.g.., left / right, up / down, forward / backward, in a circle) that is different than the first direction may be associated with a second user input requiring a second action by the system. Similarly, pressing the foot pedal or a button operatively coupled with the system in a predefined manner may be associated with a third user input requiring a third action by the system, and movement of the operator’s head back and forth or up and down repeatedly may be associated with a fourth user input requiring a fourth action by the system. Moreover, forces applied to the robot arm and / or the surgical instrument coupled thereto such a double or triple tap by the user also may be associated with user input requiring a predefined action by the system, e.g., initiation of713732261 v3 68225887-081001 instrument centering mode to track an instrument in the FOV of the scope, as described in further detail below. Various predefined gestural patterns associated with different components or operators of the system may be redundant such that the associated user input may be the same for different gestural patterns. The predefined gestural patterns may be detected by, e.g., an optical scanning device such as a scope or optical scanner 202 via optical scanner interface module 812 or directly by force applied to robot arm 300 via force detection module 822 or other components of the system.

[0171] Actions responsive to user input associated with predefined gestural patterns may include, for example, enabling tool tracking to servo (i.e., move) the scope based on the motion of a handheld tool and / or automatically to maintain the handheld tool within a FOV of the scope; engaging the brakes on (e.g., preventing further movement of) the robot arm; engaging a software lock on the robot arm; dynamically changing the length of time that the robot arm takes to transition between states from a default setting; loading a virtual menu overlay on the video feed whereby a surgical instrument in the FOV of the scope functions as a pointer to trigger further actions available from the virtual menu; start / stop a recording of image data; and / or identifying which member of the surgical staff is touching the robot arm, if any. This information may be used to ensure that the system does not move if the surgeon is not touching the robot arm, e.g., to avoid the scenario where an external force is acting on the robot arm (e.g., a light cable or other wire being pulled across the robot arm) and the system perceives the force to be intentional from the surgeon. The same information may be used to detect the gaze direction of the surgeon, e.g., whether the surgeon is looking at the video feed or somewhere else in the room, such that the system may freeze the robot arm if the surgeon’s gaze is not in the direction it should be. Additionally, the system may reposition a FOV of a camera based on, for example, the direction a surgeon is facing or based on the objects that the surgeon appears to be looking at, based on the data from the optical scanner 202. Moreover, moving the distal tip of a surgical instrument to a center portion of the scope’s FOV, e.g., defined by a predetermined boundary region, and holding the position for more than a predetermined time threshold may be associated with a user input detected by gesture detection module 826 to enable tool tracking, as described in further detail below.713732261 v3 69225887-081001

[0172] Moreover, a predefined gestural pattern such as double tapping a distal portion of the robot arm or a surgical instrument coupled thereto, e.g., a scope, and / or a predetermined sinusoidal movement of the camera head of the scope about the trocar may be associated with a user input detected by gesture detection module 826 to start and / or stop a recording of image / audio data by the optical scanning devices. Specifically, there may be key moments during a procedure that the user may want recorded, and which the user may want to be able to locate in a quick manner without having to go through an entire recording of the entire procedure to find the key moments. By providing an easy way for the user to initiate and stop a recording via simple predefined gestural patterns, such that the recording is saved to a folder with a timestamp associated with that particular procedure, the user may easily locate the recording for review and / or teaching purposes. This feature may be particularly useful for diagnostic procedures. In some embodiments, in response to detection of the predefined gestural pattern by gesture detection module 826, the system may record and save a predetermined portion of the image data, e.g., ten seconds before and ten seconds after the predefined gestural pattern is detected. Moreover, the select recordings of key moments by the user may be used by the system to indicate key phase segmentation for a given procedure. The user further may generate case notes via the recordings, e.g., by indicating progression through different phases of a procedure when performing a procedure based on a template of the procedure accessible via the system, which may be uploaded to the hospitals’ Electronic Health Record (“EHR”) system and / or Electronic Medical Record (“EMR”) system.

[0173] As described above, responsive to detection of a predefined gestural pattern by the user, e.g., a predefined pattern of movement of the distal tip of the surgical instrument within the FOV of the scope, gesture detection module 826 may cause a virtual menu to overlay on the video feed, such that the surgical instrument within the FOV of the scope functions as a pointer, as shown in FIG. 13. Moreover, gesture detection module 826 may detect further predefined patterns of movement of the distal end of the surgical instrument, e.g., two quick movements in the same direction or a circular movement over a select area of the virtual menu, which may be interpreted as a selection actuation, e.g., a click on the virtual menu. For example, as shown in FIG. 13, the virtual menu overlay on the video feed may include menu options in the corners of the video feed, e.g., “hot corners”, such as: turning on / off instrument centering mode where the713732261 v3 70225887-081001 system automatically moves the robot arm coupled to a scope to follow the surgical instrument and / or zoom in or out to change the FOV of the scope and maintain the target instrument within a predetermined reference distance from the tip of the scope; adjusting the holding force of robot arm coupled to a retractor, e.g., the amount of force that may be applied to the distal tip of the surgical instrument before the system transitions from passive mode to co-manipulation mode; turning on / off audio; and turning on / off haptic feedback. As will be understood by a person having ordinary skill in the art, more or less menu options may be provided via the virtual menu.

[0174] In some embodiments, initiation of the display of the virtual menu overlay on the video feed may be triggered by, e.g., actuation of an external actuator such as foot pedal 213, a predefined pattern of force applied to the robot arm such double tapping wrist portion 311 and / or the surgical instrument coupled to the robot arm as detected by encoders at the distal end of the robot arm, voice activation, wireless buttons, hot buttons, etc. In some embodiments, the operator may actively switch the system to a command mode, e.g., via user interface 808, where particular movements or gestures of the robot arm, surgical instrument, operator, or otherwise as described herein are monitored by gesture detection module 826 to determine if they are consistent with a predefined gestural pattern associated with a predefined user input.Accordingly, the user may initiate various operational modes / functions of the co-manipulation robot system without having to step away from the surgical site, thereby making the overall surgical procedure more seamless by reducing or eliminating interruptions.

[0175] Operational mode determination module 828 may be executed by processor 802 for analyzing the operating characteristics of robot arm 300 to determine whether to switch the operational mode of robot arm 300 to any of the operational modes described herein, e.g., passive mode, co-manipulation mode, robotic assist mode, and haptic mode, as well as any submodes of the operational modes, e.g., instrument centering mode, etc. For example, operational mode determination module 828 may include a passive mode determination module that may be executed by processor 802 for analyzing the operating characteristics of robot arm 300 to determine whether to switch the operational mode of robot arm 300 to the passive mode where the system applies impedance to the joints of robot arm 300 via motor interface module 824 in an amount sufficient to maintain robot arm 300, and accordingly a surgical instrument attached thereto, if any, in a static position, thereby compensating for mass of robot arm 300 and the713732261 v3 71225887-081001 surgical instrument, and any other external forces acting of robot arm 300 and / or the surgical instrument. If robot arm 300 is moved slightly while in the passive mode, but not with enough force to switch out of the passive mode, the system may adjust the amount of impedance applied the robot arm 300 to maintain the static position and continue this process until robot arm 300 is held in a static position. For example, the passive mode determination module may determine to switch the operational mode of robot arm 300 to the passive mode if movement of the robot arm due to movement at the handle of the surgical instrument as determined by force detection module 822 is less than a predetermined amount, e.g., no more than 1 to 5 mm, for at least a predetermined dwell time period associated with robot arm 300. The predetermined dwell time period refers to the length of time that robot arm 300 and / or the surgical instrument attached thereto, if any, are held in a static position. For example, the predetermined dwell time may range between, e.g., 0.1 to 3 seconds or more, and may be adjusted by the operator.

[0176] In some embodiments, the passive mode determination module may determine to switch the operational mode of robot arm 300 to the passive mode if movement of the distal end of the robot arm due to movement at the handle of the surgical instrument as determined by force detection module 822 has a velocity that is less than a predetermined dwell velocity / speed. For example, if the passive mode determination module determines that the distal end of the robot arm 300 and / or the surgical instrument attached thereto, if any, moves at a speed that is lower than the predetermined dwell speed during an entire predetermined dwell period, then operational mode determination module 828 may switch the operational mode of robot arm 300 to the passive mode.

[0177] Operational mode determination module 828 further may include a co-manipulation mode determination module that may be executed by processor 802 for analyzing the operating characteristics of robot arm 300 to determine whether to switch the operational mode of robot arm 300 to the co-manipulation mode where robot arm 300 is permitted to be freely moveable responsive to movement at the handle of the surgical instrument for performing a surgery using the surgical instrument, while the system applies an impedance to robot arm 300 via motor interface module 824 in an amount sufficient to account for mass of the surgical instrument and robot arm 300, e.g., gravity compensation. Moreover, the impedance applied to robot arm 300 may provide a predetermined level of viscosity perceivable by the operator. The co-713732261 v3 72225887-081001 manipulation mode determination module may determine to switch the operational mode of robot arm 300 to the co-manipulation mode if force applied at robot arm 300 due to force applied at the handle of the surgical instrument exceeds a predetermined threshold associated with robot arm 300 (e.g., a “breakaway force”). The predefined force threshold may be dependent on the type of surgical instrument that is being used and / or whether there is an external force being applied to the surgical instrument.

[0178] In some embodiments, the predefined force threshold may be increased if a force is exerted on the surgical instrument by tissue or an organ or otherwise, depending on the direction of the breakaway force. For example, if the breakaway force is in the same direction as the force exerted on the surgical instrument from the tissue or organ, the predefined force threshold may be increased by an amount equal to or commensurate with the force exerted on the surgical instrument from the tissue or organ, as described in U.S. Patent No. 11,839,442 to Linard and WO 2024 / 150115 to Basafa, the entire contents of each of which are incorporated herein by reference. In some embodiments, the user may actuate a “high force mode,” e.g., via user interface 808, where the predefined force threshold is increased to accommodate for engaging with heavier tissue or organs. For example, the predefined force threshold may be selectively increased by 20-100% or more, and further may be stored in a surgeon profile associated with the user.

[0179] Operational mode determination module 828 further may include a haptic mode determination module that may be executed by processor 802 for analyzing the operating characteristics of robot arm 300 to determine whether to switch the operational mode of robot arm 300 to the haptic mode where the system applies an impedance to robot arm 300 via motor interface module 824 in an amount higher than applied in the co-manipulation mode, thereby making movement of robot arm 300 responsive to movement at the handle of the surgical instrument more viscous in the co-manipulation mode. For example, the haptic mode determination module may determine to switch the operational mode of robot arm 300 to the haptic mode if at least a portion of robot arm 300 and / or the surgical instrument attached thereto is within a predefined virtual haptic boundary. Specifically, a virtual haptic boundary may be established by the system, such that the robot arm or the surgical instrument coupled thereto should not breach the boundary. For example, a virtual boundary may be established at the713732261 v3 73225887-081001 surface of the patient to prevent any portion of the robot arms or the instruments supported by the robot arms from contacting the patient, except through the one or more trocars. Similarly, the virtual haptic boundary may include a haptic funnel to help guide the instrument into the patient as the operator inserts the instrument into a trocar port. Accordingly, the viscosity of robot arm 300 observed by the operator will be much higher than in co-manipulation mode. In some embodiments, the haptic mode determination module may determine to switch the operational mode of robot arm 300 to the haptic mode based on the identity of the surgical instrument.

[0180] Moreover, a virtual haptic boundary, e.g., haptic shell, may be established at a predetermined distance surrounding the workspace to prevent over-extension of the robot arm away from the operation site, as well as to minimize “runaway” of the robot arm. For example, the haptic mode determination module may generate temporary localized virtual haptic boundaries at the distal ends of the robot arms during predetermined phases of a procedure / clinical workflow to prevent “runaway” and further enhance safety of the system. The predetermined phases may include, e.g., during draping / tear-down of the drape, immediately after tool removal is detected, and / or immediately after coupler body removal is detected. For example, increasing viscosity of the robot arms during draping / tear-down may help stabilize the robot arms and prevent excessive movement / runaway thereof, and increasing viscosity during tool / coupler body removal may prevent “runaway” due to a force applied to the robot arm by the user during the removal process. The localized virtual haptic boundary may be temporary in that it may only be applied for a predetermined time period after the predetermined phase is identified.

[0181] Operational mode determination module 828 further may include a robotic assist mode determination module that may be executed by processor 802 for analyzing the operating characteristics of robot arm 300 to determine whether to switch the operational mode of robot arm 300 to the robotic assist mode where processor 802 may instruct associated motors via motor interface module 824 to cause movement of corresponding link and joints of robot arm 300 to achieve a desired outcome. For example, the robotic assist mode determination module may determine to switch the operational mode of robot arm 300 to the robotic assist mode if a predefined condition exists based on data obtained from, e.g., optical scanner interface module713732261 v3 74225887-081001812. Moreover, the robotic assist mode determination module may determine that a condition exists, e.g., that one or more trocars are not in an optimal position relative to the robot arm, for example, due to movement of the patient and / or during initial setup, based on the perceived location of the one or more trocars received by optical scanner interface module 812 or trocar position detection module 818, such that robot arm 300 should be repositioned to maintain the trocar in the optimal position, e.g., in an approximate center of the movement range of robot arm 300, thereby minimizing the risk of reaching a joint limit of the robot arm during a procedure. Thus, in robotic assist mode, processor 802 may instruct system to reposition robot arm 300, e.g., via vertical / horizontal adjustment by platform 200 and / or via the joints and links of robot arm 300, to better align the surgical instrument workspace, and optimize a preset robot arm configuration.

[0182] In some embodiments, the robotic assist mode determination module may determine that a condition exists, e.g., initiation of an “instrument centering” mode by the user and identification of a target surgical instrument such as a handheld tool within the FOV of the scope attached to the robot arm, such that the robot arm should switch to robotic assist mode to provide assisted scope control to center the instrument within the FOV of the scope, as described in U.S. Patent No. 11 ,844,583 to Ye and WO 2024 / 150115 to Basafa. For example, during the robotic instrument centering mode, image motion calculation may be used to quantify image motion, e.g., the changes across consecutive images acquired from the scope resulting from movement of the scope during operation.

[0183] In the instrument centering mode of robotic assist mode, the robotic assist mode determination module may determine one or more conditions exist, e.g., the target surgical instrument within the FOV of the scope moves out of a predefined boundary region, e.g., a predefined rectangular or circular region about a center point of the FOV, within the FOV indicating that the robot arm should reposition the scope to maintain the target surgical instrument within the predefined boundary region within the FOV of the scope, and / or the resolution of the target surgical instrument in the scope feed falls below a predetermined resolution threshold indicating that the robot arm should move the scope to zoom out, and / or the detected size of the surgical instrument falls below a predetermined size threshold indicating that the robot arm should move the scope to zoom in. In some embodiments, the predefined713732261 v3 75225887-081001 boundary region may be a selected fraction of the FOV, e.g., the inner two-thirds of the FOV of the scope. Thus, movement of the target surgical instrument within the predefined boundary region may not cause movement of the robot arm, and accordingly movement of the scope; whereas movement of the target surgical instrument outside of the predetermined boundary region causes the robot arm to move the scope to maintain the target surgical instrument within the predefined boundary region within the FOV of the scope. Further, upon initiation of the instrument centering mode by the system to track a handheld surgical instrument within the FOV of the scope, using object segmentation to distinguish surgical instruments when more than one surgical instrument is within the FOV of the scope, the system may disregard non-target surgical instruments, e.g., surgical instrument s) coupled to a robot arm within the FOV of the scope, and track only the target surgical instrument, e.g., the handheld surgical instrument within the FOV of the scope.

[0184] As described above, a predefined gesture, e.g., double or triple tapping of the scope by the user while the robot arm coupled scope is in passive mode, perceived by gesture detection module 826 may initiate instrument centering mode without requiring the user to move away from the surgical site. For example, upon detection of the predefined gesture, the system may cause the robot arm to control the scope to maintain a surgical instrument currently in the FOV of the scope within the predetermined boundary region in the instrument centering mode. The system may continue to track the surgical instrument until another predefined gesture is detected that is associated with stopping instrument centering mode, or with tracking a different surgical instrument within the FOV of the scope, thereby automatically readjusting the pairing distance between the surgical instrument and the scope. For example, if the user double taps the scope while the system is currently tracking a surgical instrument in the instrument centering mode, the system may stop tracking the surgical instrument and permit the user to introduce another surgical instrument within the FOV of the scope within a predetermined pairing time period, e.g., 5 seconds, and track the surgical instrument detected within the FOV of the scope in the instrument centering mode upon the expiration of the predetermined pairing time period. If a surgical instrument is not detected within the FOV of the scope, the system may switch the operational mode of the robot arm, e.g., to the co-manipulation mode. As described in further detail below, the operational parameters (e.g., predetermined limits) during instrument centering713732261 v3 76225887-081001 mode, e.g., the maximum amount of “zoom” by the scope, may be automatically adjusted based on the known / predicted surgical phase of the procedure.

[0185] Phase determination module 830 may be executed by processor 802 for monitoring and analyzing data obtained via one or more data sources, e.g., image data captured by the optical scanners of the system, video feed of a scope attached to a robot arm of the system, voice commands / annotations by the user, telemetry data indicative of robot arm kinematics, or any combination thereof, in conjunction with associated timestamps to determine a predefined phase of a surgical procedure based on the data. For example, as described above, phase determination module 830 may use one or more object segmentation algorithms to segment image data received from / generated by optical scanner interface module 812, e.g., from one or more LiDAR scanners, 360 cameras, and / or proximity sensors of the system, to identify one or more objects (e.g., equipment / instruments), anatomical structures, and / or persons within the image data. Phase determination module 830 further may use one or more machine learning algorithms to determine the predefined surgical phase based on image segmentation data indicative of the one or more identified objects, anatomical structures, and / or persons as an input. For example, the machine learning algorithms may be trained to determine the predefined surgical phase based at least partially on historical data of similar procedures and / or real-time robot arm positions derived from the telemetry data as an input. Additionally, or alternatively, phase determination module 830 may determine the predefined surgical phase based on voice commands / annotations by the user recorded by the system, either passively or actively, during past similar procedures and / or in real-time during a current procedure. For example, phase determination module 830 may determine the predefined surgical phase based at least partially on surgeon annotation data from a previous procedure by comparing real-time image segmentation data (e.g., objects, anatomical structures, and / or persons identified in real-time image data) and / or real-time robot arm kinematics with image segmentation data and robot arm kinematics generated / observed / recorded by the system at the time of the surgeon annotation data from the previous procedure.

[0186] Knowledge of the phase of a surgical operation may, for example, indicate that the scope should focus on a specific surgical instrument and / or a specific anatomical structure in the instrument centering mode. As described above, the system may use object segmentation to713732261 v3 77225887-081001 identify anatomical structures within the FOV of the scope during a procedure, such that upon detection of a specific surgical instrument / anatomical structure, phase determination module 830 may determine that the surgical procedure has entered a known phase. For example, during a cholecystectomy procedure, upon detection and identification of the cystic duct and target artery, e.g., via object segmentation and an stored / online database, phase determination module 830 may determine the predefined phase of the cholecystectomy procedure, e.g., when surgical clips are to be applied, such that the robotic assist mode determination module may determine that the scope should track the cystic duct and the artery, and therefore maintain the cystic duct and the artery within the FOV of the scope during this known phase of the cholecystectomy procedure. Accordingly, the system may cause the robot arm to cause movement of the scope to center its FOV on the cystic duct and the artery.

[0187] As will be understood by a person having ordinary skill in the art, phase determination module 830 may be trained to identify various phases of various surgical procedures, such that the system may provide instrument centering to focus the FOV of the scope on key surgical instruments and / or anatomical structures based on the phase of the surgical procedure. Moreover, the system may provide automated centering to focus the FOV of the scope on key anatomical structures rather than a surgical instrument based on the type of surgical instrument identified within the FOV of the scope using any of the surgical instrument identification methods described herein. For example, if the surgical instrument is identified as a suture device, the system may provide assisted scope control to focus the FOV of the scope on the anatomical structure(s) being sutured rather than the suture device.

[0188] Additionally, or alternatively, phase determination module 830 may receive user input, e.g., via voice commands / annotations received by a microphone of the system, indicative of the phase of the surgical procedure. For example, as the user operates the system to perform a procedure, the user may be prompted to verbally describe when the user transitions from one phase to another phase, which may be recorded as surgeon annotations in a dataset, e.g., associated with the user’s surgeon profile, along with telemetry data indicative of robot arm kinematics during the annotated phase of the surgical procedure, image data captured by the optical / proximity sensors, and associated timestamps. The dataset may then be used to train a machine learning algorithm, such that phase determination module 830 may execute the machine713732261 v3 78225887-081001 learning algorithm to predict / estimate predefined phases of similar subsequent surgical procedures performed by the same user, e.g., by uploading the user-specific surgeon profile to the system.

[0189] For example, the trained machine learning algorithm may estimate the surgical phase based at least partially on the user-specific surgical phase annotations, the known robot arm kinematics during the surgical phase in real-time, and image data obtained by one or more optical sensors and / or a scope coupled to the robot arm. In some embodiments, the machine learning algorithm further may be trained via a dataset of annotated surgical phase data derived from previous similar surgical procedures performed by additional surgeons. Based on the predicted surgical phase of a surgical procedure, phase determination module 830 may estimate a surgical procedure end time, e.g., the time to completion of the surgical procedure, which may be communicated to operating room staff, e.g., via GUI 210 and / or a mobile application, to thereby facilitate preparation of a subsequent surgical procedure. In addition, phase determination module 830 may automatically generate a revised surgery schedule of the system, e.g., the hospital’s schedule of surgeries to be performed with a given system, based on the estimated surgical procedure end time, to thereby maximize efficiency and / or total number of surgeries abled to be performed per day. Accordingly, phase determination module 830 may access the system’s surgery schedule, e.g., by accessing a hospital’s Electronic Health Record (“EHR”) system, Electronic Medical Record (“EMR”) system, and / or Picture Archiving and Communication system (PACS), and further may upload the revised surgery schedule to the EHR / EMR / PACS, either manually by a user or automatically by the system. Moreover, a user may manually revise the surgery schedule based at least partially on the revised surgery schedule generated by phase determination module 830.

[0190] As a clinical procedure may be described as a sequence of clinical procedures steps, phase determination module 830 may learn these different steps to allow the system to infer in real-time the actual step for a given procedure. For example, learning clinical steps from procedures may allow or enable: adjustment of algorithm settings, adjustment of robot arm configuration to facilitate user action in a given phase, adjustment of a scope position based on the phase of the procedure, the system to give the practical custom reminders, the system to notify staff of an estimated procedure end time and / or workflow delays to facilitate block time713732261 v3 79225887-081001 optimization, the system to alert staff if necessary equipment is not available in the room, and / or the system to alert staff of the occurrence of an emergency situation. Block time, defined as the reserved, pre-scheduled time allocated to specific surgeons or medical equipment, e.g., the comanipulation surgical system described herein, is a crucial key performance indicator for hospital administrators. Effective block time utilization may be measured as the ratio of time used to time available for any given resource, including operating rooms and robotic assistance provided by the system, wherein time may be measured in minutes, hours, etc.

[0191] Operating room efficiency may frequently be inconsistent, e.g., due to a variety of possible delays throughout the surgical workflow, causing idle time, e.g., wasted time where the resource could be but is not used, leading to underutilization of block time, which may have a detrimental effect on hospital revenue, resource allocation, and overall patient care. For example, delays may be caused by: surgeon delays (e.g., surgeons arriving late to scheduled procedures, leading to idle operating room time); staffing and expertise (e.g., variability in staffing levels and the experience of operating room personnel can create inefficiencies during surgeries, extending turnover and preparation times); anesthesia delays (e.g., waiting for anesthesia to begin or wear off can cause idle time before, during, or after surgeries); room disinfection and turnover (e.g., the time to disinfect / turnover an operating room after a surgery can exceed the allocated target time, e.g., 10 minutes, prolonging turnover time and delaying the next surgery; pre-surgery delays (e.g., idle time before surgery can result from locating consent forms, preparing equipment, operating room setup, or staff assisting with unrelated duties such as phone calls or administrative tasks; equipment and supply issues (e.g., delays can occur when necessary instruments or supplies are not readily available, interrupting the surgical process); surgical complexity and intraoperative complications (e.g., unforeseen complexities and complications in real-time can extend the duration of surgeries beyond the predicted timeframe); variability in surgeon operating pace (e.g., differences in individual surgeons’ speed and workflow can lead to variations in procedure duration, making it challenging to maintain consistent schedules and causing potential idle time between surgeries; and post-surgery delays (e.g., downtime after surgery is often due to waiting for anesthesia to wear off, recovery room availability, or resolving inventory discrepancies such as sponge counts).713732261 v3 80225887-081001

[0192] Unoptimized block time may be indicative of mismanagement of operating room resources, resulting in increased downtime and lost revenue opportunities. This inconsistency not only impacts the financial performance of the hospital but also may affect overall patient care and operational workflows. Thus, monitoring and enhancing block time utilization may ensure efficient resource management and improved surgical outcomes, thereby improving daily surgical flow and optimizing long-term system performance. Accordingly, knowledge of the phase of a surgical operation in real-time may be used in conjunction with additional real-time data, e.g., telemetry data, image / audio data, etc., by phase determination module 830 to continuously monitor the progress of surgeries, capturing crucial data points (e.g., surgeon arrival time, anesthesia administration, surgical phase transitions, equipment availability, turnover times, etc.), thereby providing administrators and the surgical team with a real-time overview of the surgical workflow, which may be used to identify and address delays immediately and efficiently. Phase determination module 830 may aggregate the collected data into graphical representations illustrating block time utilization in real-time.

[0193] For example, FIG. 14A shows exemplary dashboards providing block time utilization insights in real-time, much like air traffic control for procedures, which may be displayed via GUI 210, a mobile application, and / or a wearable as described in further detail below. Specifically, FIG. 14A illustrates an exemplary surgical workflow from start to end (e.g., completion of system teardown) for a gastric bypass procedure performed by a specific surgeon with the system in an allocated operating room. As shown in FIG. 14 A, the procedure was scheduled to begin at 2:20 pm, with an estimated procedure end time of 4:10 pm, and 30 minutes allocated for system teardown (e.g., the block time reserved for the system for this particular procedure is from 2:20 pm to 4:40 pm). The dashboard also may graphically illustrate the current status of the system, e.g., “at surgery”, “teardown”, “idle”, etc., an estimated delay and the degree / timing of the delay, time remaining for a surgical phase or overall procedure, the system ID, the procedure type, the surgeon profile uploaded, etc. For example, as further shown in FIG. 14A, the dashboard may indicate when there is an estimated delay, e.g., a 20 minute delay during the procedure, and update the time remaining accordingly, as well as the amount of time the system is in idle mode, e.g., 20 minutes after the allocated teardown time, indicating that the system could be in use for another procedure, but is not, thus underutilizing the reserved713732261 v3 81225887-081001 block time. Accordingly, the dashboard may inform system utilization within a hospital, e.g., which particular system is in each operating room, the type of surgery being performed with a particular system, the estimated procedure end time for a particular surgery being performed by each system, etc.

[0194] As described above, the estimated surgical procedure end times for particular surgeries and / or for particular surgeons may be used to facilitate block time planning. For example, FIG. 14B illustrates exemplary block time planning for a five day surgery schedule. As shown in FIG. 14C, the dashboard may provide retrospective block time utilization insights by graphically illustrating the block time utilization of previous surgeries, which may further be surgeon specific, including, e.g., first case prep time, intraoperative time, closure time, teardown / turnover time, idle time, and / or block time reserved for a different surgeon. Accordingly, the retrospective block time utilization insights may inform future block time planning, as well as facilitate identification of causes / instances of block time underutilization / idle time. Moreover, data indicative of block time utilization over time may be graphically displayed for particular surgeons, as shown in FIG. 14D, and / or for particular systems, as shown in FIG. 14E.

[0195] Accordingly, tracking block time utilization may advantageously: optimize resource allocation (e.g., tracking block time and idle time allows administrators to better manage operating room resources, reallocating them when block time is underutilized); improve operating room efficiency (e.g., accurate data on block time usage may help optimize scheduling, ensuring surgeons utilize their allotted time effectively and reducing unnecessary idle periods); maximize revenue (e.g., monitoring block time may help hospitals fill operating room slots, preventing lost revenue from unused operating room time, thus securing a steady income stream); promote surgeon accountability (e.g., quantifying block time usage may encourage surgeons to adjust their schedules and reduce inefficiencies, fostering more efficient use of resources); improve cost management (e.g., tracking block time enables identification of high- performing, efficient surgeons and facilitates management of operating room costs, ensuring profitability and sustainability for the hospital); and enhance reporting (e.g., detailed utilization reports based on precise time metrics provide insights that may help optimize operating room performance, reduce idle time, and improve overall hospital operations).713732261 v3 82225887-081001

[0196] During a clinical procedure, the surgeon will often realize simple and routine surgical tasks such as grasping, retracting, cutting etc. Learning these different tasks may allow the system to infer in real-time preferences and habits of the surgeon regarding a sequence of a procedure in real-time. Phase determination module 830 further may tune (e.g., adjust and optimize) phase determination algorithms during the procedure based on this sequence recognition and help the user to be better at this simple surgical task. An example of such a task is the automated retraction of a liver during a gall bladder procedure. By aggregating the information over many cases, the optimized force vectors may be developed. Further, some complications may occur during a clinical procedure that may result in unexpected steps or surgical acts. Accordingly, learning how to discriminate these unexpected events helps the system to enable some specific safety features. For example, in case of emergency, the robot arms may be stopped or their motion restricted depending on the level of emergency detected by the system.

[0197] Trajectory generation module 832 may be executed by processor 802 for generating a trajectory from the current position of the distal end of the robot arm to a desired position of the robot arm. For example, to provide instrument centering in the robotic assist mode, trajectory generation module 832 may generate a trajectory which will result in moving the distal end of the scope from its current position to a desired position to maintain the target surgical instrument within the FOV of the scope. The generated trajectory is configured to provide the robot arm with precise control over the speed and smoothness of the motion of the surgical instrument, e.g., a scope, as it is moved along the trajectory. In addition, based on knowledge of the position of the trocar port(s) generated by trocar position detection module 818 and optionally, the type of surgical procedure to be performed, trajectory generation module 832 may generate a trajectory which will result in moving the robot arms, and any surgical instruments coupled thereto, from its current position to a desired position relative to the trocar port(s) that optimizes the workspace about the surgical site. As shown in FIG. 15, trajectory generation module 832 may implement trajectory generation process 1500 and query an online trajectory generation module trained with data obtained from previous surgical procedures including of preferred trajectories of various surgeons and associated surgical instrument locations and / or preferred robot arm configurations relative to known trocar port locations, to thereby generate the trajectory based on the713732261 v3 83225887-081001 determined robot arm configuration / surgical instrument orientation, as well as a specific surgeon’s preferences and / or an average preference of various surgeons.

[0198] Surgeon personalization module 834 may be executed by processor 802 for determining and / or storing information indicative of a specific user’s preferences regarding operation of the system including operating parameters. For example, surgeon personalization module 834 may learn a surgeon’s preferences based on data from past procedures and / or sensors collecting information about current procedure including a surgeon’s current pose, a surgeon’s height, a surgeon’s hand preference, and other similar factors. For example, surgeon personalization module 834 may record when a user interacts with the system and also record what the user does with the system, such that the dataset may allow for surgeon preferences to be “learned” and updated over time, as described below with regard to FIG. 16. This learning may be done either via traditional algorithmic methods (i.e., trends over time, averaging, optical flow, etc.) or via machine learning approaches (classification, discrimination, neural networks, reinforcement learning, etc.).

[0199] For example, when the system provides instrument centering by tracking a surgical instrument within the FOV of the scope, the operator at any time may “override” executed motion of the robot arm along the trajectory generated by trajectory generation module 832 in the robotic assist mode, e.g., by applying a force to the handle of the scope that exceeds a predetermined force threshold to cause the system to automatically switch the robot arm to the co-manipulation mode. Data indicative of the overriding motion by the user may be recorded by surgeon personalization module 834 and used to train the online trajectory generation module to thereby inform subsequent trajectories generated by trajectory generation module 832 for a similar procedure. For example, in the instrument centering mode, the system may provide a predetermined override time period where the user may manually move the scope, e.g., to change the FOV, clean the scope, adjust the zoom, etc., and return the scope to a stationary position before the system exits instrument centering mode.

[0200] For example, the predetermined override time period may be, e.g., 6 to 12 seconds, or preferably 8 seconds. During the override time period, the user may adjust the reference distance, e.g., the distance between the tip of the scope and the tip of the target instrument, by713732261 v3 84225887-081001 bringing the instrument tip to the desired distance from the scope for the scope to follow once the target instrument is detected in the instrument centering mode. Upon exiting of the instrument centering mode, instrument centering mode may be reentered via, e.g., actuation at GUI 210. Moreover, the overriding motion of the robot arm may be recorded as the operator’s preferred trajectory given the current position of the target surgical instrument, such that surgeon personalization module 834 may update trajectory generation module 832 associated with the given operator’s surgeon profile to include the operator’s preferred trajectory. The online trajectory generation module may be updated / trained with data indicative of the operator’s preferred trajectory and associated surgical instrument location, as shown in FIG. 15.

[0201] In addition, as described above, the surgeon profile associated with a specific user may include data (e.g., the user’s annotations from previous procedures) indicative of the user’s surgical preferences such as, e.g., the user’s surgical phase annotations and / or the user’s preferred operating parameters such as the user’s preferred optical scanner parameters. Accordingly, the user may upload their surgeon profile on the same or a separate system when performing a subsequent surgical procedure, such that the system may operate based on the user’s saved surgeon preferences. For example, the microphones of the system may permit the user to use voice commands to annotate and save preferred optical scanner preferences such as, e.g., angles / orientation, zoom levels, and movement speed for a particular procedure, which may be used to train a machine learning model that, when deployed by surgeon personalization module 834, adapts the system in real-time to the surgeon’s preferences saved to the surgeon’s profile. For example, the user’s voice commands may instruct the system to, e.g., save the camera distance for an anastomosis, update the camera movement speed to be slower during a dissection procedure, follow the tip of a surgical instrument in the instrument centering mode and then zoom out to view the complete instrument tip, etc. Moreover, surgeon personalization module 834 may execute a voice recognition algorithm to identify a particular surgeon based on detection of the surgeon’s voice within the operating room, e.g., either actively or passively via the system’s microphones, such that the profile associate with the particular surgeon may be automatically uploaded upon identification of the surgeon. For example, surgeon personalization module 834 may generate a prompt for the user to accept loading of the surgeon profile identified by the system based on the surgeon’s voice.713732261 v3 85225887-081001

[0202] FIG. 16 illustrates data flow 1600 for updating the system configurations based on learned behaviors of the user. As shown in FIG. 16, the system may be connected to an online database, e.g., the online trajectory generation module of FIG. 15, that may store a surgeon profile and each of a plurality of possible data sources, which may include optical sensors, encoders, and / or other sensors, and / or a database of manually entered user input. The data sources may be associated with a given surgeon, their preferred robot arm arrangement and operating parameters, and each procedure performed with the system, which may allow the recording and analysis of the system configuration and how it changes from procedure to procedure, and within the procedure. In the case of machine learning, the co-manipulation capability of the system may be leveraged such that the user’s actions may be used to annotate the data to create a training dataset. For example, surgeon personalization module 834 may record trajectories of a given surgeon using the instrument centering mode in a previous procedure, and train / update trajectory generation module 832 with the recorded trajectories to enhance that surgeon’s experience in a subsequent procedure, as well as other surgeons using the instrument centering mode.

[0203] Moreover, a distributed network of co-manipulation robotic (“cobot”) surgical systems may be used in multiple hospitals, each of which may be connected to an online database. This arrangement may provide considerably more data and user information that may be used by any of the cobot systems in operation. The systems may aggregate the data from the distributed network of systems to identify the optimum configuration / trajectories based on factors such as procedure type, surgeon experience, patient attributes etc. Through analytics or clinician input, the cobot systems may identify a routine procedure versus a procedure that may be more complicated. This information may be used to provide advice or guidance to novice surgeons. In addition, learning from a large number of procedures may result in a greater level of optimization of the cobot system setup for a given procedure. This may include, e.g., cart position, individual robot arm position, surgical table height and orientation, port placement, setup joints position, scope trajectories during instrument centering. These settings may be based on patient height, weight, and sex, and further may be interdependent. For example, the optimal port placement may depend on patient table orientation.713732261 v3 86225887-081001

[0204] Moreover, centralizing procedure data may enable the running of large data analytics on a wide range of clinical procedures coming from different users. Analysis of data may result in optimized settings for a specific procedure, including, e.g., optimized system positioning, optimal ports placement, optimal algorithms settings for each robot arm, and / or detection of procedure abnormalities (e.g., excessive force, time, bleeding, etc.). These optimal settings or parameters may depend on patient and tool characteristics. As described above, a surgeon may load and use optimal settings from another surgeon or group of surgeons. This way, an optimal setup may be achieved depending on, e.g., the surgeon’s level of expertise. To keep track of the various users in the distributed network of cobot systems, it may be beneficial to identify each user. As such, the user may log into the cobot system and access their profile online as necessary. This way the user may have access to their profile anywhere and will be able to perform a clinical procedure with their settings at a different hospital location.

[0205] 3D reconstruction generation module 836 may be executed by processor 802 for using one or more object segmentation algorithms to segment image data received from / generated by optical scanner interface module 812, generating segmentation data, and generating a 3D reconstruction of bodily tissues / organs and foreign objects such as surgical instrument(s) within the image data based on the segmentation data. In addition, 3D reconstruction generation module 836 may generate 3D reconstructions of one or more objects and / or persons in the operating room, e.g., patient bed, surgeon, assistant(s), etc., within image / depth data captured by optical scanners 202 and / or proximity sensors 212, as well as calculate the positions of the objects / persons relative to each other within the operating room.

[0206] For example, 3D reconstruction generation module 836 and / or optical scanner interface module 812 may perform filtering / other signal processing algorithms, e.g., median filter, Gaussian noise removal, anti-aliasing algorithms, morphological operations, ambient light adjustments, etc., on image / depth data acquired from one or more optical scanning devices, e.g., optical scanner 202 and / or proximity sensors 212. 3D reconstruction generation module 836 may perform 3D object segmentation using, e.g., template matching, machine learning, Brute force matching, color plus depth segmentation, object segmentation, 2D-3D registration, pixel value thresholding, etc., and then transform object coordinates to task space, e.g., by converting a position and an orientation of an object from the optical scanning device’s coordinate frame to713732261 v3 87225887-081001 the coordinate frame of the task needed (e.g., a robot frame for robot control, a cart frame for system setup, etc.). Additionally, or alternatively, transforming object coordinates to task space may include using known optical scanning device to the support platform (e.g., a cart) transformations, the surgical robot transformations, and / or the user interface screen transformations, and generating new transformations for specific tasks such as tracking the surgeon’s body (e.g., face, hands, etc.) with respect to different elements of the system (e.g., support platform, robot arms, screen, etc.), tracking the surgical table with respect to the cart platform, tracking patient orientation for system setup, tracking trocar port location and orientation for setup, and tracking the position of operating room staff for safety.

[0207] Moreover, 3D reconstruction generation module 836 may receive depth data obtained by proximity sensors 212 coupled to platform 200 and process the depth data to generate a virtual map of the area surrounding platform 200, which may be displayed to the operator via a monitor, e.g., display 210. For example, 3D reconstruction generation module 836 may, in accordance with system safety features, generate and display a virtual map of the area surrounding platform 200, e.g., via GUI 210, graphically depicting the 3D reconstructions of the identified objects / persons within the operating room in real-time, as shown in FIG. 17A, to guide movement of platform 200 and robot arms 300a, 300b through the operating room. As shown in FIG. 17A, the virtual map may include graphical representations of platform 200 (including robot arms 300a, 300b), as well as one or more objects, e.g., patient table PT, and / or one or more persons, e.g., operator O, person Pl, and person P2, within the area surrounding platform 200 in the same co-ordinate space as the platform and robot arms. Specifically, the virtual map may graphically illustrate the proximity between platform 200 and the one or more objects / persons, e.g., as platform 200 is being moved through the operating room by operator O.

[0208] For example, the system may cause display 210 to display the virtual map as operator O moves platform 200 through the operating room, e.g., relative to patient table PT, such that operator O can view the virtual map on display 210 in real-time. Accordingly, operator O may see objects and / or persons in the area surrounding platform 200 that operator O could not otherwise see with their own eyes, e.g., due to platform 200 and / or robot arms 300a, 300b obstructing the view of operator O, and thereby avoid collisions between platform 200 and / or robot arms 300a, 300b with the objects / persons in the operating room. As shown in FIG. 17A,713732261 v3 88225887-081001 the virtual map generated by 3D reconstruction generation module 836 may be a “bird’s eye view” of the area surrounding platform 200, e.g., within the operating room, from a top perspective. Alternatively, the virtual map may illustrate a perspective view of the operating room.

[0209] As described above, the optical scanners of the system may include one or more 360 cameras in addition to one or more LiDAR scanners. Accordingly, 3D reconstruction generation module 836 may receive depth data obtained by the LiDAR scanner and RGB image data from the 360 camera, and process the depth data and RGB image data together to generate a more detailed virtual map of the area surrounding platform 200, as shown in FIG. 17B, including graphical representations of the operating room (e.g., walls, doors, ceiling, floor, etc.), objects (e.g., platform 200 and robot arms 300, patient table PT, laparoscopy tower LT, monitors / screens, mayo stand MS, trash bins, etc.), and persons (e.g., operator O, person Pl, person P2, etc.). Advantageously, the 360 camera may fill in any gaps in the depth data captured by the LiDAR scanner, such that the generated virtual map provides a more comprehensive 3D reconstruction of the operating room and the objects / persons therein, which may be displayed, e.g., to facilitate positioning of platform 200, and accordingly robot arms 300, during setup, as described above. For example, a patient table may not be perfectly rectangular from a top view, particularly when draped, and thus, based on the combined data from the one or more LiDAR scanners and 360 cameras, 3D reconstruction generation module 836 may generate a more accurate graphical representation of the draped patient table that depicts the non-linear contour of the drape on the patient table. For example, 3D reconstruction generation module 836 may use Real-Time Appearance-Based Mapping (RTAB-Map), an RGB-D, Stereo and Lidar graphbased Simultaneous Localization and Mapping (SLAM) approach, to compress a 3D point cloud of the patient table / drape generated from depth data captured by the LiDAR scanner to a 2D representation, as shown in FIG. 17B. As will be understood by a person having ordinary skill in the art, the graphical representations generated by 3D reconstruction generation module 836 based on the combined data from the one or more LiDAR scanners and 360 cameras may be used for any of the image-enabled applications described herein, e.g., generating a “digital twin” of the system for review and / or training purposes and / or generating an operating room layout based on surgeon preferences, as described in further detail below.713732261 v3 89225887-081001

[0210] Moreover, the system may cause display 210 to display an alert, e.g., a visual or audible alert, when the virtual map indicates that platform 200 and / or robot arms 300a, 300b are approaching or within a predetermined distance from the one or more objects / persons within the operating room. In some embodiments, the system may only cause display 210 to display the virtual map while platform 200 is being moved within the operating room, e.g., when the braking mechanism is disengaged such that mobility of platform 200 is permitted. Accordingly, once the braking mechanism is re-engaged and / or when the platform 200 is stationary, display 210 may stop displaying the virtual map. In some embodiments, when the virtual map indicates that platform 200 and / or robot arms 300a, 300b are approaching or within the predetermined distance from the one or more objects / persons within the operating room, the controller may override actuation of the actuator by the operator and reengage the braking mechanism to thereby prevent further movement of platform 200.

[0211] Based on the data captured by optical scanner 202, 3D reconstruction generation module 836 may generate a virtual model of the pieces of capital equipment and / or other objects in an operating room that are within a range of movement of the robot arms in the same coordinate space as the robot arms and surgical instruments coupled thereto, such that the virtual model may be stored and monitored, e.g., to detect potential collisions, and / or displayed to the user during a setup stage and / or an operational stage, e.g., to guide setup of the operating room. Additionally, 3D reconstruction generation module 836 may track the position and orientation of each virtual model, and the objects within the virtual models as the objects move relative to each other, such that the system may alert the user if the proximity of (i.e., spacing between) any of the virtual models or objects falls below a predefined threshold, e.g., within 50 mm, 75 mm, from 30 mm or less to 100 mm, or more. The system may use this information to recommend a repositioning of platform 200 and / or other components of the system, the surgical table, and / or patient, and / or prevent the robot arm from switching to the co-manipulation mode as a result of the force applied to the robot arm by the collision with the staff member, even if the force exceeds the predetermined force threshold of the robot arm. Moreover, the system may stop or inhibit (e.g., prevent) further movement of a robot arm, e.g., freeze the robot arm, if the proximity of any of the virtual models or objects, e.g., a robot arm reaches or falls below the predefined threshold relative to other objects within the surgical space.713732261 v3 90225887-081001

[0212] FIGS. 18A-18C illustrate screenshots of an exemplary graphical user interface displaying the virtual map generated by 3D reconstruction generation module 836 to guide movement of the platform relative to a surgical bed, e.g., during setup. The graphical user interface may be configurable by a user and may be integrated with display 210. As shown in FIG. 18A, when the surgical bed is not within an acceptable proximity of the platform sensors, e.g., optical sensors 202 and / or proximity sensors 212, the graphical user interface may not display the virtual map depicting graphical representations of the platform and robot arms and the surgical bed, and may display an alert that the distance to the surgical bed / operation table is undetectable. As shown in FIG. 18B, upon detection of the surgical bed by the platform sensors, e.g., during movement of the platform, the graphical user interface may automatically display the virtual map, e.g., a “bird’s eye view” of the area surrounding the platform, depicting the platform and robot arms relative to the surgical bed. The display may depict the graphical representation of the surgical bed in a fixed position on the display, while the graphical representation of the platform and / or the robot arms is depicted as moving relative to the fixed surgical bed in realtime as the platform is moved by the user to guide movement and positioning of the platform and the robot arms by the user relative to the surgical bed. Alternatively, as shown in FIG. 18B, the display may depict the graphical representation of the platform and / or the robot arms in a fixed position on the display, while the graphical representation of the surgical bed is depicted as moving relative to the fixed platform as the platform is moved by the user to guide movement and positioning of the platform and the robot arms by the user relative to the surgical bed.

[0213] Moreover, the graphical user interface may display information indicative of, e.g., the distance between the platform (and / or the most extended stage / robot arm) and the surgical bed and the angle of the platform relative to the surgical bed, in real-time as the platform is moved relative to the surgical bed. As described above, the display may include graphical visual markers to guide positioning of the platform relative to the surgical bed. For example, a graphical illustration of a line / area fixed relative to the graphical representation of the surgical bed may be displayed, such that, when the graphical representation of the platform and / or the robot arms is aligned with or within the graphical illustration of the line / area, thereby indicating that the platform and robot arms are within an acceptable position / range from the surgical bed, the user may be alerted, e.g., via an audible and / or visual alert. For example, the graphical713732261 v3 91225887-081001 illustration of the line / area may change colors when the graphical representation of the platform and / or the robot arms is aligned with or within the graphical illustration of the line / area.

[0214] As described above, trocar position detection module 818 may detect the position of one or more trocars, if any, disposed in the patient and / or exposed skin of the patient, e.g., via object segmentation based on depth data obtained by the one or more optical sensors, and identify the boundary of the surgical site including the one or more trocars on the patient. Accordingly, as shown in FIGS. 18B and 18C, 3D reconstruction generation module 836 may generate a display including graphical illustration of the surgical site on the patient, e.g., boundary 1801, and / or of an alignment of between the platform and the surgical site in real-time as the platform is moved relative to the surgical bed, and accordingly the surgical site, to guide movement and positioning of the platform and the robot arms by the user relative to the surgical site. For example, a graphical illustration, e.g., dashed line 1802, may be displayed relative to boundary 1801 as the platform is moved relative to the surgical bed, and may be indicative of the position and / or orientation of the platform relative to the surgical site, e.g., a center of the surgical site, in real-time.

[0215] When the motion of the platform has stopped, e.g., when the platform has reached the target location relative to the surgical bed, the graphical user interface may stop displaying the virtual map, as shown in FIG. 18C. Accordingly, the graphical user interface may display the virtual map only during movement of the platform by the user, such that when the platform stops moving, the virtual map is no longer displayed. Alternatively, the graphical user interface may display the virtual map once the surgical bed is within an acceptable proximity to the platform sensors until the platform stops moving at the target location relative to the surgical bed. The graphical user interface further may indicate when the platform is in a locked state where mobility of the platform is prohibited, e.g., when the user actuates a locking pedal to engage the braking mechanism of the platform. In some embodiments, the graphical user interface may display the virtual map until the braking mechanism is engaged.

[0216] Moreover, 3D reconstruction generation module 836 further may generate and store an interactive virtual 3D reconstruction of the surgical scene during a surgical procedure including graphical representations of, e.g., the patient bed, the patient, trocar ports, platform 200713732261 v3 92225887-081001 and robot arms 300, surgical tools, etc., based on depth data obtained by optical scanner interface module 812, e.g., for training purposes and / or to improve surgical procedure efficiency. For example, the interactive virtual 3D reconstruction of the surgical scene may permit a remote user, e.g., via a virtual reality headset, to be immersed within the virtual 3D reconstruction of the surgical scene and observe / walkthrough the surgical procedure performed in the virtual 3D reconstruction by a “digital twin” of the system, e.g., a 3D reconstruction of the system. Accordingly, by interacting with the virtual 3D reconstruction, a trainee may learn how specific procedures are performed using the system and a surgeon may learn how to improve their use of the system to perform specific surgical procedures.

[0217] In addition, 3D reconstruction generation module 836 may generate and display a virtual map of an operating room, e.g., via GUI 210, a mobile application, and / or a virtual reality headset, graphically depicting one or more objects and / or persons and their spatial relationships relative to each other within the operating room based on surgeon preference to guide setup of the operating room for a specific procedure by surgical staff. For example, the virtual map of the operating room may include a digital twin of the system, e.g., at least platform 200 and robot arms 300, and its spatial distance relative other objects and / or persons within the virtual map. As will be understood by a person skilled in the art, operating room setup currently and predominantly relies on surgeon’s preference cards, such as that shown in FIG. 19A, which are essential for the efficient operation of the operating room, acting as a critical blueprint for surgical workflows and logistics. These cards bridge the gap between the availability of surgical inventory and the specific requirements of surgeons or particular procedures. By doing so, they play a pivotal role in optimizing operating room utilization, which in turn increases revenue, a benefit for all involved parties. However, despite their crucial role in the surgical care pathway, preference cards are often neglected, inadequately maintained, and fall below expected standards. Accordingly, 3D reconstruction generation module 836 may at least partially automate organization of the operating room based on the surgeon’s preference, e.g., based on surgeon input data indicative of the surgeon’s procedure preference saved to the surgeon’s profile. In addition, the automated organization of the operating room layout may further be generated at least partially based on dimensions / parameters of the operating room that the procedure will be performed in including, for example, the size of the operating room, as well as713732261 v3 93225887-081001 the location / orientation of fixed aspects of the operating room, e.g., the door(s), lights fixtures, wash station, electrical outlets, etc. For example, the operating room layout for the same procedure to be performed by the same surgeon may vary based on whether the door is on the right or the left side of the operating room.

[0218] For example, 3D reconstruction generation module 836 may generate a detailed 3D reconstruction of the operating room based on image data received by optical scanner interface module 812. The image data may be captured by the optical scanners during an environmental scan of the operating room, which may include guiding the user through the operating room to capture a comprehensive set of image data following an optimal trajectory. 3D reconstruction generation module 836 may post-process the captured image data using machine learning to enhance the accuracy and detail of the 3D reconstruction model, resulting in a 3D mesh that includes labeled features and a schematic representation of the operating room layout, which may include operating room elements such as walls, camera control unit (CCU) towers, the patient bed, surgical tables, fluoroscopy systems, doors, other recognizable objects, and / or a digital twin of the system. The initial schematic representation of the operating room, as shown in FIG. 19B, may be presented to the surgeon, e.g., via GUI 210, as a starting point, and the surgeon may interact with the graphical representations of the objects in the schematic representation to customize their preferred operating room layout, e.g., positioning of various objects in the operating room including surgical tools, patient, and ergonomic screen positioning, as shown in FIG. 19C, which may be saved to the surgeon’s profile for the specific procedure. The 3D reconstruction of the operating room layout may be displayed in one or more views including, e.g., a top view as shown in FIGS. 19B and 19C, a 3D perspective view, etc.

[0219] For example, in some embodiments, the initial schematic representation may be a predetermined / known ideal operating room layout for a particular surgical procedure, and may be automatically preconfigured based on available instruments and operating room dimensions / parameters on a particular day, which may be manually entered by a user or automatically determined based on image data may be captured by the optical scanners. Additionally, or alternatively, the initial schematic representation may be saved as part of the installation process of the system, e.g., the operating room layout may be manually configured by staff for each of the different procedures that may be performed by a surgeon, which may or713732261 v3 94225887-081001 may not be surgeon specific, such that the layout for each particular procedure may be saved as the initial schematic representation of the operating room for the respective procedure.

[0220] 3D reconstruction generation module 836 may further analyze the surgeon’s final preferred layout and generate warnings and / or suggestions if the setup is determined to be suboptimal for the intended procedure, e.g., based on spatial constraints, positioning of equipment, accessibility issues, etc., such that the surgeon may choose to make informed adjustments to their preferred layout to ensure that the final setup is both practical and conducive to an efficient and safe surgical procedure. Accordingly, a 3D reconstruction of the final operating room layout may be displayed to the surgical staff, e.g., via GUI 210, a mobile application, and / or a virtual reality headset, including equipment and position labels to guide setup of the operating room in accordance with the surgeon’s preference, e.g., by allowing the surgical staff to match the positioning of objects in the operating room with that shown in the graphical reconstruction. Moreover, the surgeon’s preferred layout, once saved, may be uploaded from the surgeon’s profile for a subsequent procedure, e.g., manually via GUI 210 or via voice command, which may be particularly useful in cases where the scrub assistant or circulators are not familiar with the particular surgeon’s operating room layout preferences. In addition, the surgeon may, at any time during a surgical procedure, cause the system to save the real-time operating room layout as the surgeon’s preferred layout for the particular surgical phase and / or for the particular surgical procedure, and upload the preferred layout to the surgeon’s profile.

[0221] In some embodiments, 3D reconstruction generation module 836 may execute machine learning algorithms to recommend an optimal operating room layout based on the known size / parameters of the operating room, e.g., determined from the image data captured by the optical scanners during the environmental scan of the operating room. In addition, based on the known surgery schedule, e.g., obtained from the EHR / EMR / PACS as described above, 3D reconstruction generation module 836 may optimize available resources. For example, 3D reconstruction generation module 836 may determine what additional equipment / instruments will be required for subsequent surgical procedures throughout the day, and generate / include graphical representations of the additional equipment / instruments in the virtual map of the operating room for setup of an earlier procedure, such that the staff may include the additional713732261 v3 95225887-081001 equipment / instruments in the operating room prior to when they are actually needed for the corresponding procedure, to thereby minimize the time required to setup the operating room for the subsequent surgical procedures. For example, if a C-arm is required for the third case on the surgery schedule, 3D reconstruction generation module 836 may include a graphical representation of the C-arm in the virtual map of the operating room layout for the first or second case on the surgery schedule, e.g., based on spatial constraints / available space within the operating room of the first and second cases, such that the C-arm may be pre-positioned within the operating room prior to operating room setup for the third case.

[0222] Moreover, in addition to guiding operating room setup based on a surgeon’s preference, 3D reconstruction generation module 836 further may generate a 3D reconstruction of the patient and / or equipment / instruments to facilitate patient preparation based on the surgeon’s preference, which may typically be included in a preoperative preference card filled out by the surgeon. For example, a surgeon may have a preference regarding leads / lines placement on a patient for a particular procedure. Accordingly, 3D reconstruction generation module 836 may generate a virtual map including graphical representations of the leads / lines relative to the patient’s body, which may be displayed to the staff to guide patient preparation in alignment with the surgeon’s preferences, as shown in FIG. 19D, thereby contributing to an overall smoother surgical workflow. As shown in FIG. 19D, the graphical display further may include a checklist of actions items to be completed during patient preparation. Additionally, 3D reconstruction generation module 836 may generate an alert, e.g., an audio or visual alert, indicative of whether the operating room / patient is correctly setup / prepped in accordance with the virtual map, e.g., if the placement of equipment / instruments within the operating room and / or on the patient’s body falls within a predetermined spatial threshold from the graphical representations thereof in the virtual map.

[0223] In addition, 3D reconstruction generation module 836 may generate a 3D reconstruction of platform 200 and robot arms 300 (e.g., the digital twin of the system) during a surgical procedure based on known kinematics thereof, e.g., based on motor currents measured by motor interface module 824, including graphical representations of objects and / or persons detected within the operating room, as shown in FIG. 20A, a top view, and FIG. 20B, a perspective view. The digital twin of the system may include associated timestamps, such that a713732261 v3 96225887-081001 user may playback the 3D construction and observe the configuration of platform 200 and robot arms 300 throughout the surgical procedure, e.g., for training purposes and / or to improve surgical procedure efficiency. 3D reconstruction generation module 836 further may generate graphical representations of various parameters, e.g., forces applied to the patient / trocar by the surgical instrument determined by force detection module 820, and cause the graphical representations to be displayed alongside and in synchronization with the 3D reconstruction of platform 200 and robot arms 300 during the surgical procedure, as shown in FIG. 20C, to provide further insight of the surgical procedure. As shown in FIG. 20C, 3D reconstruction generation module 836 also may cause image data, e.g., timestamped video feed of optical scanners 202, to be displayed alongside and in synchronization with the 3D reconstruction of platform 200 and robot arms 300 during the surgical procedure.

[0224] Case counting module 838 may be executed by processor 802 for automatically determining when the co-manipulation surgical system has been used for a surgical procedure (e.g., a “case”) and keeping track of the number of cases performed by the system, e.g., for customer invoicing purposes. For example, a subscription model may be used where the customer, e.g., a surgeon or hospital, is charged based on their usage of the system, which may require an accurate case count. Accordingly, case counting module 838 may automatically determine and keep an accurate case count without requiring user input from the customer, e.g., from hospital staff or field engineers, to provide information regarding system usage.

[0225] As described herein, data indicative of various key events between the initial setup stage and the teardown stage at the end of a procedure collected during use of the system may be leveraged to capture key state transitions of the system workflow, which may be used by case counting module 838 to determine that the system has been used to perform a surgical procedure (e.g., case count = 1). For example, a general case may begin when system starts up, then continues to initial setup where user selects the surgeon profile, procedure type, and drape mode to drape the system. After initial setup, the system proceeds to sterile setup where the couplers are attached, and system is positioned in the desired position relative to the patient bed for the surgical procedure. After sterile setup completes, the system proceeds to the intraoperative stage where tools are attached and used during the procedure. When the procedure is complete, the system goes into the teardown stage where the tools and couplers are detached, and system is713732261 v3 97225887-081001 deployed to the compact and / or stow pose and moved away from the patient bed. For each general case, event logs, robot telemetry logs, and timestamped videos are obtained via image data / video feed captured by the scope and / or optical scanners (e.g., depth cameras). Table 1 below summarizes how the key states may be identified from the different data sources.Table 1713732261 v3 98225887-081001

[0226] Case counting module 838 may use a combination of one or more of the above detected data points to extract the timestamp of the key state transitions and capture the duration of each key state. Moreover, case counting module 838 may filter the extracted data based on, e.g., surgeon profile selection, typical procedure duration, and / or minimum first tool attached to last tool attached time, to narrow down the search of an actual procedure and provide robust case count information. Case counting module 838 may then aggregate and store the total count of surgical procedures performed by the system over time, which may be queried at a predetermined frequency preferred / used by the customer for invoicing, e.g., monthly, to determine the total case count, e.g., number of surgical procedures performed by the system, within the predetermined period associated with the predetermined frequency, e.g., a month.

[0227] Additionally, or alternatively, case counting module 838 may extract usage information from audio data captured via the system within the operating room. For example, case counting module 838 and / or language model module 846, described in further detail below, may execute speech recognition techniques and / or large language models to identify audible conversations during the procedure, procedure phases, and time of the key state transitions. The language model may be used in combination with the data sources discussed above to further refine the usage criteria. Moreover, in some embodiments, the system further may use a combination of external data inputs from additional data capturing devices in the operating room and run a visual understanding machine learning model to extract case relevant information therefrom for use in refining usage information extracted directly from the system, as described above.

[0228] The data generated by case counting module 838 may be transmitted to an invoicing system, e.g., via the Cloud. For example, the data may be automatically uploaded to the Cloud713732261 v3 99225887-081001 through an upload agent that uploads data whenever the system is connected to the Internet (e.g., 5G / WiFi / Ethernet), and data transforms may be deployed to the Cloud and run to extract the key events log, which may be ingested into a case count database that lists information of date / time, system serial number, surgeon name, procedure name, duration of tool attached, duration of coupler attach, and timestamps of each key state of the system workflow from system initialization, initial setup completion, sterile setup completion, procedure completion, teardown completion, and shutdown. In addition, the case count database may be connected to other CRM systems to extract site information and joined together with the case count information, as well as to payment systems to bill the customer and send the invoice automatically through the payment system based on the query result of the system usage at any frequency preferred / used by the customer, e.g., monthly, quarterly, etc. The extracted information further may provide insight as to how the system is actively used in the hospital, how different sites use the system, and how a case performed using the system compares to a case manually performed by a surgeon, as well as for comparing statistics between different surgeons and providing surgeon feedback to improve the surgeon’s use of the system. In some embodiments, instead of performing data processing and data transformation in the Cloud, case counting module 838 may perform data processing directly on the system with another program running to consume data locally, generate case count information locally, and save the case count information in a local database of the system. An upload agent may then upload only the new entries of the local database information to the Cloud to update the case count database on the Cloud.

[0229] Surgical item retention module 840 may be executed by processor 802 for detecting foreign objects, e.g., surgical items such as sponges, gauzes, and needles, entering and leaving the patient’s body based on image data obtained by optical scanner interface module 812. For example, as described above, optical scanner interface module 812 may execute object segmentation algorithms to analyze image data obtained from the scope and identify predefined surgical items within the FOV of the scope. Additionally, or alternatively, optical scanner interface module 812 may execute object segmentation algorithms to analyze depth data captured by optical scanners 202 to identify / detect predefined surgical items at the surgical site. Surgical item retention module 840 may track and keep count of the identified surgical items that enter and leave the patient’s body during a surgical procedure, and automatically generate an alert,713732261 v3 100225887-081001 e.g., via GUI 210 and / or indicators 334, at the completion of the surgical procedure, e.g., based on detected surgical phases and / or key state transitions of the surgical workflow, if the count of identified surgical items that leave the patient’s body does not match the count of identified surgical items that entered the body during the surgical procedure. Surgical item retention module 840 may be programmed to distinguish between surgical items that are intended to be retained within the patient’s body, e.g., sutures, staples, etc., and the surgical items that should not be retained within the patient’s body, e.g., sponges, gauzes, needles, etc., such that surgical item retention module 840 only tracks and keeps a count of the surgical items that should not be retained within the patient’s body. Moreover, if a surgical item’s exit count is less than its exit count, surgical item retention module 840 may identify the timestamp of the image data of the surgical procedure of when that surgical item was last detected in the image data, such that the image associated with the timestamp may be displayed to the surgeon to facilitate locating and removal of the surgical item.

[0230] Report generation module 842 may be executed by processor 802 for generating an operation summary report associated with a specific surgeon profile including information regarding the specific surgeon’s use of the system to perform a surgical procedure. For example, the information may include the timestamped force profile of the surgical instrument throughout a surgical procedure generated by force detection module 820, as shown in FIG. 21 A. By reviewing the force profile for a past surgical procedure, the surgeon may see when and where excessive force was applied to the patient’s body, e.g., at the trocar incision sites, and reduce such forces in future procedures to thereby reduce the risk of and / or prevent postoperative hernia due to excessive body wall forces. The information further may include, e.g., operation parameters of the co-manipulation surgical system during a surgical procedure, time spent in a procedural phase of the surgical procedure, detection of fault conditions and screenshots and / or timestamped video clips related thereto, instances of when the surgeon manually overrides an automated movement of the robot arms such as readjustment of a scope coupled to the robot arm, amount of repositioning / actuation of the stage assembly of the surgical platform and the Q3 setup joint, indications of moments of excessive force, etc. Accordingly, based on the information provided in the report, the surgeon may reduce interruption of the workflow of future procedures to improve use of the co-manipulation surgical system.713732261 v3 101225887-081001

[0231] Moreover, during surgical procedures performed with the system, the surgical team may discuss various steps, and decisions, while optical scanners capture key moments, and the surgeon may audibly provide dictation for important milestones, decisions, and / or complications. Accordingly, in some embodiments, the microphones of the system may “passively listen”, e.g., automatically record audio data within the operating room, upon system start up, during a surgical procedure performed with the system, and / or for a predetermined time period after the completion of a surgical procedure, or alternatively, until shutdown of the system, such that report generation module 842 may execute a machine learning algorithm to analyze the recorded audio and automatically generate a comprehensive surgery report summarizing the surgical procedure performed with the system by a user including, for example, surgical procedures / events performed, surgical phases, instruments used, and corresponding timestamps. For example, the recorded audio may include voice commands by the user, such as to save a particular view of the image data captured by the system’s optical scanners and / or scope feed, e.g., a screenshot or video snippet, such that the corresponding saved data may be included in the comprehensive surgery report. Moreover, the recorded audio may include voice commands to query the EHR / EMR / PACS systems to inquire about patient history, allergies, key biomarkers, etc., such that the system may provide the requested information in response in real-time, and / or in the summary report.

[0232] Such summary reports are typically required to be manually created by the surgeon at some time after the surgical procedure to create a record of the steps performed during the surgical procedure, e.g., for hospital billing purposes, which may be based on the surgical procedures / events performed by the surgeon. Thus, the automatically generated surgery report may save time and improve post-operative documentation accuracy, while minimizing the risk of missing important details during post-operative reporting / documenting. In some embodiments, report generation module 842 further may access a library of hospital billing codes, e.g., via the EHR / EMR, and automatically map particular billing codes to key events / surgical phases detected and described in the surgery report, to facilitate accurate accounting of the surgical procedures performed by the surgeon during an operation, and accordingly, accurate billing of the operation for the hospital.713732261 v3 102225887-081001

[0233] As described herein, report generation module 842 may use data obtained from additional data sources, e.g., telemetry data, video / image data, etc., including surgeon’s commands / annotations and knowledge of surgical phases, in conjunction with the passively recorded audio data to identify key events before, during, and after a surgical procedure to facilitate generation of a comprehensive summary report. For example, in generating the surgery report, report generation module 842 may combine key discussions (e.g., based on passively and / or actively recorded audio data, including the surgeon’s direct input), workflow actions (e.g., decisions made during the procedure), optical scanner feed insights (e.g., captured image data indicative of notable anatomy, important visual markers, and significant events), tool usage, and voice annotations to provide a holistic view of the surgical operation, without interrupting the surgical team. The surgery report may be communicated to the surgeon for review and approval, and further may be used for post-operative analysis, documentation, and improving the accuracy of medical records.

[0234] FIG. 21 B illustrates an exemplary interactive dashboard that aggregates the information compiled by report generation module 842 for ease of understanding and review by a user. For example, the dashboard shown in FIG. 21B illustrates an interactive summary report for a gastric sleeve procedure including, e.g., information indicative of whether the procedure was a solo surgery, a planned surgery, and / or for training a resident; timestamps of various key events such as when the braking mechanism of the wheels are engaged / disengaged, when patient preparation was completed, when an incision was made, and when the incision was closed; force profiles of the independent robot arms such as during instrument centering mode (“ScoPilot”) and while attached to a grasper tool; recorded video data such as the video feed of the scope for playback and review; timestamped annotations regarding corresponding surgical phases; the 3D reconstructed digital twin of the operating room including the system and robot arms, equipment, tools, persons, etc.; surgeon audio notes recorded during the procedure; and Al generated case notes. In addition, report generation module 842 may generate scores based on the performance by the surgical team, e.g., an ergonomic score, efficient motion score, etc., which also may be included in the interactive summary report, as shown in FIG. 2 IB.

[0235] In addition, the information aggregated by report generation module 842 may be transmitted to one or more mobile applications, e.g., a hospital staff mobile application, as shown713732261 v3 103225887-081001 in FIG. 22A, and / or a surgeon mobile application, as shown in FIG. 22B, and / or may be uploaded to the hospital’s EHR / EMR / PACS. As shown in FIG. 22A, the staff mobile application may include graphically displayed information indicative of, e.g., the status / phase of each independent co-manipulation surgical system being used in each operating room (e.g., OR 1, OR 2, OR 3, etc.) to improve the staff s efficiency of operating room teardown and setup for subsequent procedures, inventory of surgical instruments and accessories for use with the system for hospital inventory management, general surgeon specific data such as usage of the system, etc. In some embodiments, if the inventory of particular surgical instruments / accessories if determined to be low, e.g., falls below a predetermined threshold, the system may automatically trigger a restock of the particular item to ensure the inventory is always sufficient. For example, the system may generate an alert to inform the user / hospital of the low inventory so that the inventory may be replenished.

[0236] As shown in FIG. 22B, the surgeon mobile application may permit a surgeon to view their own surgeon-specific information, e.g., via surgeon specific login credentials. For example, the surgeon may view graphical representations of the information aggregated by report generation module 842 such as usage of the co-manipulation surgical system for a given type of surgical procedure, time spent using the system top perform specific surgical procedures, force profiles, etc. Surgeons further may share their data / statistics with each other via the surgeon mobile application, e.g., for education purposes. Additionally, as described above, the staff and / or surgeon mobile applications may be used to facilitate operating room setup based on the surgeon’s preference. For example, the surgeon may (via the surgeon mobile application) input their preferred operating room organization, which may then be displayed (via the staff mobile application) to guide setup of the operating room in accordance with the surgeon’s preference.

[0237] Wearables interface module 844 may be executed by processor 802 for wirelessly communicating with wearables worn by the user / surgeon, e.g., smart watches, smart rings, smart glasses, etc., to capture additional real-time data that may be used in any of the applications described herein, to communicate information to the user, as well as to receive and track physiological surgeon parameters during a surgical procedure. For example, as shown in FIG. 23, wearables interface module 844 may collect image and / or audio data captured by smart glasses worn by a user, which may be used in conjunction with image data captured by the713732261 v3 104225887-081001 optical scanners of the system and / or by a scope operatively coupled to the system, and / or audio data recorded by the microphones of the system to, e.g., generate a more comprehensive virtual map of the operating room, understand the current operating room layout to guide setup thereof in accordance with the surgeon’s preference, detect input gestures, determine predefined surgical phases, identify surgeon preferences based on real-time surgeon actions during a procedure, case count, identify and track objects within the operating room and / or within the patient’s body, identify trocars on the patient’s body, etc.

[0238] Moreover, wearables interface module 844 may cause the smart glasses worn by the user to communicate (e.g., visually and / or audibly) information to the user to facilitate / guide user actions during the procedure. For example, the smart glasses may communicate real-time audio and / or augmented alerts / instructions / feedback to guide the user on instrument changes (e.g., what instrument will be needed in the next surgical phase), the surgeon’s preferences (e.g., preferred instrument types for particular procedures, preferred equipment / instrument placement, etc.), trocar placement on the patient’s body, etc., thereby reducing inefficiencies, delays, and variability in surgical outcomes, and enhancing overall surgical team coordination and workflow efficiency. In some embodiments, wearables interface module 844 may automatically selectively collect image / audio data captured by the smart glasses based on, e.g., the position / orientation of the smart glasses and / or other indicators, such that only relevant image / audio data is collected, e.g., when the user, and accordingly the smart glasses, are facing towards the surgical workspace, thereby reducing unnecessary data overload while maintaining focus on the most critical moments during surgery. Additionally, after a procedure, the captured image / audio data may be used for post-operative analysis, training, and quality assurance. For example, this data may be reviewed to identify areas for further improvement in surgical efficiency and team coordination by, e.g., having examples of instructions displayed on the digital twin platform and / or ranking teams or individual. Accordingly, the smart glasses may advantageously standardize key aspects of the surgical process, mitigate the challenges of adapting to varying surgeon preferences, reduce the learning curve of an assistant working with a new or unfamiliar surgeon or on a new procedure, minimize the time needed by an assistant to accommodate surgeon-specific workflows, and permit the user to focus on the task at hand by virtue of the hand-free nature of the smart glasses.713732261 v3 105225887-081001

[0239] The smart glasses may include, e.g., Ray-Ban Meta glasses (made available by Luxottica Group S.p.A, Milan, Italy), Orion AR glasses (made available by Meta Platforms, Inc., Menlo Park, California), or the like, and / or may adapt Al-driven platforms such as those made available by expanded existence, Inc., Orlando, Florida. In addition, wearables interface module 844 may collect data indicative of the surgeon’s stress level, heart rate, blood oxygenation, etc. during specific procedural phases of a surgical procedure, which may be conveyed in the operation summary report generated by report generation module 842. Wearables interface module 844 further may wirelessly communicate with other devices such as mobile phones to permit the surgeon to place phone calls during a surgical procedure without interrupting the surgical workflow.

[0240] Language model module 846 may be executed by processor 802 for receiving and analyzing audio commands from the user, e.g., via one or more audio microphones of the system, and feeding the audio data into a large language model (LLM, e.g., a chatbot such as ChatGPT made available by OpenAI or OneMeta made available by OneMeta Inc.) and / or a large medical model (LMM, e.g., Gemini made available by Google) to co-analyze the audio data via natural language processing (NLP), identify essential information in the audio data to identify the command / question, and generate audio data indicative of answers to the identified questions, which may then be emitted to the user, e.g., via one or more speakers of the system. For example, as shown in FIG. 24A, the user may ask the system what instrument is being used and language model module 846 may generate a response based on any of the automated surgical instrument identification techniques described above, e.g., object segmentation of image data captured by a scope and / or optical scanners 202. Accordingly, language model module 846 may use image data / video feed received by optical scanner interface module 812 as an input to the LLM / LMM. As another example, the user may ask what the next phase of the surgical procedure is and language model module 846 may generate a response based on any of the automated surgical phase detection techniques described above, e.g., stored surgeon annotations and / or system inferences based on identified surgical tools and known surgical procedure type.

[0241] In addition, language model module 846 also may generate data indicative of commands to be implemented by the system, to thereby cause the system to perform actions in accordance with the user’s audio command. For example, as shown in FIG. 24B, the user may713732261 v3 106225887-081001 ask the system to look at (e.g., track) a specific anatomical structure in the instrument centering mode and language model module 846 may generate a command that causes the robot arm to control the scope coupled thereto to track the specific anatomical structure in the instrument centering mode. As another example, the user may ask the system to move the stage assembly up “a little bit” and language model module 846 may generate a command that causes the stage assembly to move the base of the robot arms vertically by an estimated distance in response. If the user is unsatisfied with the estimated distance, e.g., if the distance was not enough, the user may ask the system to move the stage assembly up “a little more,” resulting in an additional upward movement of the stage assembly. Language model module 846 may learn over time what a specific surgeon means by “a little bit” based on similar previous commands by the surgeon, such that the next time the surgeon asks the system to move the stage assembly by “a little bit,” language model module 846 will generate a more accurate corresponding movement. Accordingly, language model module 846 may access the surgeon’s profile including learned surgeon behaviors and preferences generated and stored by surgeon personalization module 834, e.g., information indicative of a specific user’s preferences regarding operation of the system including operating parameters, as described above.

[0242] Moreover, by incorporating image data as an input, language model module 846 may generate more accurate system commands, e.g., if the video distance is the same as a previously recorded procedure. In addition, language model module 846 also may cause the system to perform actions in accordance with the user’s voice commands obtained during training whether directly with the system or via a digital twin of the system, as described above. For example, during training, the trainee may ask the system questions, e.g., “what does this button do?” or “how do I lock the system?”, such that language model module 846 may process the command and generate a detailed explanation and / or prompt the trainee to take specific actions and to confirm when each action is completed. Language model module 846 further may generate step- by-step instructions responsive to the trainee’s command to guide the trainee in performing a specific action, e.g., positioning the arms, adjusting settings, managing the interface, etc. Similarly, outside of training purposes, e.g., during a surgical procedure, language model module 846 may provide real-time assistance for system adjustments and workflow support responsive to voice commands by the surgical team. Depending on the assistance requested by the user,713732261 v3 107225887-081001 language model module 846 further may communicate image and / or video data to the user, e.g., via GUI 210, to guide the user through a specific action. Accordingly, language model module 846 may provide voice-enabled accurate, on-demand support and / or answers to common questions from the surgical team, thereby streamlining operations and ensuring faster decisionmaking and adjustments without interrupting the workflow, while reducing the need for manual searching or troubleshooting during critical moments of the procedure.

[0243] Predictive maintenance module 848 may be executed by processor 802 for monitoring and analyzing data obtained from various data sources, e.g., application event logs, robot telemetry data, 3D depth data, color images, and laparoscopic camera data, collectively referred to herein as “fingerprints” of the system to perform predictive data analytics on the data, e.g., by comparing the fingerprint data to manufacture and / or installation baseline data of the system, to determine if any field inspection and / or maintenance / service is needed. Unexpected system and equipment failures may lead to significant operational disruptions and financial losses, and traditional maintenance strategies often rely on scheduled maintenance or reactive approaches, which may not effectively address issues before they impact system performance.

[0244] Traditional maintenance strategies may include, e.g., scheduled maintenance, reactive maintenance, and / or manual data analysis. Scheduled maintenance strategies involve conducting maintenance tasks at fixed intervals such as weekly, monthly, or annually. For example, field service technicians may inspect system health and performance every fixed number of months, regardless of system condition. Individual reports from these scheduled maintenance checks provide periodic snapshots of the system’s condition, and such visualizations in these reports may illustrate relevant insights across commercially deployed systems. Reactive maintenance strategies traditionally involve addressing system issues after they have caused noticeable problems or failures. For example, if a robot arm or collar button of the system malfunctions, maintenance personnel may be called to address the issue. This solution focuses on fixing problems as they arise and does not involve regular maintenance check or predictive analysis. Reported issues may be documented and analysis may be performed to provide insights into any patterns that are observed on a case-by-case basis. Manual data analysis strategies involve manually analyzing operational data to identify trends or anomalies. For example, a user, e.g., an analyst, may review system performance logs to inspect data trends and flag known issues713732261 v3 108225887-081001 that may have previously been reported. This solution may also be used periodically and may use known data analytics tools to extract and visualize relevant trends.

[0245] In contrast, predictive maintenance module 848, which may include a Cloud infrastructure, may ingest data received from various data sources (e.g., telemetry data, video data, audio data, etc.), process the data through data pipelines, update system records, monitor and analyze the operational fingerprints of the system, establish expected baseline values for each fingerprint from the time of system manufacture release, the baseline values corresponding to a healthy operational state of the system, track the fingerprints changes as the system goes through its life cycle, and perform predictive analysis using historical data and pattern recognition to detect anomalies and predict maintenance needs proactively. Once anomalies are detected indicating a system is close to an unhealthy state, an alert may be generated and transmitted, e.g., to system manufacturer personnel, for purposes of scheduling predictive maintenance. Additionally, or alternatively, the alert may be sent to the surgeon, staff, and / or hospital, e.g., via GUI 210 and / or a mobile application. The personnel may be onsite to resolve the issue and bring the system back to a healthy state before the system experiences significant operational disruptions, thereby minimizing system downtime, e.g., the total time a system is in critical fault state and not available for use, and extending the lifespan of critical features of the system. In contrast, system uptime may refer to the total time when the system is available for use, and system ontime may refer the total time that a system is powered on.

[0246] The fingerprints of the system analyzed and monitored by predictive maintenance module 848 may include, for example, motor currents, tool sensors, collar buttons, e.g., setup mode actuator 336 of collar 330, proximity sensors, e.g., proximity sensors 212a, 212b, immobilization, use errors, stage usage, temperature, voltage, system faults, fault start time and seconds to fault, disk usage, and / or CPU consumption. For example, as described above, motor interface module 824 may receive and record motor current readings through system telemetry across each motor of the system, e.g., Ml, M2, M3, M4, and the motors associated with the stage assembly, each representing a different motor current value. Predictive maintenance module 848 may monitor the motor current readings during various pose-to-pose transitions, e.g., during the setup phase of the system, which may include steps such as from stow mode to drape mode and from drape mode to a preset configuration, etc. In addition, based on the collected data,713732261 v3 109225887-081001 predictive maintenance module 848 may determine, e.g., average, maximum, and integral values for each motor current during each transition. For example, the integral value is the sum of motor current over time.

[0247] Predictive maintenance module 848 may monitor the collected data, track changes in motor current values over time, filter data by system or robot arm, compare the collected data against baseline values, e.g., compare the data collected during transition from stow mode to drape mode against baseline values for the same transition under normal operating conditions (e.g., a healthy operational state), and analyze trends to thereby detect potential issues before they occur. For example, deviations from expected patterns or sudden spikes in motor current values may indicate anomalies. Moreover, predictive maintenance module 848 may generate an alert when deviations exceed predefined thresholds, such that predictive maintenance may be planned based on the observed trends and historical data, ensuring timely interventions to prevent system failures.

[0248] Predictive maintenance module 848 further may aggregate the collected data into tables and / or charts, as shown in FIGS. 25A to 25F, which may be displayed, e.g., via GUI 210 and / or a mobile application, to permit visualization of the data by a user to allow the user to manually track changes in motor current values over time. For example, FIG. 25A is a table illustrating motor current readings during transition from stow mode to drape mode for a system (e.g., denoted as U1004) including average, max, and integral motor current values for the four motors of the robot arms of the system, e.g., M1-M4. FIG. 25B is a table illustrating motor current readings during transition from stow mode to drape mode for a system (e.g., denoted as U1003) including average, max, and integral motor current values for the four motors of the robot arms of the system. FIG. 25C is a table illustrating motor current readings during transition from stow mode to drape mode for a system (e.g., denoted as U1003) including average, max, and integral motor current values for the four motors of the robot arms of the system, as well as the duration of the transition from stow mode to drape mode for each robot arm in each instance. FIG. 25D is a chart illustrating exemplary average motor current readings for the four motors of the robot arms of a system (e.g., denoted as U1003) during transition from stow mode to drape mode over time. FIG. 25E is a table illustrating the positions of the bases / stages and joints of the robot arms for predefined starting positions, e.g., a preset713732261 v3 110225887-081001 configuration, stow mode, and compact mode, such that a user may verify if the current base / stage positions and robot arm joint positions are close to the expected configured values. FIG. 25F is a table illustrating exemplary motor current readings of a system during transition from compact mode to drape mode during an automated setup sequence of the system workflow including average, maximum, minimum, and integral motor current values for the four motors of the robot arms of the system, as well as the duration of the transition from stow mode to drape mode for each robot arm in each instance.

[0249] In addition, predictive maintenance module 848 may monitor the motor current readings to track user interaction during the co-manipulation mode. For example, as described above, force detection module 820 may sense forces applied to the robot arm and motor interface module 824 may apply force / torque to the joints of the robot arm to thereby ensure transparency of user applied motion, and thus, by tracking motor current, predictive maintenance module 848 may estimate how much effort the system is having to apply to move the robot arms through its range of motion, thereby providing an understanding of how transparent the system feels to the user in the co-manipulation mode. Thus, if over time, the system requires an increasing amount of effort to move through its range of motion and becomes less transparent to the user, e.g., by comparing the trend to a baseline value, predictive maintenance module 848 may generate an alert such that the robot arm may be replaced via early intervention to ensure the ease of use of the system.

[0250] Regarding the tool sensor fingerprint, as described above, the system preferably may include at least two tool sensors, e.g., Hall effect sensors, for redundancy and for detecting when the coupler body, e.g., coupler body 500, is coupled to the coupler interface, e.g., coupler interface 400, and when a surgical instrument is coupled to the coupler body when the coupler body is coupled to the coupler interface. Accordingly, predictive maintenance module 848 may track key tool events such as attachment and detachment of the coupler body to the coupler interface. For example, predictive maintenance module 848 may analyze the tool sensor values for each robot arm and compare them against established threshold ranges specific to each event type to identify anomalies and generate corresponding alerts. Predictive maintenance module 848 may generate an alert configured to notify users when deviations exceed set event detection713732261 v3 111225887-081001 thresholds, enabling timely corrective actions and ensuring optimal tool sensing performance across the system.

[0251] Moreover, predictive maintenance module 848 may aggregate the collected tool sensor values into tables and / or charts, as shown in FIGS. 25Gto 25K, which may be displayed, e.g., via GUI 210 and / or a mobile application, to permit visualization of the data by a user to allow the user to manually track variations in tool sensor values across different tool events and robot arms over time. As shown in FIGS. 25 G to 25K, the average of the tool sensor values for both redundant tool sensors may be displayed and / or the tool sensor values for each redundant tool sensor may be displayed (e.g., both combined as “tool id” and individually as “tool_sensor_l” and “tool_sensor_2”). By examining these charts, the user may identify deviations from expected threshold values, such as robot arms with larger standard deviations indicating poorer condition. Additionally, as described above, the system may include a plurality of encoders, e.g., encoders E7, that measure angulation between distal wrist link 316 and surgical instrument coupler interface 400 at joint 328 along Q7. Accordingly, visualizations of tool sensor values at specific Q7 angles may help contextualize tool sensor values, thereby aiding in the adjustment of event detection thresholds for improved accuracy.

[0252] For example, FIG. 25G is a table illustrating exemplary average tool sensor values of both tool sensors of each robot arm of the system when only the coupler body (e.g., 5 mm or 10 mm coupler body) is attached to the coupler interface, as well as when the coupler body (e.g., 5 mm or 10 mm coupler body) is attached to the coupler interface and the surgical instrument / tool is attached to the coupler body, including the mean, standard deviation, minimum, and maximum tool sensor values, as well as the tool event number and coupler body size. FIG. 25H is a table illustrating exemplary tool sensor values of a single tool sensor for each robot arm of the system when only the coupler body (e.g., 5 mm or 10 mm coupler body) is attached to the coupler interface, as well as when the coupler body (e.g., 5 mm or 10 mm coupler body) is attached to the coupler interface and the surgical instrument / tool is attached to the coupler body, including the mean and standard deviation tool sensor values, as well as the tool event number and coupler body size. FIG. 251 is a chart illustrating exemplary average tool sensor values of both tool sensors of a single robot arm of the system when only the coupler body (e.g., 5 mm or 10 mm coupler body) is attached to the coupler interface, as well as when the coupler body (e.g., 5 mm713732261 v3 112225887-081001 or 10 mm coupler body) is attached to the coupler interface and the surgical instrument / tool is attached to the coupler body over time. FIG. 25 J is a chart illustrating exemplary tool sensor values of a robot arm when a 5 mm coupler body is attached to the coupler interface and the surgical instrument / tool is attached to the coupler body for the same events, including the mean and standard deviation tool sensor values. FIG. 25K is a table illustrating exemplary average tool sensor values of both tool sensors of a robot arm when only the coupler body (e.g., 5 mm or 10 mm coupler body) is attached to the coupler interface, as well as when the coupler body (e.g., 5 mm or 10 mm coupler body) is attached to the coupler interface and the surgical instrument / tool is attached to the coupler body, including the mean, standard deviation, minimum, and maximum tool sensor values, as well as the tool event number and coupler body size, which may be used to establish / adjust event detection thresholds.

[0253] As described above, each encoder E7 may include two encoders for redundancy to provide more accurate position data and facilitate detection of a fault condition, e.g., when the readings between an encoder and a redundant encoder differs. Similarly, each setup actuator 336 of each collar 330 (e.g., the collar button) may include two underlying buttons for redundancy. Accordingly, regarding the collar button fingerprint, predictive maintenance module 848 may monitor collar button performance by analyzing metrics indicative of usage patterns and reported faults collected through continuous data logging from each button, e.g., button presses, error rates, and usage frequency, to thereby identify trends and generate an alert if the predefined thresholds based on historical data are exceeded. For example, detection of sudden spikes in faults or discrepancies between robot arms, e.g., one robot arm’s collar button exhibits more frequent errors than the other robot arm, may be indicative of an anomaly. According...

Claims

1. 225887-081001WHAT IS CLAIMED:

1. A co-manipulation surgical system to assist with a surgical procedure, the comanipulation surgical system comprising: a robot arm comprising a plurality of links and joints and a distal end configured to be removably coupled to a surgical instrument for performing the surgical procedure; an optical sensor configured to collect image data; and a controller operatively coupled to the optical sensor, the controller having instructions that, when executed by one or more processors of the controller, cause the controller to: receive information indicative of a type of the surgical procedure to be performed; receive image data collected by the optical sensor, the image data indicative of a surgical site on a patient’s body; identify a location of one or more trocar ports disposed within the surgical site; identify an optimal robot arm configuration based on the location of the one or more trocar ports and the type of the surgical procedure, the optimal robot arm configuration comprising an optimal position of the distal end of the robot arm relative to the one or more trocar ports; and adjust the robot arm via the plurality of links and joints to the optimal robot arm configuration.

2. The co-manipulation surgical system of claim 1, wherein the controller is configured to execute a machine learning algorithm to identify the optimal robot arm configuration, the machine learning algorithm trained with a dataset of previously identified optimal robot arm configurations relative to one or more previously identified trocar ports for a same type of surgical procedure as the surgical procedure.

3. The co-manipulation surgical system of claim 1, wherein the controller is configured to: generate a surgical site heat map comprising one or more previously identified trocar ports for a same type of surgical procedure as the surgical procedure; and cause a display to display the surgical site heat map. 3732261 v3 127225887-0810014. The co-manipulation surgical system of claim 3, wherein the controller is configured to: calculate an optimal trocar port placement location based on an average center of the one or more previously identified trocar ports for the same type of surgical procedure as the surgical procedure; and generate a recommendation comprising the optimal trocar port placement location.

5. The co-manipulation surgical system of claim 1, wherein the controller is configured to determine an optimal trocar port of the one or more trocar ports for the surgical procedure based on a type of the surgical instrument.

6. The co-manipulation surgical system of claim 1, wherein the controller is configured to: identify a patient bed as a reference point, the patient’s body disposed on the patient bed; and execute one or more algorithms to delineate the surgical site.

7. The co-manipulation surgical system of claim 6, wherein the controller is configured to: determine an angle of the patient bed; and automatically adjust the robot arm via the plurality of links and joints to the optimal robot arm configuration based on the angle of the patient bed.

8. The co-manipulation surgical system of claim 6, wherein the controller is configured to: determine a height of the patient bed; and automatically adjust the robot arm via the plurality of links and joints to the optimal robot arm configuration based on the height of the patient bed.

9. The co-manipulation surgical system of claim 1, wherein the controller is configured to: 3732261 v3 128225887-081001 identify a boundary of the surgical site; and identify the location of the one or more trocar ports disposed within the boundary of the surgical site.

10. The co-manipulation surgical system of claim 1, wherein the optimal robot arm configuration is configured to maximize a workspace of the robot arm relative to surgical site.

11. The co-manipulation surgical system of claim 1, wherein the controller is configured to automatically adjust the robot arm via the plurality of links and joints to the optimal robot arm configuration responsive to user input.

12. The co-manipulation surgical system of claim 1, wherein the optimal robot arm configuration comprises an optimal position of the plurality of links and joints of the robot arm relative to the one or more trocar ports.

13. A co-manipulation surgical system comprising: a surgical platform comprising a plurality of wheels configured to permit mobility of the surgical platform; an optical sensor mounted on the surgical platform and configured to collect image data; and a controller operatively coupled to optical sensor, the controller having instructions that, when executed by one or more processors of the controller, cause the controller to: receive image data collected by the optical sensor, the image data indicative of an operating room environment comprising a patient bed and one or more objects within an operating room; generate a 3D reconstruction of the operating room environment based on the image data, the 3D reconstruction comprising graphical representations of the surgical platform and the one or more objects relative to the patient bed within the operating room; cause a graphical user interface to display the 3D reconstruction of the operating room environment; 3732261 v3 129225887-081001 receive user input data indicative of a surgeon preference of an organization of at least one of the surgical platform or the one or more objects relative to the patient bed within the operating room for a surgical procedure; generate a preferred 3D reconstruction of the operating room environment based on the user input data; and cause a display to display the preferred 3D reconstruction of the operating room environment to facilitate setup of the surgical platform and the one or more objects relative to the patient bed within the operating room for the surgical procedure.

14. The co-manipulation surgical system of claim 13, wherein the one or more objects comprise at least one of a camera control unit tower, a surgical table, or surgical tools.

15. The co-manipulation surgical system of claim 13, wherein the user input data comprises a surgeon preference of an organization of one or more persons relative to the patient bed within the operating room, and wherein the preferred 3D reconstruction of the operating room environment generated by the controller comprises graphical representations of the one or more persons relative to the patient bed within the operating room.

16. The co-manipulation surgical system of claim 13, wherein the controller is further configured to save the preferred 3D reconstruction of the operating room environment for the surgical procedure in a surgeon profile associated with the surgeon.

17. The co-manipulation surgical system of claim 13, wherein the controller is further configured to analyze the user input data and generate a warning if the surgeon preference of the organization of the at least one of the surgical platform or the one or more objects relative to the patient bed within the operating room is determined to be suboptimal for the surgical procedure based on at least one of spatial constraints, positioning of equipment, or accessibility issues.

18. The co-manipulation surgical system of claim 13, wherein the controller is further configured to generate a recommendation for an optimal organization of the at least one of the 3732261 v3 130225887-081001 surgical platform or the one or more objects relative to the patient bed within the operating room based on the user input data and at least one of spatial constraints, positioning of equipment, or accessibility issues.

19. The co-manipulation surgical system of claim 18, wherein the controller is further configured to: receive additional user input data indicative of an informed surgeon preference of the organization of the at least one of the surgical platform or the one or more objects relative to the patient bed within the operating room based on the recommendation; and generate an optimal 3D reconstruction of the operating room environment based on the additional user input data.

20. The co-manipulation surgical system of claim 13, wherein the controller is further configured to generate an optimal trajectory for movement of the surgical platform within the operating room by a user during an environmental scan to permit the optical sensor to collect comprehensive image data indicative of the operating room environment.

21. The co-manipulation surgical system of claim 13, further comprising a mobile application operatively coupled to the controller, wherein the controller is configured to cause, via the mobile application, a mobile device comprising the graphical user interface to display the 3D reconstruction of the operating room environment.

22. The co-manipulation surgical system of claim 13, further comprising a mobile application operatively coupled to the controller, wherein the controller is configured to cause, via the mobile application, a mobile device comprising the display to display the preferred 3D reconstruction of the operating room environment.

23. A co-manipulation surgical system to assist with a surgical procedure comprising a setup stage, an intraoperative stage, and a teardown stage, the co-manipulation surgical system comprising: 3732261 v3 131225887-081001 one or more robot arms each comprising a plurality of links and joints and a distal end configured to be removably coupled to a surgical instrument for performing the surgical procedure; and a controller operatively coupled to the one or more robot arms, the controller having instructions that, when executed by one or more processors of the controller, cause the controller to: detect when the co-manipulation surgical system is in at least one of the setup stage, the intraoperative stage, or the teardown stage; determine, upon detection that the co-manipulation surgical system has been in at least one of the setup stage, the intraoperative stage, or the teardown stage, that a surgical procedure has been performed by the co-manipulation surgical system; aggregate a total count of surgical procedures determined to have been performed by the co-manipulation surgical system within a predetermined period; and generate information indicative of the total count of surgical procedures performed by the co-manipulation surgical system within the predetermined period for transmission to a payment system configured to generate an invoice based on the total count.

24. The co-manipulation surgical system of claim 23, wherein the controller is configured to transmit the information indicative of the total count to the payment system at a predetermined frequency.

25. The co-manipulation surgical system of claim 23, wherein the co-manipulation surgical system comprises two robot arms.

26. The co-manipulation surgical system of claim 23, wherein the controller is configured to detect that the co-manipulation surgical system is in the intraoperative stage when the distal end of at least one of the one or more robot arms transitions from being decoupled from the surgical instrument to being coupled to the surgical instrument. 3732261 v3 132225887-08100127. The co-manipulation surgical system of claim 23, wherein the controller is configured to detect that the co-manipulation surgical system is in the teardown stage when the distal end of each of the one or more robot arms coupled to a surgical instrument transitions from being coupled to the surgical instrument to being decoupled from the surgical instrument.

28. The co-manipulation surgical system of claim 23, wherein the distal end of each of the one or more robot arms comprises a coupler interface configured to be removably coupled to a coupler body, the coupler body configured to be removably coupled to the surgical instrument to thereby couple the surgical instrument to the distal end of the respective robot arm.

29. The co-manipulation surgical system of claim 28, wherein the controller is configured to detect that the co-manipulation surgical system is in the setup stage when the coupler interface of at least one of the one or more robot arms transitions from being decoupled from the coupler body to being coupled to the coupler body.

30. The co-manipulation surgical system of claim 28, wherein the controller is configured to detect that the co-manipulation surgical system is in the teardown stage when the coupler interface of each of the one or more robot arms coupled to a coupler body transitions from being coupled to the coupler body to being decoupled from the coupler body.

31. The co-manipulation surgical system of claim 23, further comprising: a graphical user interface operatively coupled to the controller, the graphical user interface comprising a plurality of selectable preset surgical procedure configurations of the one or more robot arms, wherein the controller is configured to detect that the co-manipulation surgical system is in the setup stage or the teardown stage upon selection of at least one of the plurality of selectable preset surgical procedure configurations.

32. The co-manipulation surgical system of claim 31, wherein the plurality of selectable preset surgical procedure configurations comprises a drape mode, 3732261 v3 133225887-081001 wherein the controller is configured to, upon selection of the drape mode, cause the one or more robot arms to transition to a predetermined drape pose via the plurality of links and joints of the one or more robot arms to facilitate draping of the one or more robot arms, and wherein the controller is configured to detect that the co-manipulation surgical system is in the setup stage upon selection of the drape mode.

33. The co-manipulation surgical system of claim 31, further comprising: a platform coupled to a base of each of the one or more robot arms, wherein the plurality of selectable preset surgical procedure configurations comprises a stow mode, wherein the controller is configured to, upon selection of the stow mode, cause the one or more robot arms to transition to a retracted stow pose above the platform via the plurality of links and joints of the one or more robot arms and rotate about the respective base of the one or more robot arms such that the distal end of each of the one or more robot arms extends towards the rear of the platform and the one or more robot arms are within a footprint of the platform in the retracted stow pose above the platform, and wherein the controller is configured to detect that the co-manipulation surgical system is in the teardown stage upon selection of the stow mode.

34. The co-manipulation surgical system of claim 33, wherein the plurality of selectable preset surgical procedure configurations comprises a compact mode, wherein the controller is configured to, upon selection of the compact mode, cause the one or more robot arms to transition to a semi-retracted compact pose via the plurality of links and joints of the one or more robot arms and rotate about the respective base of the one or more robot arms such that each of the one or more robot arms extends away from the platform to facilitate transportation of the co-manipulation surgical system, the semi-retracted compact pose being less retracted than the retracted stow pose, and wherein the controller is configured to detect that the co-manipulation surgical system is in the teardown stage upon selection of the compact mode.

35. The co-manipulation surgical system of claim 23, further comprising: 3732261 v3 134225887-081001 a graphical user interface operatively coupled to the controller, the graphical user interface configured to permit selection of at least one of a surgeon profile or a procedure type, wherein the controller is configured to detect that the co-manipulation surgical system is in the setup stage upon selection of the at least one of the surgeon profile or the procedure type.

36. The co-manipulation surgical system of claim 23, further comprising: an optical sensor configured to collect image data, wherein the controller is configured to detect when the co-manipulation surgical system is in the at least one of the setup stage, the intraoperative stage, or the teardown stage based on the image data and associated timestamps of the image data.

37. The co-manipulation surgical system of claim 23, further comprising: an audio sensor configured to collect audio data, wherein the controller is configured to execute a speech recognition technique to identify one or more procedural phases of the surgical procedure, and wherein the controller is configured to detect when the co-manipulation surgical system is in the intraoperative stage based on the one or more identified procedural phases.

38. The co-manipulation surgical system of claim 23, wherein the controller is configured to determine that the surgical procedure has been performed by the co-manipulation surgical system upon detection of a transition from at least one of the setup stage to the intraoperative stage or the intraoperative stage to the teardown stage.

39. The co-manipulation surgical system of claim 23, wherein at least one of the one or more processors comprises Cloud-based software.

40. The co-manipulation surgical system of claim 23, wherein the controller is configured to transmit, via a Field Service Action, the information indicative of the total count for upload to a Cloud server. 3732261 v3 135225887-08100141. A co-manipulation surgical system to assist with a surgical procedure, the comanipulation surgical system comprising: a robot arm comprising a plurality of links and joints and a distal end configured to be removably coupled to a scope configured to collect image data; an optical sensor configured to collect image data; and a controller operatively coupled to the scope and the optical sensor, the controller having instructions that, when executed by one or more processors of the controller, cause the controller to: receive image data collected by the optical sensor indicative of an external surgical site of the surgical procedure; receive image data collected by the scope indicative of an internal surgical site of the surgical procedure within a patient; execute an object segmentation algorithm to detect one or more predefined surgical items within the external and internal surgical sites based on the image data collected by the optical sensor and the scope, respectively; track, for each detected predefined surgical item, a number of times the detected predefined surgical item enters the internal surgical site from the external surgical site and a number of times the detected predefined surgical item is removed from the internal surgical site to the external surgical site; and generate, when the surgical procedure is completed, an alert if the number of times the detected predefined surgical item is removed from the internal surgical site to the external surgical site is less than the number of times the detected predefined surgical item enters the internal surgical site from the external surgical site.

42. The co-manipulation surgical system of claim 41, wherein the one or more predefined surgical items comprises at least one of a sponge, a gauze, or a needle.

43. The co-manipulation surgical system of claim 41, further comprising: a platform coupled to a base of the robot arm, wherein the optical sensor is mounted on the platform. 3732261 v3 136225887-08100144. The co-manipulation surgical system of claim 41, wherein the controller is configured to: determine one or more procedural phases of the surgical procedure based on the image data collected by at least one of the optical sensor or the scope; and determine when the surgical procedure is completed based on the one or more procedural phases of the surgical procedure.

45. The co-manipulation surgical system of claim 41, wherein the controller is configured to identify, if the number of times the detected predefined surgical item is removed from the internal surgical site to the external surgical site is less than the number of times the detected predefined surgical item enters the internal surgical site from the external surgical site, a timestamp of the image data associated with when the detected predefined surgical item was last detected within the internal surgical site.

46. The co-manipulation surgical system of claim 45, wherein the controller is further configured to cause a display to display the image data associated with the identified timestamp.

47. A co-manipulation surgical system to assist with a surgical procedure, the comanipulation surgical system comprising: a robot arm comprising a plurality of links and joints and a distal end configured to be removably coupled to a surgical instrument for performing the surgical procedure; an optical sensor configured to collect image data; and a controller operatively coupled to the optical sensor, the controller having instructions that, when executed by one or more processors of the controller, cause the controller to: receive the image data collected by the optical sensor indicative of the surgical procedure; execute an object segmentation algorithm to detect one or more predefined surgical instruments and / or one or more predefined anatomical structures associated with the surgical procedure based on the image data; and 3732261 v3 137225887-081001 determine a surgical phase of the surgical procedure based on the one or more predefined surgical instruments and / or one or more predefined anatomical structures.

48. The co-manipulation surgical system of claim 47, wherein the controller is configured to execute a machine learning algorithm to determine the surgical phase of the surgical procedure based on the one or more predefined surgical instruments and / or one or more predefined anatomical structures, the machine learning algorithm trained via a dataset of known surgical phases for a plurality of surgical procedures comprising the one or more predefined surgical instruments and / or one or more predefined anatomical structures.

49. The co-manipulation surgical system of claim 47, further comprising one or more microphones configured to receive audio data.

50. The co-manipulation surgical system of claim 49, wherein the controller is configured to: receive audio data from the one or more microphones, the audio data indicative of the surgical phase of the surgical procedure; and determine a surgical phase of the surgical procedure based at least partially on the audio data.

51. The co-manipulation surgical system of claim 50, wherein the audio data is indicative of a verbal description by a user of the surgical phase of the surgical procedure, and wherein the controller is configured to record the determined surgical phase and known kinematics of the robot arm during the determined surgical phase to a surgeon profile associated with the user.

52. The co-manipulation surgical system of claim 51, wherein the controller is configured to prompt a user to provide the verbal description, the verbal description comprising information of when the user transitions from one surgical phase of the surgical procedure to another surgical phase of the surgical procedure. 3732261 v3 138225887-08100153. The co-manipulation surgical system of claim 51, wherein the surgeon profile is configured to be uploaded to another co-manipulation surgical system, such that the user may access data recorded to the surgeon profile via the another co-manipulation surgical system.

54. The co-manipulation surgical system of claim 51, wherein the controller is configured to execute a machine learning algorithm to determine the surgical phase of the surgical procedure based on the one or more predefined surgical instruments, the one or more predefined anatomical structures, and / or known kinematics of the robot arm during the surgical procedure, the machine learning algorithm trained at least partially via data recorded to the surgeon profile.

55. The co-manipulation surgical system of claim 47, wherein the controller is configured to determine the surgical phase of the surgical procedure based at least partially on known kinematics of the robot arm during the surgical procedure.

56. The co-manipulation surgical system of claim 47, wherein the controller is configured to automatically adjust a configuration of the robot arm based on the surgical phase of the surgical procedure.

57. The co-manipulation surgical system of claim 47, the controller is configured to cause, when the robot arm is removably coupled to a scope in an instrument centering mode, the robot arm via the plurality of links and joints to automatically track a surgical instrument within a field of view of the scope based on the surgical phase of the surgical procedure.

58. The co-manipulation surgical system of claim 47, the controller is configured to cause, when the robot arm is removably coupled to a scope in an instrument centering mode, the robot arm via the plurality of links and joints to automatically track an anatomical structure within a field of view of the scope based on the surgical phase of the surgical procedure. 3732261 v3 139225887-08100159. The co-manipulation surgical system of claim 47, the controller is configured to identify a predefined surgical task based on the one or more predefined surgical instruments and / or one or more predefined anatomical structures.

60. The co-manipulation surgical system of claim 59, wherein the predefined surgical task comprises grasping, retracting, or cutting.

61. The co-manipulation surgical system of claim 47, wherein the controller is configured to: estimate a surgical procedure end time based on the surgical phase of the surgical procedure; and communicate the estimated surgical procedure end time to operating room staff to facilitate preparation of a subsequent surgical procedure.

62. A co-manipulation surgical system to assist with a surgical procedure, the comanipulation surgical system comprising: a robot arm comprising a plurality of links and joints and a distal end configured to be removably coupled to a surgical instrument for performing the surgical procedure; an optical sensor configured to collect image data; and a controller operatively coupled to the optical sensor, the controller having instructions that, when executed by one or more processors of the controller, cause the controller to: receive the image data collected by the optical sensor, the image data indicative of a surgical scene during the surgical procedure; generate an interactive virtual 3D reconstruction of the surgical scene during the surgical procedure based on the image data, the interactive virtual 3D reconstruction comprising graphical representations of the robot arm and one or more objects or persons associated with performing the surgical procedure; and permit a user to remotely interact with the interactive virtual 3D reconstruction of the surgical scene via a virtual display device operatively coupled to the controller. 3732261 v3 140225887-08100163. A co-manipulation surgical system to assist with a surgical procedure, the comanipulation surgical system comprising: a robot arm comprising a plurality of links and joints and a distal end configured to be removably coupled to a surgical instrument for performing the surgical procedure on a patient; and a controller operatively coupled to the robot arm, the controller having instructions that, when executed by one or more processors of the controller, cause the controller to: calculate a force applied by the surgical instrument to one or more tissues of the patient during the surgical procedure; generate a graphical representation indicative of a magnitude of the force applied by the surgical instrument to one or more tissues of the patient during the surgical procedure; and cause a display to display the graphical representation in real-time during the surgical procedure.

64. The co-manipulation surgical system of claim 63, wherein the controller is configured to: receive a video feed from a scope operatively coupled to the controller, the video feed indicative of a surgical site of the surgical procedure.

65. The co-manipulation surgical system of claim 64, wherein the controller is configured to cause the display to display the graphical representation overlaid on the video feed in real-time during the surgical procedure.

66. The co-manipulation surgical system of claim 64, further comprising an optical sensor configured to collect image data, wherein the controller is configured to: receive timestamped image data from the optical sensor indicative of the surgical procedure; synchronize the timestamped image data with the video feed from the scope; and cause a graphical user interface to display the timestamped image data alongside the video feed from the scope. 3732261 v3 141225887-08100167. The co-manipulation surgical system of claim 64, wherein the controller is configured to: receive timestamped telemetry data associated with the robot arm; synchronize the timestamped telemetry data with the video feed from the scope; and cause a graphical user interface to display the timestamped telemetry data alongside the video feed from the scope.

68. The co-manipulation surgical system of claim 64, wherein the controller is configured to: receive timestamped data indicative of kinematics of the robot arm during the surgical procedure; generate a timestamped 3D reconstruction of the robot arm during the surgical procedure; synchronize the timestamped 3D reconstruction with the video feed from the scope; and cause a graphical user interface to display the timestamped 3D reconstruction of the robot arm alongside the video feed from the scope.

69. The co-manipulation surgical system of claim 68, wherein the controller is configured to: generate a graphical representation of a force profile indicative of the force applied by the surgical instrument to one or more tissues of the patient during the surgical procedure; synchronize the timestamped 3D reconstruction and the video feed from the scope with the force profile; and cause the graphical user interface to display the graphical representation of the force profile alongside the timestamped 3D reconstruction of the robot arm alongside the video feed from the scope.

70. The co-manipulation surgical system of claim 63, wherein the controller is configured to generate an alert if the force applied by the surgical instrument to one or more tissues of the patient during the surgical procedure exceeds a predetermined threshold. 3732261 v3 142225887-08100171. A co-manipulation surgical system to assist with a surgical procedure, the comanipulation surgical system comprising: a robot arm comprising a plurality of links and joints and a distal end configured to be removably coupled to a surgical instrument for performing the surgical procedure; one or more microphones configured to receive audio data; and a controller operatively coupled to the microphones, the controller having instructions that, when executed by one or more processors of the controller, cause the controller to: receive audio data from the one or more microphones, the audio data indicative of a user command; and pass the audio data into an algorithm configured to identify the user command and generate data indicative of a response to the user command.

72. The co-manipulation surgical system of claim 71, wherein the response to the user command comprises an audio response, and wherein the controller is configured to cause one or more speakers to emit an audio response corresponding to the response to the user command.

73. The co-manipulation surgical system of claim 71, wherein the response to the user command comprises an action to be performed by the robot arm, and wherein the controller is configured to cause the robot arm to move via the plurality of links and joints in accordance with the action.

74. The co-manipulation surgical system of claim 73, wherein the controller is configured to: access a surgeon profile associated with a surgeon, the surgeon profile comprising saved surgeon preferences from previous surgical procedures; and cause the robot arm to move via the plurality of links and joints in accordance with the action and the saved surgeon preferences.

75. The co-manipulation surgical system of claim 71, further comprising: an optical sensor configured to collect image data, 3732261 v3 143225887-081001 wherein the controller is configured to generate data indicative of the response to the user command based at least partially on image data received by the optical sensor.

76. The co-manipulation surgical system of claim 71, wherein the algorithm comprises a large language model.

77. The co-manipulation surgical system of claim 71, wherein the controller is configured to: receive audio data from the one or more microphones indicative of a surgical phase of the surgical procedure; and determine the surgical phase of the surgical procedure based on the audio data.

78. The co-manipulation surgical system of claim 71, wherein the controller is configured to: receive audio data from the one or more microphones indicative of verbal notes by a user; generate textual case notes based on the audio data; and store the textual case notes in a surgeon profile associates with the user.

79. The co-manipulation surgical system of claim 71, further comprising an optical sensor configured to collect image data, wherein the controller is configured to: receive timestamped image data from the optical sensor indicative of a surgical procedure; use computer vision to detect one or more surgical instruments associated with the surgical procedure based on the timestamped image data; and generate textual case notes indicative of a surgical task associated with the one or more surgical instruments during the surgical procedure.

80. A co-manipulation surgical system to assist with a surgical procedure, the comanipulation surgical system comprising: 3732261 v3 144225887-081001 a robot arm comprising a plurality of links, a plurality of joints, a proximal region operatively coupled to a base, and a distal region configured to be removably coupled to the surgical instrument; a plurality of motors operatively coupled to corresponding joints of the plurality of joints; and a controller operatively coupled to the robot arm, the controller having instructions that, when executed by one or more processors of the controller, cause the controller to: receive fingerprint data indicative of an operational fingerprint of the system; establish a baseline value associated with the operational fingerprint, the baseline value corresponding to a healthy operational state of the system; and aggregate the fingerprint data over time, wherein an inspection or maintenance service is predicted to be required if the fingerprint data deviates from the baseline value by more than a predetermined threshold value.

81. The co-manipulation surgical system of claim 80, wherein the controller is configured to establish the baseline value based on manufacture and / or installation baseline data associated with the operational fingerprint of the system.

82. The co-manipulation surgical system of claim 80, wherein the predetermined threshold value is selected such that fingerprint data that deviates from the baseline value by more than the predetermined threshold value indicates that the system is closer to an unhealthy operational state than the healthy operational state.

83. The co-manipulation surgical system of claim 80, wherein the fingerprint data comprises telemetry data, 3D depth data, color image data, audio data, laparoscopic video data, and / or application event logs.

84. The co-manipulation surgical system of claim 80, wherein the controller is configured to automatically transition, upon selection of a second preset configuration, the robot arm from a first preset configuration to the second preset configuration via the plurality of motors, the second preset configuration different from the first preset configuration. 3732261 v3 145225887-08100185. The co-manipulation surgical system of claim 84, wherein the fingerprint data comprises a motor current reading across a motor of the plurality of motors as the robot arm transitions from the first preset configuration to the second preset configuration, and wherein the baseline value comprises a baseline motor current reading across the motor as the robot arm transitions from the first preset configuration to the second preset configuration under normal operating conditions.

86. The co-manipulation surgical system of claim 85, wherein the controller is configured to calculate an average, maximum, and / or integral value of the motor current reading over time, and wherein the inspection or maintenance service is predicted to be required if the average, maximum, and / or integral value of the motor current reading deviates from the baseline value by more than the predetermined threshold value.

87. The co-manipulation surgical system of claim 85, wherein motor current readings that deviate from the baseline value by more than the predetermined threshold value indicates that the system requires an increasing amount of effort to transition the robot arm from the first preset configuration to the second preset configuration.

88. The co-manipulation surgical system of claim 85, wherein the first and second preset configurations are selectable from a plurality of preset configurations, the plurality of preset configurations comprises a compact configuration, a stow configuration, and / or a drape configuration.

89. The co-manipulation surgical system of claim 84, wherein the fingerprint data comprises a motor voltage reading across a motor of the plurality of motors as the robot arm transitions from the first preset configuration to the second preset configuration, and wherein the baseline value comprises a baseline motor voltage reading across the motor as the robot arm transitions from the first preset configuration to the second preset configuration under normal operating conditions. 3732261 v3 146225887-08100190. The co-manipulation surgical system of claim 80, wherein the fingerprint data comprises a tool sensor value indicative of whether the surgical instrument is removably coupled to the distal region of the robot arm, and wherein the baseline value comprises a baseline tool sensor value indicative of whether the surgical instrument is removably coupled to the distal region of the robot arm under normal operating conditions.

91. The co-manipulation surgical system of claim 90, wherein the surgical instrument is configured to be removably coupled to the distal region of the robot arm via a coupler body configured to be removably coupled to the surgical instrument and removably coupled to a coupler interface at the distal region of the robot arm, wherein the tool sensor value is indicative of whether the coupler body is removably coupled to the coupler interface when the surgical instrument is removably coupled to the coupler body and whether the coupler body is removably coupled to the coupler interface when the surgical instrument is not removably coupled to the coupler body, and wherein the baseline tool sensor value is indicative of whether the coupler body is removably coupled to the coupler interface when the surgical instrument is removably coupled to the coupler body under normal operating conditions and whether the coupler body is removably coupled to the coupler interface when the surgical instrument is not removably coupled to the coupler body under normal operating conditions.

92. The co-manipulation surgical system of claim 90, wherein the controller is configured to calculate an average, minimum, maximum, and / or standard deviation value of the tool sensor value, and wherein the inspection or maintenance service is predicted to be required if the average, minimum, maximum, and / or standard deviation value of the tool sensor value deviates from the baseline value by more than the predetermined threshold value. 3732261 v3 147225887-08100193. The co-manipulation surgical system of claim 90, wherein the tool sensor value comprises a first tool sensor value generated by a first tool sensor and a second tool sensor value generated by a second tool sensor.

94. The co-manipulation surgical system of claim 90, wherein the controller is configured to receive data indicative of angulation between the surgical instrument and a distal- most link of the plurality of links of the robot arm, the angulation configured to facilitate contextualization of the tool sensor value.

95. The co-manipulation surgical system of claim 80, wherein the controller is configured to enable a function of the system upon manual actuation of an actuator by a user, wherein the fingerprint data comprises data indicative of performance of the actuator, and wherein the baseline value comprises a baseline value indicative of performance of the actuator under normal operating conditions.

96. The co-manipulation surgical system of claim 95, wherein the actuator comprises first and second underlying actuators, each configured to be actuated via actuation of the actuator, and wherein the data indicative of performance of the actuator comprises data indicative of inconsistencies between performance of the first underlying actuator and performance of the second underlying actuator upon manual actuation of the actuator by the user.

97. The co-manipulation surgical system of claim 96, wherein the data indicative of inconsistencies between performance of the first underlying actuator and performance of the second underlying actuator comprises an error rate of the first and second underlying actuators, and wherein the inspection or maintenance service is predicted to be required if the error rate of the first underlying actuator deviates from the error rate of the second underlying actuator by more than the predetermined threshold value. 3732261 v3 148225887-08100198. The co-manipulation surgical system of claim 96, wherein performance of the first and second underlying actuators comprises a duration of the actuation of the first and second underlying actuators in response to the manual actuation of the actuator by the user.

99. The co-manipulation surgical system of claim 95, wherein the data indicative of performance of the actuator comprises a current total amount of travel of the actuator responsive to actuation by the user over a current life of the actuator, wherein the baseline value comprises an expected total amount of travel of the actuator over an expected lifespan of the actuator under normal operating conditions, and wherein the inspection or maintenance service is predicted to be required if the current total amount of travel of the actuator approaches the expected total amount of travel of the actuator by more than a predetermined threshold.

100. The co-manipulation surgical system of claim 95, wherein the actuator is operatively coupled to the robot arm, the system further comprising a second robot arm operatively coupled to a second actuator configured to be actuated to enable a second function of the system, and wherein the fingerprint data comprises data indicative of inconsistencies between performance of the actuator and performance of the second actuator.

101. The co-manipulation surgical system of claim 100, wherein the data indicative of inconsistencies between performance of the actuator and performance of the second actuator comprises an error rate of the actuator and the second actuator, and wherein the inspection or maintenance service is predicted to be required if the error rate of the actuator deviates from the error rate of the second actuator by more than the predetermined threshold value.

102. The co-manipulation surgical system of claim 80, wherein the controller is configured to generate an alert if a proximity sensor of the system switches from an inactive state to an active state, 3732261 v3 149225887-081001 wherein the fingerprint data comprises data indicative of when the proximity sensor of the system switches between the active state and the inactive state within a predetermined time period, and wherein the alert is determined to be a false alert if the proximity sensor does not switch from the active state back to the inactive state within the predetermined time period or if an amount of times that the proximity sensor switches from the inactive state to the active state within the predetermined time period exceeds a predetermined threshold.

103. The co-manipulation surgical system of claim 102, wherein the predetermined time period begins when the proximity sensor initially switches from the inactive state to the active state.

104. The co-manipulation surgical system of claim 102, wherein the controller is configured to adjust the predetermined time period based on a time period that the proximity sensor remains in the active state and / or the amount of times that the proximity sensor switches from the inactive state to the active state within the predetermined time period.

105. The co-manipulation surgical system of claim 80, wherein the controller is configured to generate an alert if a braking mechanism of the system is in a disengaged state for a time period that exceeds a predetermined time period, wherein the fingerprint data comprises data indicative of when the braking mechanism switches between an engaged state and the disengaged state, and wherein the alert is determined to be a false alert if the braking mechanism switches from the disengaged state to the engaged state back to the disengaged state and back to the engaged state within a predetermined time period threshold.

106. The co-manipulation surgical system of claim 80, wherein the controller is configured to detect when operation of the system by a user deviates from a recommended system workflow. 3732261 v3 150225887-081001107. The co-manipulation surgical system of claim 106, wherein the controller is configured to automatically transition, upon selection of a preset configuration, the robot arm to the preset configuration via the plurality of motors, and wherein operation of the system by the user deviates from the recommended system workflow when the transition of the robot arm to the preset configuration is interrupted.

108. The co-manipulation surgical system of claim 107, wherein the preset configuration comprises a retracted configuration of the robot arm, and wherein operation of the system by the user deviates from the recommended system workflow when the preset configuration is not selected by the user prior to shutdown of the system.

109. The co-manipulation surgical system of claim 106, wherein the controller is configured to generate an alert if a number of times that the operation of the system by the user deviates from the recommended system workflow exceeds a predetermined threshold.

110. The co-manipulation surgical system of claim 80, further comprising a stage assembly configured to move the base of the robot arm in one or more degrees of freedom.

111. The co-manipulation surgical system of claim 110, wherein the controller is configured to: detect when movement of the base in the one or more degrees of freedom via the stage assembly exceeds a predetermined movement threshold; and adjust the predetermined movement threshold based on an amount of times that the movement of the base in the one or more degrees of freedom via the stage assembly exceeds the predetermined movement threshold.

112. The co-manipulation surgical system of claim 110, wherein the fingerprint data comprises a current total amount of travel of the stage assembly in the one or more degrees of freedom over a current life of the stage assembly, 3732261 v3 151225887-081001 wherein the baseline value comprises an expected total amount of travel of the stage assembly in the one or more degrees of freedom over an expected lifespan of the stage assembly under normal operating conditions, and wherein the inspection or maintenance service is predicted to be required if the current total amount of travel of the stage assembly in the one or more degrees of freedom approaches the expected total amount of travel of the stage assembly in the one or more degrees of freedom by more than a predetermined threshold.

113. The co-manipulation surgical system of claim 80, wherein the fingerprint data comprises a temperature of the system, and wherein the baseline value comprises a baseline temperature of the system under normal operating conditions.

114. The co-manipulation surgical system of claim 80, wherein the controller is configured to: detect an occurrence of a fault condition of the system; determine an amount of time between the occurrence of the fault condition and a resolution of the fault condition; and determine a total number of the occurrence of the fault condition over a current life of the system.

115. The co-manipulation surgical system of claim 114, wherein the fingerprint data comprises a frequency of the occurrence of the fault condition, and wherein the baseline value comprises a baseline frequency value of the occurrence of the fault condition under normal operating conditions.

116. The co-manipulation surgical system of claim 80, wherein the controller is configured to: execute a predictive maintenance algorithm to determine if the fingerprint data deviates from the baseline value by more than the predetermined threshold value; and 3732261 v3 152225887-081001 generate an alert if the fingerprint data deviates from the baseline value by more than the predetermined threshold value, the alert indicative that the inspection or maintenance service is required.

117. The co-manipulation surgical system of claim 116, wherein the controller is configured to analyze the fingerprint data over time to identify one or more patterns associated with the operational fingerprint, and wherein the predictive maintenance algorithm is configured to compare the one or more patterns to the baseline value to determine if the fingerprint data deviates from the baseline value by more than the predetermined threshold value.

118. The co-manipulation surgical system of claim 116, wherein the predictive maintenance algorithm comprises a machine learning algorithm configured to perform predictive analysis, the machine learning algorithm trained with historical data indicative of the operational fingerprints from previous surgical procedures.

119. The co-manipulation surgical system of claim 116, wherein the controller is configured to cause a display to display the alert.

120. The co-manipulation surgical system of claim 80, wherein the controller is configured to: generate a graphical representation of the aggregated fingerprint data over time and the baseline value; and cause a display to display the graphical representation, wherein the inspection or maintenance service is determined to be required based on the displayed graphical representation.

121. The co-manipulation surgical system of claim 120, further comprising a graphical user interface operatively coupled to the controller, the graphical user interface comprising the display. 3732261 v3 153225887-081001122. The co-manipulation surgical system of claim 121, wherein the graphical user interface comprises a mobile device comprising a mobile application configured to display the graphical representation.

123. A co-manipulation surgical system to assist with a surgical procedure, the comanipulation surgical system comprising: a robot arm comprising a plurality of links and joints and a distal end configured to be removably coupled to a surgical instrument for performing the surgical procedure; an optical sensor configured to collect image and / or depth data; and a controller operatively coupled to the optical sensor, the controller having instructions that, when executed by one or more processors of the controller, cause the controller to: receive the image and / or depth data collected by the optical sensor, the image and / or depth data indicative of at least one object or person within a field of view of the optical sensor; identify the at least one object or person within the field of view of the optical sensor based on the image and / or depth data; determine whether the co-manipulation surgical system is within a predefined authorized clinical zone based on the identified at least one object or person; and automatically provide, if the co-manipulation surgical system is within the predefined authorized clinical zone, a level of authorization for access to the co-manipulation surgical system based on the identified at least one object or person.

124. The co-manipulation surgical system of claim 123, wherein the controller is configured to perform at least one of image classification, objection detection, depth estimation, or temporal analysis to identify the at least one object or person within the field of view of the optical sensor based on the image and / or depth data.

125. The co-manipulation surgical system of claim 123, wherein the predefined authorized clinical zone comprises an operating room, and wherein the controller is configured to determine that the co-manipulation surgical system is within the operating room if the identified object comprises capital equipment associated with the operating room. 3732261 v3 154225887-081001126. The co-manipulation surgical system of claim 123, wherein the predefined authorized clinical zone comprises a proximity of a pre-authorized user, and wherein the controller is configured to determine that the co-manipulation surgical system is within the proximity of the pre-authorized user if the identified person comprises a user associated with a predefined level of authorization.

127. The co-manipulation surgical system of claim 126, wherein the controller is configured to: automatically load a user profile associated with the identified person, the user profile comprising the predefined level of authorization associated with the identified person; and automatically provide, if the co-manipulation surgical system is within the proximity of the pre-authorized user, the predefined level of authorization for access to the co-manipulation surgical system.

128. The co-manipulation surgical system of claim 123, wherein the controller is configured to restrict access to the co-manipulation surgical system if the co-manipulation surgical system is not within the predefined authorized clinical zone.

129. The co-manipulation surgical system of claim 128, further comprising: a graphical user interface operatively coupled to the controller, wherein the controller is configured to cause the graphical user interface to display an alert indicative of the restricted access if the co-manipulation surgical system is not within the predefined authorized clinical zone. 3732261 v3 155

Images