Method for arthroscopic surgery video segmentation and apparatus therefor

Abstract

Methods, non-transitory computer-readable media, and arthroscopic video segmentation devices and systems are disclosed that facilitate improved automated segmentation analysis of video of arthroscopic procedures. With this technology, a machine learning model can be used to automatically segment a video feed of an arthroscopic surgery, and one or more tags related to the segment can be associated with the video feed. The generated video can be output in real-time to provide segmented information related to the surgical procedure, or can be saved with one or more segments tagged for playback for training or informational purposes.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 004,585, filed April 3, 2020, the entire contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] This disclosure generally relates to methods, systems, and apparatuses related to the segmentation or tagging of arthroscopic surgical videos, and more specifically, to methods and apparatuses for improving the processing of arthroscopic surgical video data to provide automatic segmentation of the video so that surgeons can easily navigate in the final recorded video. Background Technology

[0004] Arthroscopic surgery is useful in the treatment of many joint-related pathologies. At the heart of arthroscopy is the visualization platform, as it provides a window into the joint. For a skilled surgeon, rich information can be extracted from individual images. Even more information can be extracted when these images are combined into video sequences. Exemplary features that a skilled surgeon can identify from video sequences of arthroscopic surgery include: identification of anatomical structures when only a portion of the macroscopic structure is observed, pathology associated with specific anatomical structures, and approximate values ​​of measurements related to the field of view.

[0005] While some surgeons can record video feeds of the entire surgery, such videos are tedious because surgeons may only be interested in viewing certain portions of the video. Therefore, useful segmentation that occurs during the surgical procedure is necessary. Currently, segmentation is provided by "in-circle surgeon" systems, where surgeons manually start and stop recording based on certain activities they wish to record. This requires surgeon input and can potentially distract the surgeon from their immediate task. In other systems, surgical video can be processed, and surgeons can add manual triggers to label or tag specific video frames. Tagging video frames during surgery requires manual intervention from the surgeon. Alternatively, postoperative analysis requires additional time from the surgeon.

[0006] Another problem with the current system involves surgeons accessing specific video segments after surgery for review or to send to colleagues or patients. If the entire surgery is recorded continuously, the surgeon can quickly scan the entire procedure via video playback, but there's a lack of markers indicating when specific activities during the surgery occurred. Alternatively, if a series of video excerpts has been created, for example using a "surgeon in the circle" technique, the videos may be difficult to review quickly and will lack any annotations that provide quick context for searching within the video excerpts.

[0007] What is needed are methods and systems for providing arthroscopic video analysis that allow for automatic segmentation of the video during arthroscopic surgery, enabling surgeons to easily navigate within the final recorded video. Summary of the Invention

[0008] Methods, non-transient computer-readable media, and arthroscopic video segmentation devices and systems that facilitate improved automatic segmentation and analysis of videos used in arthroscopic procedures are disclosed.

[0009] According to some embodiments, a method for automatically segmenting surgical video data is disclosed. The method includes obtaining video data from a camera configured to capture video data of a surgical procedure by a computing device, wherein the video data includes a field of view of an anatomical region of a patient during the surgical procedure. A first video data stream and a second video data stream are generated by the computing device. One or more machine learning models are applied to the second video data stream by an AI processing unit associated with the computing device. The AI ​​processing unit identifies one or more segments of the second video data stream based on the applied machine learning models. A set of labels associated with the one or more segments of the second video data stream is generated by the AI ​​processing unit.

[0010] According to some embodiments, the method further includes the computing device associating the set of tags with the first video data stream. Merged video feed data including one or more tags from the first video data stream is generated by the computing device.

[0011] According to some embodiments, the set of tags is stored in the video metadata of the first video data stream.

[0012] According to some embodiments, the method further includes the computing device outputting the merged video feed data for display on a display interface, wherein the merged video feed data is output in real time.

[0013] According to some embodiments, the surgical procedure includes an arthroscopic surgical procedure.

[0014] According to some embodiments, the camera is located on the endoscope device.

[0015] According to some embodiments, identifying one or more segments of the second video data stream further includes the AI ​​processing unit analyzing each of a plurality of frames in the second video data stream. The AI ​​processing unit determines one or more features present in each of the plurality of video frames in the second video data stream based on an applied machine learning model. The one or more segments of the second video data stream are identified by the AI ​​processing unit based on the identified one or more features present in each of the plurality of video frames in the second video data stream.

[0016] According to some embodiments, one or more features present in each of the plurality of video frames identified include anatomical structures associated with the surgical procedure and one or more of tools, digital markers, digital annotations, or implants associated with the surgical procedure.

[0017] According to some embodiments, identifying one or more segments of the second video data stream further includes the AI ​​processing unit comparing each of the plurality of frames with the previous frame of the plurality of frames based on the existing one or more features. The AI ​​processing unit identifies a state change of one of the one or more features based on the comparison. New segments in the second video data stream are identified by the AI ​​processing unit based on the identified state change.

[0018] According to some embodiments, one or more features present in each of the plurality of video frames identified include anatomical structures associated with the surgical procedure and one or more of tools, digital markers, digital annotations, or implants associated with the surgical procedure.

[0019] According to some embodiments, the state change includes a change in the contact between the anatomical structure and the tools associated with the surgical procedure, or a change in one or more of the tools, the digital markers, the digital annotations, or the implant.

[0020] According to some embodiments, the method further includes downsampling the second video data stream by the computing device before the AI ​​processing unit applies the one or more machine learning models.

[0021] According to some embodiments, a video data segmentation system is disclosed. The video segmentation system includes: a camera configured to capture video data; and a computing device including a first processor coupled to a memory and configured to execute programming instructions stored in the first memory to perform the methods disclosed herein.

[0022] According to some embodiments, the camera is located on the endoscope device.

[0023] According to some embodiments, a non-transient computer-readable medium stores instructions for automatically segmenting surgical video data, the instructions including executable code that, when executed by one or more processors, causes the one or more processors to perform the methods disclosed herein. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with the written description, serve to explain the principles, features, and characteristics of the invention. In the drawings:

[0025] Figure 1 An operating room including an exemplary computer-assisted surgical system (CASS) according to an embodiment is shown.

[0026] Figure 2 An example of an electromagnetic sensor device according to some embodiments is shown.

[0027] Figure 3A Alternative examples of electromagnetic sensor devices with three vertical coils according to some embodiments are shown.

[0028] Figure 3B Alternative examples of electromagnetic sensor devices with two non-parallel fixed coils according to some embodiments are shown.

[0029] Figure 3C Alternative examples of electromagnetic sensor devices with two non-parallel, separate coils according to some embodiments are shown.

[0030] Figure 4 Examples of an electromagnetic sensor device and a patient's bone according to some embodiments are shown.

[0031] Figure 5A Illustrative control instructions provided by the surgical computer to other components of CASS according to an embodiment are shown.

[0032] Figure 5B Illustrative control instructions provided by components of a CASS according to an embodiment to a surgical computer are shown.

[0033] Figure 5C An illustrative embodiment is shown in which a surgical computer, according to an embodiment, is connected to a surgical data server via a network.

[0034] Figure 6 A surgical patient care system and illustrative data source according to an embodiment are shown.

[0035] Figure 7A An exemplary flowchart for determining a preoperative surgical plan is shown according to an embodiment.

[0036] Figure 7B An exemplary flowchart is shown for determining the care period, including preoperative, intraoperative, and postoperative actions, according to an embodiment.

[0037] Figure 7C An illustrative graphical user interface according to an embodiment is shown, including images depicting implant placement.

[0038] Figure 8 An environment including an arthroscopic surgery video segmentation system according to an embodiment is shown.

[0039] Figure 9 yes Figure 8 The diagram shows a block diagram of an arthroscopic surgery video segmentation system.

[0040] Figure 10 An example is shown. Figure 8 A more detailed block diagram of the arthroscopic surgery video segmentation system is shown below.

[0041] Figure 11 An example is shown. Figure 8 A block diagram of an exemplary artificial intelligence processing unit in the arthroscopic surgery video segmentation system shown.

[0042] Figure 12 A flowchart of an exemplary method for automating arthroscopic video segmentation using a machine learning model, according to an embodiment, is shown.

[0043] Figure 13 A flowchart illustrating an exemplary method for identifying segments in automated arthroscopic video segmentation according to an embodiment is shown.

[0044] Figure 14 The following is illustrated: a knee joint according to an embodiment, including patient anatomy and in... Figure 12 and 13 The method analyzes an example frame of the video display of the surgical tools.

[0045] Figure 15 An example of a method for using according to an embodiment is shown. Figure 12 and 13 The example static image is a segmented video display of a method for surgical procedures of the knee joint.

[0046] Figure 16 An example of a method for using according to an embodiment is shown. Figure 12 and 13 An example static image of the interface for displaying and playing back segmented video using the method.

[0047] Figure 17 An example of searching based on an embodiment is shown. Figure 12 and 13 An example static image of the interface of a segmented video generated by the method. Detailed Implementation

[0048] This disclosure describes an improved method for arthroscopic video data analysis that correlates analytical information with video feed output. In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of exemplary embodiments. However, it will be apparent to those skilled in the art that embodiments can be practiced without any of these specific details.

[0049] This disclosure is not limited to the specific systems, apparatus, and methods described, as they can vary. The terminology used in the description is for the purpose of describing a particular version or embodiment only and is not intended to be limiting.

[0050] As used herein, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the / described” include plural references. Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. Nothing in this disclosure should be construed as an admission that the embodiments described herein are not entitled to bring forward the date of this disclosure due to prior inventions. As used herein, the term “comprising” means “including, but not limited to,” “including.”

[0051] definition

[0052] For the purposes of this disclosure, the term "implant" is used to refer to a prosthetic device or structure manufactured to replace or enhance a biological structure. For example, in a total hip replacement procedure, a prosthetic acetabular cup (implant) is used to replace or enhance a patient's worn or damaged acetabulum. While the term "implant" is generally considered to refer to an artificial structure (in contrast to a transplant), for the purposes of this specification, an implant may include biological tissue or material transplanted to replace or enhance a biological structure.

[0053] For the purposes of this disclosure, the term "real-time" is used to refer to computations or operations performed immediately upon the occurrence of an event or the receipt of input by an operating system. However, the use of the term "real-time" is not intended to exclude operations that introduce some delay between input and response, provided that the delay is an unintended consequence of the machine's performance characteristics.

[0054] While much of this disclosure refers to surgeons or other medical professionals by specific titles or roles, nothing in this disclosure is intended to be limited to any particular title or function. A surgeon or medical professional can include any physician, nurse, medical professional, or technician. Any of these terms or titles may be used interchangeably with the systems disclosed herein, unless otherwise expressly stated. For example, in some embodiments, the reference to a surgeon may also apply to a technician or nurse.

[0055] The systems, methods, and apparatus disclosed herein are particularly well-suited for use with surgical navigation systems (e.g., Surgical procedures using a surgical navigation system. NAVIO is a registered trademark of BLUE BELTTECHNOLOGIES, Inc., Pittsburgh, Pennsylvania, a subsidiary of SMITH & NEPHEW, Inc., Memphis, Tennessee.

[0056] CASS Ecosystem Overview

[0057] Figure 1 Illustrations of an example computer-assisted surgical system (CASS) 100 according to some embodiments are provided. As described in further detail in the following sections, CASS uses computers, robotics, and imaging technologies to assist surgeons in performing orthopedic surgical procedures such as total knee replacement (TKA) or total hip replacement (THA). For example, surgical navigation systems can help surgeons locate the patient's anatomy, guide surgical instruments, and implant medical devices with high precision. Surgical navigation systems such as CASS 100 often employ various forms of computing technology to perform a wide range of standard and minimally invasive surgical procedures and techniques. Moreover, these systems allow surgeons to more accurately plan, track, and navigate the placement of instruments and implants relative to the patient's body, as well as to perform preoperative and intraoperative body imaging.

[0058] The actuator platform 105 positions surgical instruments relative to the patient during surgery. The exact components of the actuator platform 105 will vary depending on the embodiment employed. For example, for knee surgery, the actuator platform 105 may include an end effector 105B that holds the surgical instruments or apparatus during its use. The end effector 105B may be a handheld device or instrument used by the surgeon (e.g., ...). (Handheld component, cutting guide, or clamp), or alternatively, the end effector 105B may include a device or instrument held or positioned by the robotic arm 105A. Although in Figure 1A single robotic arm 105A is shown, but in some embodiments, multiple devices may be present. For example, there may be one robotic arm 105A on each side of the operating table T, or two devices on one side of the operating table T. The robotic arm 105A may be mounted directly to the operating table T, located on a floor platform (not shown) adjacent to the operating table T, mounted on a floor bar, or mounted on a wall or ceiling of the operating room. The floor platform may be fixed or movable. In one particular embodiment, the robotic arm 105A is mounted on a floor bar located between the patient's legs or feet. In some embodiments, the end effector 105B may include a suture retainer or stapler to aid in wound closure. Furthermore, in the case of two robotic arms 105A, the surgical computer 150 may drive the robotic arms 105A to work together to suture the wound upon closure. Alternatively, the surgical computer 150 may drive one or more robotic arms 105A to suture the wound upon closure.

[0059] The actuator platform 105 may include a limb locator 105C for positioning a patient's limb during surgery. An example of the limb locator 105C is the Smith and Nephew Spider2 system. The limb locator 105C can be manually operated by the surgeon, or alternatively, the limb position can be changed based on instructions received from the surgical computer 150 (described below). Although in Figure 1 A limb locator 105C is shown, but in some embodiments, multiple devices may be present. As an example, there may be one limb locator 105C on each side of the operating table T, or two devices on one side of the operating table T. The limb locator 105C may be mounted directly to the operating table T, located on a floor platform (not shown) next to the operating table T, mounted on a pole, or mounted on the wall or ceiling of the operating room. In some embodiments, the limb locator 105C may be used in unconventional ways, such as as a retractor or a specific bone retainer. As an example, the limb locator 105C may include an ankle boot, soft tissue clip, bone clip, or soft tissue retractor key, such as a hook-shaped, curved, or angled blade. In some embodiments, the limb locator 105C may include a suture retainer to assist in wound closure.

[0060] The actuator platform 105 may include tools such as screwdrivers, light or lasers indicating axes or planes, levels, pin drivers, pin pullers, plane checkers, indicators, fingers, or some combination thereof.

[0061] Resection device 110 ( Figure 1(Not shown) Bone or tissue resection is performed using techniques such as mechanical, ultrasonic, or laser methods. Examples of resection devices 110 include drilling devices, deburring devices, vibratory sawing devices, vibratory impact devices, reamers, ultrasonic bone cutting devices, radiofrequency ablation devices, reciprocating motion devices (e.g., files or broaches), and laser ablation systems. In some embodiments, the resection device 110 is held and operated by a surgeon during surgery. In other embodiments, an actuator platform 105 may be used to hold the resection device 110 during use.

[0062] The actuator platform 105 may also include a cutting guide or clamp 105D for guiding a saw or drill used to remove tissue during surgery. Such a cutting guide 105D may be integrally formed as part of the actuator platform 105 or the robotic arm 105A, or the cutting guide may be a separate structure that can be mateably and / or removably attached to the actuator platform 105 or the robotic arm 105A. The actuator platform 105 or the robotic arm 105A may be controlled by the CASS 100 to position the cutting guide or clamp 105D near the patient's anatomy according to a preoperative or intraoperative surgical plan, such that the cutting guide or clamp will produce precise bone cuts according to the surgical plan.

[0063] Tracking system 115 uses one or more sensors to collect real-time positional data for locating patient anatomy and surgical instruments. For example, for TKA procedures, the tracking system can provide the position and orientation of end effector 105B during the procedure. In addition to positional data, data from tracking system 115 can also be used to infer the velocity / acceleration of the anatomy / instrument, which can be used for tool control. In some embodiments, tracking system 115 can use an array of trackers attached to end effector 105B to determine the position and orientation of end effector 105B. The position of end effector 105B can be inferred based on the position and orientation of tracking system 115 and a known relationship in three-dimensional space between tracking system 115 and end effector 105B. Various types of tracking systems can be used in various embodiments of the invention, including but not limited to infrared (IR) tracking systems, electromagnetic (EM) tracking systems, video or image-based tracking systems, and ultrasound registration and tracking systems. Using the data provided by tracking system 115, surgical computer 150 can detect objects and prevent collisions. For example, surgical computer 150 can prevent robotic arm 105A and / or end effector 105B from colliding with soft tissue.

[0064] Any suitable tracking system can be used to track surgical objects and patient anatomy in the operating room. For example, a combination of infrared and visible light cameras can be used in an array. Various illumination sources (e.g., infrared LED light sources) can illuminate the scene, enabling three-dimensional imaging. In some embodiments, this can include stereo, three-view, four-view, etc., imaging. In addition to camera arrays fixed to a trolley in some embodiments, additional cameras can be placed throughout the operating room. For example, handheld tools or headgear worn by the operator / surgeon can include imaging capabilities that transmit images back to a central processor to correlate those images with those acquired by the camera array. This can provide more robust images for environments modeled using multiple perspectives. Furthermore, some imaging devices can have appropriate resolution on the scene or appropriate viewing angles to pick up information stored in quick response (QR) codes or barcodes. This helps identify specific objects that have not been manually registered with the system. In some embodiments, the cameras can be mounted on a robotic arm 105A.

[0065] As discussed in this paper, while most tracking and / or navigation technologies utilize image-based tracking systems (e.g., IR tracking systems, video or image-based tracking systems, etc.), electromagnetic (EM)-based tracking systems are becoming increasingly common for various reasons. For example, implantation of standard optical trackers requires tissue resection (e.g., down to the cortex) and subsequent drilling and driving of cortical pins. Additionally, because optical trackers require a direct line of sight to the tracking system, placement of such trackers may need to be away from the surgical site to ensure they do not restrict the movement of surgeons or medical professionals.

[0066] Typically, EM-based tracking devices include one or more coils and a reference field generator. The one or more coils can be energized (e.g., via a wired or wireless power source). Once energized, the coils generate an electromagnetic field that can be detected and measured (e.g., by the reference field generator or additional devices) in a manner that allows the position and orientation of the one or more coils to be determined. As will be understood by one of ordinary skill in the art, for example... Figure 2 The single coil shown is limited to detecting five (5) total degrees of freedom (DOF). For example, sensor 200 is able to track / determine movement in the X, Y, or Z directions, as well as rotation about the Y-axis 202 or the Z-axis 201. However, due to the electromagnetic properties of the coil, it is not possible to accurately track rotational motion about the X-axis.

[0067] Therefore, in most electromagnetic tracking applications, such as Figure 3AThe three-coil system shown is used to achieve tracking in all six degrees of freedom (i.e., forward / backward 310°, up / down 320°, left / right 330°, roll 340°, pitch 350°, and yaw 360°) that allow a rigid body to move in three-dimensional space. However, including two additional coils and their 90° offset angle of positioning may require a much larger tracking device. Alternatively, as those skilled in the art will know, fewer than three complete coils can be used to track all 6DOF. In some EM-based tracking devices, two coils can be fixed to each other, for example... Figure 3B As shown in the diagram. Since the two coils 301B and 302B are rigidly fixed to each other, not perfectly parallel, and have known positions relative to each other, this arrangement can be used to determine the sixth degree of freedom 303B.

[0068] While using two fixed coils (e.g., 301B, 302B) allows for EM-based tracking in 6DOF, the sensor device has a much larger diameter than a single coil due to the additional coils. Therefore, practical applications of EM-based tracking systems in surgical settings may require tissue resection and drilling of a portion of the patient's bone to allow insertion of the EM tracker. Alternatively, in some embodiments, a single coil or 5DOF EM tracking device can be implanted / inserted into the patient's bone using only a pin (e.g., without drilling or extensive bone resection).

[0069] Therefore, as described herein, there is a need for a solution that limits the use of the EM tracking system to devices small enough to be inserted / embedded using small-diameter needles or pins (i.e., without requiring new incisions or large-diameter openings in the bone). Thus, in some embodiments, a second 5DOF sensor, not attached to the first sensor and therefore having a small diameter, can be used to track all 6DOF. Now refer to... Figure 3C In some embodiments, two 5DOF EM sensors (e.g., 301C and 302C) may be inserted into the patient (e.g., in the patient's bone) at different locations with different angular orientations (e.g., angle 303C is non-zero).

[0070] Now for reference Figure 4This illustrates an example embodiment of inserting a first 5DOF EM sensor 401 and a second 5DOF EM sensor 402 into the patient's bone 403 using a standard hollow needle 405 typical in most orthopedic procedures. In another embodiment, the first sensor 401 and the second sensor 402 may have an angular offset of "α" 404. In some embodiments, the offset angle "α" 404 may need to be greater than a predetermined value (e.g., a minimum angle of 0.50°, 0.75°, etc.). In some embodiments, this minimum value may be determined by the CASS during surgical planning and provided to the surgeon or medical professional. In some embodiments, the minimum value may be based on one or more factors, such as the orientation accuracy of the tracking system, the distance between the first and second EM sensors, the location of the field generator, the location of the field detector, the type of EM sensor, the quality of the EM sensor, the patient's anatomy, etc.

[0071] Therefore, as discussed herein, in some embodiments, pins / needles (e.g., sleeve mounting pins, etc.) may be used to insert one or more EM sensors. Typically, the pins / needles will be disposable components, while the sensor itself may be reusable. However, it should be understood that this is only one possible system, and various other systems may be used where the pins / needles and / or EM sensors are either single-use or reusable. In another embodiment, the EM sensor may be secured to a mounting pin / needle (e.g., using Luer lock fittings, etc.), which may allow for quick assembly and disassembly. In yet another embodiment, the EM sensor may utilize alternative sleeves and / or anchoring systems that allow for minimal intrusion into the sensor's placement.

[0072] In another embodiment, the system described above can allow for a multi-sensor navigation system that can detect and correct field distortions that plague electromagnetic tracking systems. It should be understood that field distortions can be caused by movement of any ferromagnetic material within the reference field. Therefore, as is known to those skilled in the art, a typical operating system (OR) has numerous devices that can cause interference (e.g., operating tables, LCD displays, lighting equipment, imaging systems, surgical instruments, etc.). Furthermore, field distortions are known to be difficult to detect. Using multiple EM sensors enables the system to accurately detect field distortions and / or alert the user that measurements of the current position may be inaccurate. Because the sensors (e.g., via pins / needles) are securely attached to the bone anatomy, relative measurements of the sensor positions (X, Y, Z) can be used to detect field distortions. As a non-limiting example, in some embodiments, after the EM sensors are attached to the bone, the relative distance between the two sensors is known and should be kept constant. Therefore, any change in this distance can indicate the presence of field distortion.

[0073] In some embodiments, surgeons can manually register specific objects using the system before or during surgery. For example, by interacting with a user interface, a surgeon can identify the starting position of a tool or bone structure. By tracking reference markers associated with the tool or bone structure, or by using other conventional image tracking methods, the processor can track the tool or bone as it moves through the environment in a 3D model.

[0074] In some embodiments, certain markers, such as reference markers for identifying individuals, vital instruments, or bones in an operating room, may include passive or active identifiers that can be picked up by a camera or camera array associated with a tracking system. For example, an infrared LED may flash a pattern that conveys a unique identifier to the source of the pattern, thus providing dynamic identification marking. Similarly, one-dimensional or two-dimensional optical codes (barcodes, QR codes, etc.) may be affixed to objects in the operating room to provide passive identification that can occur based on image analysis. If these codes are placed asymmetrically on the object, they can also be used to determine the object's orientation by comparing the location of the identifier to the extent of the object in an image. For example, a QR code may be placed in the corner of a tool tray, allowing tracking of the tray's orientation and identifier. Other tracking methods will be described throughout the text. For example, in some embodiments, surgeons and other personnel may wear augmented reality headsets to provide additional camera angles and tracking capabilities.

[0075] Besides optical tracking, certain features of an object can also be tracked by registering its physical properties and associating them with a trackable object (e.g., a reference marker fixed to a tool or bone). For example, a surgeon can perform a manual registration process, whereby the tracked tool and the tracked bone can be manipulated relative to each other. By striking the surface of the bone with the tip of the tool, a three-dimensional surface can be mapped onto the bone, which is associated with its position and orientation relative to the reference marker. By optically tracking the position and orientation (pose) of the reference marker associated with the bone, a model of the surface can be tracked in the environment via extrapolation.

[0076] The registration process of CASS 100 to the relevant anatomical structures of a patient may also involve the use of anatomical landmarks, such as landmarks on bone or cartilage. For example, CASS 100 may include a 3D model of the relevant bone or joint, and the surgeon can use probes attached to CASS to collect data intraoperatively on the location of bone landmarks on the patient's actual bones. Bone landmarks may include, for example, the medial and lateral malleoli, the ends of the proximal femur and distal tibia, and the center of the hip joint. CASS 100 can compare and register the location data of the bone landmarks collected by the surgeon with the probes with the location data of the same landmarks in the 3D model. Alternatively, CASS 100 can construct a 3D model of a bone or joint without preoperative image data by using location data of bone landmarks and bone surfaces collected by the surgeon using CASS probes or other means. The registration process may also include determining the individual axes of the joint. For example, for TKA, the surgeon can use CASS 100 to determine the anatomical and mechanical axes of the femur and tibia. Surgeons and CASS 100 can identify the center of the hip joint by moving the patient's legs in a spiral direction (i.e., circumferentially) so that CASS can determine the location of the hip joint center.

[0077] Organizational Navigation System 120 ( Figure 1 (Not shown) provides surgeons with real-time intraoperative visualization of the patient's bone, cartilage, muscle, nerves, and / or blood vessels surrounding the surgical area. Examples of systems that can be used for tissue navigation include fluorescence imaging systems and ultrasound systems.

[0078] Display 125 provides a graphical user interface (GUI) that displays images collected by the tissue navigation system 120, as well as other surgery-related information. For example, in one embodiment, display 125 overlays image information collected preoperatively or intraoperatively from various modalities (e.g., CT, MRI, X-ray, fluorescence, ultrasound, etc.) to provide the surgeon with various views of the patient's anatomy and real-time status. Display 125 may include, for example, one or more computer monitors. As an alternative to or supplement to display 125, one or more surgical personnel may wear an augmented reality (AR) head-mounted device (HMD). For example, in Figure 1 In this case, the surgeon 111 wears an AR HMD 155, which can, for example, overlay preoperative image data onto the patient or provide surgical planning advice. Various exemplary uses of the AR HMD 155 in surgical procedures are described in detail in the following sections.

[0079] The surgical computer 150 provides control instructions to various components of the CASS 100, collects data from those components, and provides general processing for various data required during surgery. In some embodiments, the surgical computer 150 is a general-purpose computer. In other embodiments, the surgical computer 150 may be a parallel computing platform that uses multiple central processing units (CPUs) or graphics processing units (GPUs) to perform processing. In some embodiments, the surgical computer 150 is connected to a remote server via one or more computer networks (e.g., the Internet). The remote server may be used for, for example, data storage or the execution of computationally intensive processing tasks.

[0080] Various techniques known in the art can be used to connect the surgical computer 150 to other components of the CASS 100. Furthermore, the computer can be connected to the surgical computer 150 using a variety of technologies. For example, the end effector 105B can be connected to the surgical computer 150 via a wired (i.e., serial) connection. The tracking system 115, tissue navigation system 120, and display 125 can similarly be connected to the surgical computer 150 using wired connections. Alternatively, the tracking system 115, tissue navigation system 120, and display 125 can be connected to the surgical computer 150 using wireless technologies such as, but not limited to, Wi-Fi, Bluetooth, near field communication (NFC), or ZigBee.

[0081] Dynamic impact and acetabular reamer device

[0082] The above is about Figure 1 Part of the flexibility of the described CASS design lies in the ability to add additional or alternative devices to the CASS 100 as needed to support specific surgical procedures. For example, in the case of hip surgery, the CASS 100 may include a powered impact device. The impact device is designed to repeatedly apply impact forces that a surgeon can use to perform activities such as implant alignment. For instance, in total hip replacement (THA), surgeons typically use an impact device to insert a prosthetic acetabular cup into the acetabulum of the implant host. While impact devices can be inherently manual (e.g., operated by a surgeon striking the impactor with a hammer), powered impact devices are generally easier and faster to use in the surgical setting. The powered impact device may be powered, for example, by a battery attached to it. Various attachments can be connected to the powered impact device to allow the impact forces to be directed in various ways as needed during surgery. Also in the case of hip surgery, the CASS 100 may include a powered, robot-controlled end effector to dilate the acetabulum to accommodate the acetabular cup implant.

[0083] In robot-assisted THA, the patient's anatomy can be registered to the CASS 100 using CT or other image data, identification of anatomical landmarks, a tracker array attached to the patient's bones, and one or more cameras. The tracker array can be mounted on the iliac crest using clamps and / or bone pins, and can be mounted externally through the skin or internally (posterolaterally or anterolaterally) through an incision made for performing the THA. For THA, the CASS 100 can utilize one or more femoral cortical screws inserted into the proximal femur as checkpoints to aid the registration process. The CASS 100 can also utilize one or more checkpoint screws inserted into the pelvis as additional checkpoints to aid the registration process. The femoral tracker array can be fixed or mounted in the femoral cortical screws. The CASS 100 can employ the following steps, wherein verification is performed using probes precisely placed on the monitor 125 by the surgeon on key areas of the proximal femur and pelvis identified by the surgeon. The tracker can be located on the robotic arm 105A or the end effector 105B to register the arm and / or end effector to the CASS 100. The verification process can also utilize proximal and distal femoral checkpoints. The CASS 100 can use color cues or other cues to inform the surgeon that the registration process between the bone and the robotic arm 105A or end effector 105B has been verified with a certain level of accuracy (e.g., within 1 mm).

[0084] For THA, the CASS 100 may include a puller tracking option using a femoral array, allowing the surgeon to obtain the puller's position and orientation intraoperatively and calculate the patient's hip length and offset values. Based on the information provided about the patient's hip joint and the planned implant position and orientation after puller tracking is completed, the surgeon can modify or adjust the surgical plan.

[0085] For robot-assisted THA, CASS 100 may include one or more powered reamers connected to or attached to a robotic arm 105A or an end effector 105B, which prepare the pelvic bone according to the surgical plan to receive the acetabular implant. The robotic arm 105A and / or the end effector 105B may notify the surgeon and / or control the power of the reamers to ensure that the acetabulum is resected (reamed) according to the surgical plan. For example, if the surgeon attempts to resect bone outside the boundaries of the bone to be resected according to the surgical plan, CASS 100 may disconnect the power to the reamers or instruct the surgeon to disconnect the power to the reamers. CASS 100 may provide the surgeon with the option to turn off or disengage the robotic control of the reamers. Instead of using a surgical plan in different colors, the display 125 may show the progress of the bone being resected (reamed). The surgeon can view the display of the bone being resected (reamed) to guide the reamers to complete the reaming according to the surgical plan. CASS 100 may provide the surgeon with visual or auditory cues to warn the surgeon that a resection is not in accordance with the surgical plan.

[0086] After reaming, the CASS 100 can use a manual or powered impactor attached to or connected to the robotic arm 105A or end effector 105B to impact the test and final implants into the acetabulum. The robotic arm 105A and / or end effector 105B can be used to guide the impactor to impact the test and final implants into the acetabulum according to the surgical plan. The CASS 100 can display the position and orientation of the test and final implants relative to the bone to inform the surgeon how to compare the orientation and position of the test and final implants with the surgical plan. The display 125 can display the position and orientation of the implants as the surgeon manipulates the leg and hip. If the surgeon is not satisfied with the initial implant position and orientation, the CASS 100 can provide the surgeon with the option to replan and redo the reaming and implant impact by preparing a new surgical plan.

[0087] Preoperatively, the CASS 100 can develop a proposed surgical plan based on a 3D model of the hip joint and other patient-specific information, such as the mechanical and anatomical axes of the leg bones, the epicondyle axis, the femoral neck axis, the dimensions (e.g., length) of the femur and hip, the midline axis of the hip joint, the ASIS axis of the hip joint, and the location of anatomical landmarks such as the lesser trochanter landmark, the distal landmark, and the center of rotation of the hip joint. The surgical plan developed by CASS can provide recommended optimal implant size, as well as implant location and orientation, based on the 3D model of the hip joint and other patient-specific information. The surgical plan developed by CASS can include recommended details regarding offset values, tilt and anteversion values, center of rotation, cup size, mid-range value, superior-inferior fit, femoral stem size, and length.

[0088] For THA, the surgical plan developed by CASS can be viewed preoperatively and intraoperatively, and the surgeon can modify the CASS-developed surgical plan preoperatively or intraoperatively. The CASS-developed surgical plan can display the planned hip resection and, based on the planned resection, the planned implant is superimposed onto the hip joint. CASS 100 can provide the surgeon with a choice of different surgical procedures, which will be displayed to the surgeon according to their preferences. For example, the surgeon can choose from different workflows based on the number and type of anatomical landmarks examined and acquired and / or the location and number of tracker arrays used during registration.

[0089] According to some embodiments, the powered impact device used with the CASS 100 can operate in a variety of different settings. In some embodiments, the surgeon adjusts the settings via a manual switch or other physical mechanism on the powered impact device. In other embodiments, a digital interface can be used, allowing setting input, for example, via a touchscreen on the powered impact device. Such a digital interface can allow available settings to vary based on, for example, the type of attachment connected to an electrical attachment device. In some embodiments, the settings can be changed by communicating with a robot or other computer system within the CASS 100, rather than adjusting the settings on the powered impact device itself. Such a connection can be established using, for example, a Bluetooth or Wi-Fi networking module on the powered impact device. In another embodiment, the impact device and end components can include features that allow the impact device to know which end components (cup impactor, puller handle, etc.) are attached without requiring any action from the surgeon, and adjust the settings accordingly. This can be achieved, for example, via QR codes, barcodes, RFID tags, or other methods.

[0090] Examples of possible settings include cup impact settings (e.g., unidirectional, specified frequency range, specified force and / or energy range); puller impact settings (e.g., bidirectional / oscillating within a specified frequency range, specified force and / or energy range); femoral head impact settings (e.g., unidirectional / single-shot impact with specified force or energy); and dry impact settings (e.g., unidirectional impact with specified force or energy at a specified frequency). Additionally, in some embodiments, the dynamic impact device includes settings related to acetabular liner impact (e.g., unidirectional / single-shot impact with specified force or energy). Multiple settings may be available for each type of liner (e.g., polymer, ceramic, oxinium, or other materials). Furthermore, the dynamic impact device can provide settings for different bone qualities based on preoperative testing / imaging / knowledge and / or intraoperative assessment by the surgeon. In some embodiments, the dynamic impact device can have dual functionality. For example, the dynamic impact device can not only provide reciprocating motion to deliver impact force but also provide reciprocating motion for a puller or file.

[0091] In some embodiments, the dynamic impact device includes a feedback sensor that collects data during instrument use and transmits the data to a computing device, such as a controller within the device or a surgical computer 150. This computing device can then record the data for later analysis and use. Examples of data that can be collected include, but are not limited to, sound waves, predetermined resonant frequencies for each instrument, reaction forces or rebound energy from the patient's bone, the position of the device relative to an image (e.g., fluorescence, CT, ultrasound, MRI, etc.) of a registered bone anatomy, and / or external strain gauges on the bone.

[0092] Once the data is collected, the computing device can execute one or more algorithms in real time or near real time to assist the surgeon in performing surgical procedures. For example, in some embodiments, the computing device uses the collected data to derive information such as the correct final retractor size (femur); when the shaft is fully in place (femoral side); or when the cup is in place relative to the THA (depth and / or orientation). Once this information is known, it can be displayed for the surgeon to view, or it can be used to activate haptic or other feedback mechanisms to guide the surgical procedure.

[0093] Furthermore, the data derived from the aforementioned algorithm can be used to operate the drive device. For example, during insertion of the prosthetic acetabular cup using a powered impact device, the device can automatically extend the impact head (e.g., an end effector) to move the implant into place, or shut off the device's power once the implant is fully in place. In one embodiment, the derived information can be used to automatically adjust bone quality settings, where the powered impact device should use less power to mitigate femoral / acetabular / pelvic fractures or damage to surrounding tissues.

[0094] robotic arm

[0095] In some embodiments, the CASS 100 includes a robotic arm 105A, which serves as an interface for stabilizing and holding various instruments used during surgical procedures. For example, in the case of hip surgery, these instruments may include, but are not limited to, retractors, sagittal or reciprocating saws, reamer handles, cup impactors, puller handles, and dry inserters. The robotic arm 105A may have multiple degrees of freedom (similar to a spider device) and the ability to lock into place (e.g., by pressing a button, voice activation, the surgeon removing their hand from the robotic arm, or other methods).

[0096] In some embodiments, movement of the robotic arm 105A can be achieved using a control panel built into the robotic arm system. For example, the display screen may include one or more input sources, such as physical buttons that guide the movement of the robotic arm 105A or a user interface with one or more icons. Surgeons or other healthcare professionals can engage with one or more input sources to position the robotic arm 105A during surgical procedures.

[0097] Tools or end effectors 105B attached to or integrated into the robotic arm 105A may include, but are not limited to, deburring devices, scalpels, cutting devices, retractors, joint tensioners, etc. In embodiments using the end effector 105B, the end effector may be positioned at the end of the robotic arm 105A, enabling any motor-controlled operation to be performed within the robotic arm system. In embodiments using the tool, the tool may be fixed at the distal end of the robotic arm 105A, but the motor-controlled operation may be located within the tool itself.

[0098] The robotic arm 105A can be internally motorized to stabilize it, preventing it from falling and impacting patients, operating tables, surgical personnel, etc., and allowing the surgeon to move the robotic arm without having to fully support its weight. While the surgeon moves the robotic arm 105A, it provides some resistance to prevent it from moving too quickly or activating too many degrees of freedom at once. The position and locked state of the robotic arm 105A can be tracked, for example, by a controller or surgical computer 150.

[0099] In some embodiments, the robotic arm 105A can be moved to its ideal position and orientation by hand (e.g., by a surgeon) or by internal motors to perform the task at hand. In some embodiments, the robotic arm 105A may be able to operate in a “free” mode, allowing the surgeon to position the arm in a desired location without restriction. In free mode, as described above, the position and orientation of the robotic arm 105A can still be tracked. In one embodiment, during a designated portion of the surgical plan tracked by the surgical computer 150, certain degrees of freedom can be selectively released upon input from a user (e.g., a surgeon). A design in which the robotic arm 105A is internally powered by hydraulics or motors or provides resistance to external manual movement by similar means can be described as a powered robotic arm, while an arm that is manually manipulated without power feedback but can be manually or automatically locked in place can be described as a passive robotic arm.

[0100] The robotic arm 105A or end effector 105B may include triggers or other devices to control the power of the saw or drill. Engagement of the trigger or other device by the surgeon can transition the robotic arm 105A or end effector 105B from a motorized alignment mode to a mode where the saw or drill is engaged and energized. Additionally, the CASS 100 may include a foot pedal (not shown) that, when activated, causes the system to perform certain functions. For example, the surgeon may activate the foot pedal to instruct the CASS 100 to place the robotic arm 105A or end effector 105B in an automatic mode, which positions the robotic arm or end effector relative to the patient's anatomy to perform necessary resections. The CASS 100 may also place the robotic arm 105A or end effector 105B in a cooperative mode, which allows the surgeon to manually manipulate the robotic arm or end effector and position it in a specific location. The cooperative mode can be configured to allow the surgeon to move the robotic arm 105A or end effector 105B medially or laterally while restricting movement in other directions. As discussed, the robotic arm 105A or end effector 105B may include a cutting device (saw, drill, and sharpener) or a cutting guide or clamp 105D that guides the cutting device. In other embodiments, the movement of the robotic arm 105A or the robot-controlled end effector 105B may be entirely controlled by the CASS 100 without any assistance or input from a surgeon or other medical professional, or with very little assistance or input. In still other embodiments, a surgeon or other medical professional may remotely control the movement of the robotic arm 105A or the robot-controlled end effector 105B using a control mechanism separate from the robotic arm or robot-controlled end effector device, such as a joystick or interactive monitor or display control device.

[0101] The following examples describe the use of robotic devices in hip surgery; however, it should be understood that robotic arms may have other applications in surgical procedures involving the knee, shoulder, etc. An example of the use of a robotic arm in creating anterior cruciate ligament (ACL) graft tunnels is described in WIPO Publication No. WO 2020 / 047051, filed August 28, 2019, entitled "Robotic Assisted Ligament Graft Placement and Tensioning," the entire contents of which are incorporated herein by reference.

[0102] The robotic arm 105A can be used to hold a retractor. For example, in one embodiment, the surgeon can move the robotic arm 105A to a desired position. At this point, the robotic arm 105A can lock into place. In some embodiments, the robotic arm 105A is provided with data about the patient's position so that if the patient moves, the robotic arm can adjust the retractor position accordingly. In some embodiments, multiple robotic arms can be used, thereby allowing multiple retractors to be held or more than one action to be performed simultaneously (e.g., retractor holding and dilation).

[0103] The robotic arm 105A can also be used to help stabilize the surgeon's hand when making a femoral neck incision. In this application, certain limitations can be imposed on the control of the robotic arm 105A to prevent soft tissue injury. For example, in one embodiment, the surgical computer 150 tracks the position of the robotic arm 105A as it operates. If the tracked position approaches an area where tissue damage is predicted, a command can be sent to the robotic arm 105A to stop it. Alternatively, in the case where the robotic arm 105A is automatically controlled by the surgical computer 150, the surgical computer can ensure that it does not provide any instructions to the robotic arm that would cause it to enter an area where soft tissue injury may occur. The surgical computer 150 can impose certain limitations on the surgeon to prevent the surgeon from digging too deep into the medial wall of the acetabulum or digging at an incorrect angle or orientation.

[0104] In some embodiments, the robotic arm 105A can be used to hold the cup impactor at a desired angle or orientation during cup impact. Once the final position has been reached, the robotic arm 105A can prevent any further positioning to avoid damage to the pelvis.

[0105] The surgeon can use the robotic arm 105A to position the retractor handle in the desired location, allowing the surgeon to impact the retractor into the femoral canal in the desired orientation. In some embodiments, once the surgical computer 150 receives feedback that the retractor is fully in place, the robotic arm 105A can restrict the handle to prevent further advance of the retractor.

[0106] The robotic arm 105A can also be used in surface resurfacing applications. For example, the robotic arm 105A can stabilize the surgeon while using conventional instruments and provide certain constraints or limitations to allow for the proper placement of implanted components (e.g., guidewire placement, chamfering cutter, sleeve cutter, planar cutter, etc.). In cases where only a scalpel is used, the robotic arm 105A can stabilize the surgeon's handpiece and impose limitations on it to prevent the surgeon from deviating from the surgical plan and removing unwanted bone.

[0107] Robotic arm 105A may be a passive arm. As an example, robotic arm 105A may be a CIRQ robotic arm available from Brainlab AG. CIRQ is a registered trademark of Brainlab AG, Olof-Palme-Str. 9 81829, Munich, Germany. In one particular embodiment, robotic arm 105A is an intelligent gripping arm, as disclosed in U.S. Patent Application No. 15 / 525,585 to Krinninger et al., U.S. Patent Application No. 15 / 561,042 to Nowatschin et al., U.S. Patent No. 15 / 561,048 to Nowatschin et al., and U.S. Patent No. 10,342,636 to Nowatschin et al., the entire contents of which are incorporated herein by reference.

[0108] Generation and collection of surgical procedure data

[0109] The various services provided by healthcare professionals to treat a clinical condition are collectively referred to as the "care period." For a specific surgical intervention, the care period may include three phases: preoperative, intraoperative, and postoperative. During each phase, data is collected or generated that can be used to analyze the care period in order to understand the various characteristics of the procedure and identify patterns that can be used, for example, to make decisions with minimal human intervention in a training model. The data collected during the care period may be stored as a complete dataset at the surgical computer 150 or the surgical data server 180. Thus, for each care period, there exists a dataset that includes all data about the patient collected preoperatively, all data collected or stored intraoperatively by CASS 100, and any postoperative data provided by the patient or by the healthcare professionals monitoring the patient.

[0110] As explained in further detail, data collected during the care period can be used to enhance the execution of surgical procedures or provide a holistic understanding of surgical procedures and patient outcomes. For example, in some embodiments, data collected during the care period can be used to generate surgical plans. In one embodiment, a high level of preoperative planning is refined intraoperatively while data is collected during surgery. In this way, the surgical plan can be viewed as dynamically changing in real time or near real time as new data is collected through components of CASS 100. In other embodiments, preoperative images or other input data can be used to develop a robust plan that is easy to execute during surgery. In this case, data collected by CASS 100 during surgery can be used to make recommendations to ensure that the surgeon stays within the preoperative surgical plan. For example, if the surgeon is unsure how to achieve certain prescribed cuts or implant alignments, they can consult the surgical computer 150 for recommendations. In still other embodiments, preoperative and intraoperative planning schemes can be combined, allowing a robust preoperative plan to be dynamically modified as needed or desired during the surgical procedure. In some embodiments, biomechanical models of the patient's anatomy contribute simulation data to be considered by CASS 100 in developing preoperative, intraoperative, and postoperative / rehabilitation procedures to optimize the patient's implant performance outcomes.

[0111] Besides altering the surgical procedure itself, data collected during the care period can also be used as input for other surgical aids. For example, in some embodiments, care period data can be used to design implants. Example data-driven techniques for designing, sizing, and fitting implants are described in U.S. Patent Application No. 13 / 814,531, filed August 15, 2011, entitled "Systems and Methods for Optimizing Parameters for Orthopaedic Procedures"; U.S. Patent Application No. 14 / 232,958, filed July 20, 2012, entitled "Systems and Methods for Optimizing Fit of an Implant to Anatomy"; and U.S. Patent Application No. 12 / 234,444, filed September 19, 2008, entitled "Operatively Tuning Implants for Increased Performance". The entire contents of each of these patents are incorporated herein by reference.

[0112] Furthermore, the data can be used for educational, training, or research purposes. For example, using the following... Figure 5C The web-based approach described herein allows other doctors or students to remotely view surgeries through an interface that allows them to selectively view data collected from the various components of the CASS 100. After the surgical procedure, a similar interface can be used to “replay” the surgery for training or other educational purposes, or to identify the root cause of any problems or complications that occurred during the procedure.

[0113] Data acquired during the preoperative phase typically includes all information collected or generated prior to surgery. Thus, information about the patient can be obtained, for example, from a patient entry form or electronic medical record (EMR). Examples of patient information that can be collected include, but are not limited to, patient demographics, diagnosis, medical history, medical records, vital signs, medical history information, allergies, and laboratory test results. Preoperative data may also include images relating to the anatomical region of interest. These images may be acquired, for example, using magnetic resonance imaging (MRI), computed tomography (CT), X-ray, ultrasound, or any other means known in the art. Preoperative data may also include quality-of-life data obtained from the patient. For example, in one embodiment, the patient uses a mobile application (“app”) to answer a questionnaire about their current quality of life. In some embodiments, the preoperative data used by CASS 100 includes demographics, anthropometry, culture, or other specific characteristics of the patient that may be correlated with activity levels and specific patient activities to tailor surgical plans for the patient. For example, people of certain cultures or demographics may prefer to use a squat toilet daily.

[0114] Figure 5A and 5B Examples of data that can be obtained during the intraoperative phase of the nursing period are provided. These examples are based on the above references. Figure 1 The various components of the CASS 100 are described; however, it should be understood that other types of data may be used based on the type of equipment used during the operation and its usage.

[0115] Figure 5A Examples of some control instructions provided by the surgical computer 150 to other components of the CASS 100 according to some embodiments are shown. Note that... Figure 5A The example assumes that all components of the actuator platform 105 are directly controlled by the surgical computer 150. In embodiments where components are manually controlled by the surgeon 111, instructions can be provided on the display 125 or AR HMD 155 to instruct the surgeon 111 on how to move the components.

[0116] Various components included in the actuator platform 105 are controlled by a surgical computer 150, which provides position commands indicating the location of the component within a coordinate system. In some embodiments, the surgical computer 150 provides commands to the actuator platform 105 defining how to react when a component of the actuator platform 105 deviates from the surgical plan. These commands are in... Figure 5A The term "tactile" is used as a reference. For example, the end effector 105B can provide force to resist movement outside the planned area to be removed. Other commands that the actuator platform 105 can use include vibration and audio cues.

[0117] In some embodiments, the end effector 105B of the robotic arm 105A is operatively coupled to the cutting guide 105D. In response to an anatomical model of the surgical scenario, the robotic arm 105A can move the end effector 105B and the cutting guide 105D to the appropriate position to match the location of the femoral or tibial cut to be performed according to the surgical plan. This reduces the possibility of errors, allowing the vision system and the processor utilizing that vision system to implement the surgical plan, positioning the cutting guide 105D in a precise location and orientation relative to the tibia or femur to align the cutting groove of the cutting guide with the cut to be performed according to the surgical plan. The surgeon can then use any suitable tool, such as a vibratory or rotary saw or drill, to perform the cut (or drill) with perfect placement and orientation, as the tool is mechanically limited by the characteristics of the cutting guide 105D. In some embodiments, the cutting guide 105D may include one or more pin holes, which the surgeon uses to drill and tighten or pin the cutting guide into the appropriate position before performing the resection of patient tissue using the cutting guide. This allows the robotic arm 105A to be released or ensures that the cutting guide 105D is fully fixed without moving relative to the bone to be removed. For example, this procedure can be used to create a first distal incision in the femur during total knee arthroplasty. In some embodiments, where the joint replacement is a hip replacement, the cutting guide 105D can be fixed to the femoral head or acetabulum for the corresponding hip replacement resection. It should be understood that any joint replacement utilizing a precise incision can employ the robotic arm 105A and / or the cutting guide 105D in this manner.

[0118] The resection device 110 provides a variety of commands to perform bone or tissue manipulations. Similar to the actuator platform 105, position information can be provided to the resection device 110 to specify where it should be positioned during resection. Other commands provided to the resection device 110 may vary depending on the type of resection device. For example, for mechanical or ultrasonic resection tools, commands may specify the tool's speed and frequency. For radiofrequency ablation (RFA) and other laser ablation tools, these commands may specify intensity and pulse duration.

[0119] Some components of the CASS 100 do not require direct control by the surgical computer 150; instead, the surgical computer 150 only needs to activate the components, which then execute software locally to specify how data is collected and provided to the surgical computer 150. Figure 2 In example A, two components operate in this manner: the tracking system 115 and the organization navigation system 120.

[0120] The surgical computer 150 provides the display 125 with any visualizations required by the surgeon 111 during surgery. For the monitor, the surgical computer 150 can use techniques known in the art to provide instructions for displaying images, a GUI, etc. The display 125 can include various parts of the surgical planning workflow. For example, during the registration process, the display 125 can display a preoperatively constructed 3D bone model and show the location of probes as the surgeon uses probes to collect anatomical landmarks on the patient. The display 125 can include information about the target surgical area. For example, in conjunction with TKA, the display 125 can show the mechanical and anatomical axes of the femur and tibia. The display 125 can show the varus and valgus angles of the knee joint based on the surgical plan, and the CASS 100 can show how such angles would be affected if anticipated modifications to the surgical plan were made. Therefore, the display 125 is an interactive interface that can dynamically update and display how changes to the surgical plan will affect the procedure and the final position and orientation of the implant mounted on the bone.

[0121] As the workflow progresses to preparation for bone cutting or resection, the display 125 can show the planned or recommended bone cut before any cut is performed. The surgeon 111 can manipulate the image display to provide different anatomical views of the target area and may have the option to change or modify the planned bone cut based on the patient's intraoperative assessment. The display 125 can show how the selected implant will be placed on the bone if the planned bone cut is performed. If the surgeon 111 chooses to change the previously planned bone cut, the display 125 can show how the modified bone cut will change the position and orientation of the implant when placed on the bone.

[0122] The display 125 can provide the surgeon 111 with various data and information about the patient, the planned surgical procedure, and the implant. Various patient-specific information can be displayed, including real-time data on the patient's health, such as heart rate, blood pressure, etc. The display 125 can also include information about the anatomical structures of the surgical target area, including the location of landmarks, the current state of the anatomy (e.g., whether any resections have been performed, the depth and angle of the planned and performed bone cuts), and the future state of the anatomy as the surgical plan progresses. The display 125 can also provide or show additional information about the surgical target area. For TKA, the display 125 can provide information about the gap between the femur and tibia (e.g., gap balance) and how such a gap will change if the planned surgical procedure is performed. For TKA, the display 125 can provide additional relevant information about the knee joint, such as data on joint tension (e.g., ligament laxity) and information about joint rotation and alignment. The display 125 can show how the planned implant placement and location will affect the patient when the knee is flexed. The display 125 can show how the use of different implants or the use of the same implant of different sizes will affect the surgical plan and preview how such implants will be positioned on the bone. The CASS 100 can provide such information for each planned osteotomy in either a TKA or THA. In a TKA, the CASS 100 can provide robotic control for one or more planned osteotomies. For example, the CASS 100 can only provide robotic control for the initial distal femoral resection, and the surgeon 111 can manually perform other resections (anterior, posterior, and chamfered cuts) using conventional means such as a 4-in-1 cutting guide or clamp 105D.

[0123] The display 125 can use different colors to inform the surgeon of the status of the surgical plan. For example, unremoved bone can be displayed in a first color, removed bone in a second color, and planned removal in a third color. Implants can be superimposed on the bone in the display 125, and the implant color can be changed or correspond to different types or sizes of implants.

[0124] The information and options displayed on monitor 125 can vary depending on the type of surgical procedure being performed. Furthermore, surgeon 111 can request or select a specific surgical workflow display that matches or is consistent with his or her surgical planning preferences. For example, for surgeon 111 who typically performs a tibialis resection before a femoral resection in TKA, monitor 125 and the associated workflow can be adapted to take that preference into account. Surgeon 111 can also pre-select to include or remove certain steps from the standard surgical workflow display. For example, if surgeon 111 uses resection measurements to finalize the implantation plan but does not analyze ligament-space balance when finalizing the implantation plan, the surgical workflow display can be organized into modules, and the surgeon can select which modules to display and the order in which the modules are presented based on the surgeon's preferences or the specific surgical circumstances. For example, modules involving ligament and space balance can include pre- and post-resection ligament / space balance, and surgeon 111 can select which modules to include in their default surgical planning workflow depending on whether such ligament and space balance is performed before or after (or before and after) the osteotomy.

[0125] For more specialized display devices, such as AR HMDs, the surgical computer 150 can use data formats supported by the device to provide images, text, etc. For example, if the display 125 is such as a Microsoft HoloLens... TM Or Magic LeapOne TM If the holographic device is used, the surgical computer 150 can use the HoloLens application programming interface (API) to send commands specifying the location and content of the hologram displayed in the surgeon 111's field of vision.

[0126] In some embodiments, one or more surgical planning models may be incorporated into CASS 100 and used in the development of surgical plans provided to surgeon 111. The term "surgical planning model" refers to software that simulates the biomechanical properties of anatomical structures under various conditions to determine the optimal manner of performing incisions and other surgical activities. For example, for knee replacement surgery, a surgical planning model can measure parameters of functional activities, such as deep knee flexion, gait, etc., and select incision locations on the knee to optimize implant placement. An example of a surgical planning model is LIFEMOD from Smith and Nephew. TM Simulation software. In some embodiments, the surgical computer 150 includes a computational architecture (e.g., a GPU-based parallel processing environment) that allows the full execution of a surgical planning model during surgery. In other embodiments, the surgical computer 150 may be connected via a network to a remote computer that allows such execution, such as a surgical data server 180 (see [link to surgical data server]). Figure 5CAs an alternative to a full execution of the surgical planning model, in some embodiments, a set of transfer functions is derived that simplifies the mathematical operations acquired by the model into one or more predictive equations. These predictive equations are then used instead of performing a full simulation during surgery. Further details regarding the use of transfer functions are described in WIPO Publication No. 2020 / 037308, filed August 19, 2019, entitled “Patient Specific Surgical Method and System,” the entire contents of which are incorporated herein by reference.

[0127] Figure 5B Examples of some types of data that can be provided from the various components of CASS 100 to the surgical computer 150 are shown. In some embodiments, components may stream data to the surgical computer 150 in real time or near real time during surgery. In other embodiments, components may queue data and send it to the surgical computer 150 at set intervals (e.g., per second). Data can be transmitted using any format known in the art. Thus, in some embodiments, all components transmit data to the surgical computer 150 in a common format. In other embodiments, each component may use a different data format, and the surgical computer 150 may be configured with one or more software applications capable of converting the data.

[0128] Typically, the surgical computer 150 can be used as a central point for collecting CASS data. The exact content of the data will depend on the source. For example, each component of the actuator platform 105 provides a measurement location to the surgical computer 150. Therefore, by comparing the measurement location with the location initially specified by the surgical computer 150 (see...), the data is collected and analyzed. Figure 5B By comparing these parameters, the surgical computer can identify deviations that occur during the procedure.

[0129] The resection device 110 can send various types of data to the surgical computer 150 depending on the type of device used. Exemplary data types that can be sent include measured torque, audio signatures, and measured displacement values. Similarly, the tracking technology 115 can provide different types of data depending on the tracking method employed. Exemplary tracking data types include tracked items (e.g., anatomical structures, tools, etc.), ultrasound images, and position values ​​of surface or marker collection points or axes. When the system is operating, the tissue navigation system 120 provides the surgical computer 150 with anatomical locations, shapes, etc.

[0130] While the display 125 is typically used to output data for presentation to a user, it can also provide data to the surgical computer 150. For example, in an embodiment where a monitor is used as part of the display 125, the surgeon 111 can interact with a GUI to provide input, which is then sent to the surgical computer 150 for further processing. For AR applications, the measured position and displacement of the HMD can be sent to the surgical computer 150, allowing it to update the presented view as needed.

[0131] During the postoperative phase of the care period, various types of data can be collected to quantify the overall improvement or deterioration of the patient's condition as a result of the surgery. Data can take the form of, for example, self-reported information from patients through questionnaires. For instance, in the case of knee replacement surgery, the Oxford Knee Score can be used to measure functional status, and the EQ5D-5L questionnaire can be used to measure postoperative quality of life. Other examples in the case of hip replacement surgery may include the Oxford Hip Score, the Harris Hip Score, and the WOMAC (Western University and McMaster University Osteoarthritis Index). Such questionnaires can be administered, for example, by healthcare professionals directly in a clinical setting, or using mobile applications that allow patients to answer questions directly. In some embodiments, patients may be equipped with one or more wearable devices to collect data related to the surgery. For example, after knee surgery, patients may be equipped with a knee brace that includes sensors for monitoring knee position, flexibility, etc. This information can be collected and transmitted to the patient's mobile device for the surgeon to review in order to assess the surgical outcome and address any issues. In some embodiments, one or more cameras may acquire and record movement of the patient's body parts during designated postoperative activities. This motion can be compared with biomechanical models to better understand the function of the patient's joints, and to better predict rehabilitation progress and identify any necessary corrections.

[0132] The postoperative phase of the care period can continue throughout the patient's lifespan. For example, in some embodiments, the surgical computer 150 or other components including CASS 100 can continue to receive and collect data related to the surgical procedure after it has been performed. This data may include, for example, images, question answers, “normal” patient data (e.g., blood type, blood pressure, condition, medications, etc.), biometric data (e.g., gait, etc.), and objective and subjective data on specific issues (e.g., knee or hip pain). This data may be explicitly provided to the surgical computer 150 or other CASS components by the patient or the patient's physician. Alternatively or additionally, the surgical computer 150 or other CASS components may monitor the patient's EMR and retrieve relevant information when available. This longitudinal view of patient recovery allows the surgical computer 150 or other CASS components to provide a more objective analysis of patient outcomes to measure and track the success or failure of a given procedure. For example, regression analysis of various data items collected during the care period can link the patient's condition long after the surgical procedure to the surgery. This analysis can be further enhanced by analyzing patient groups with similar procedures and / or similar anatomy.

[0133] In some embodiments, data is collected at a central location to provide easier analysis and use. In some cases, data can be collected manually from various CASS components. For example, a portable storage device (e.g., a USB stick) can be attached to the surgical computer 150 to retrieve data collected during surgery. The data can then be transferred, for example, via a desktop computer to a centralized storage device. Alternatively, in some embodiments, the surgical computer 150 is directly connected to a centralized storage device via a network 175, such as... Figure 5C As shown in the image.

[0134] Figure 5C A cloud-based implementation is illustrated, in which surgical computer 150 is connected to surgical data server 180 via network 175. This network 175 can be, for example, a private intranet or the Internet. In addition to data from surgical computer 150, relevant data can also be transferred to surgical data server 180 from other sources. Figure 5CThe example illustrates three additional data sources: patient 160, healthcare professionals 165, and an EMR database 170. Therefore, patient 160 can, for example, use a mobile application to send pre- and post-operative data to surgical data server 180. Healthcare professionals 165 include the surgeon and his or her staff, as well as any other professionals working with patient 160 (e.g., private physicians, rehabilitation specialists, etc.). It should also be noted that the EMR database 170 can be used for pre- and post-operative data. For example, assuming patient 160 has given sufficient permission, surgical data server 180 can collect the patient's pre-operative EMR. Surgical data server 180 can then continue to monitor the EMR for any post-operative updates.

[0135] At surgical data server 180, a nursing period database 185 is used to store various data collected during a patient's nursing period. The nursing period database 185 can be implemented using any technology known in the art. For example, in some embodiments, an SQL-based database can be used, where all various data items are structured in a way that allows them to be easily incorporated into two sets of SQL rows and columns. However, in other embodiments, a No-SQL database can be employed to allow unstructured data while providing the ability to process and respond to queries quickly. As understood in the art, the term "No-SQL" is used to define a class of databases that are not related in their design. Various types of No-SQL databases can generally be grouped according to their underlying data model. These groupings can include databases using column-based data models (e.g., Cassandra), document-based data models (e.g., MongoDB), key-value-based data models (e.g., Redis), and / or graph-based data models (e.g., Allego). The various embodiments described herein can be implemented using any type of No-SQL database, and in some embodiments, different types of databases can support the nursing period database 185.

[0136] Data can be transferred between various data sources and surgical data server 180 using any data format and transmission technology known in the art. It should be noted that... Figure 5C The architecture shown allows for the transfer of data from a data source to the surgical data server 180, as well as the retrieval of data from the surgical data server 180 by the data source. For example, as explained in detail below, in some embodiments, the surgical computer 150 can use data from past surgeries, machine learning models, etc., to help guide the surgical procedure.

[0137] In some embodiments, the surgical computer 150 or surgical data server 180 may perform a deidentification process to ensure that data stored in the care period database 185 meets Health Insurance Portability and Accountability Act (HIPAA) standards or other legal requirements. HIPAA provides a list of certain identifiers that must be removed from data during the deidentification process. The aforementioned deidentification process may scan for these identifiers in the data transferred to the care period database 185 for storage. For example, in one embodiment, the surgical computer 150 performs the deidentification process before initially transferring a specific data item or set of data items to the surgical data server 180. In some embodiments, unique identifiers are assigned to data from a specific care period for reidentification if necessary.

[0138] although Figure 5A –5C discusses data collection in the context of a single care period; however, it should be understood that the general concept can be extended to data collection across multiple care periods. For example, surgical data can be collected throughout the care period each time a surgery is performed using the CASS 100 and stored at the surgical computer 150 or surgical data server 180. As explained in further detail below, a robust database of care period data allows for the generation of optimized values, measurements, distances or other parameters, and other recommendations related to the surgical procedure. In some embodiments, various datasets are indexed in a database or other storage medium in a manner that allows for rapid retrieval of relevant information during the surgical procedure. For example, in one embodiment, a patient-centric set of indexes can be used so that data can be easily extracted from a specific patient or a group of patients similar to a specific patient. This concept can be similarly applied to surgeons, implant characteristics, CASS component types, etc.

[0139] Further details regarding the management of care period data are described in U.S. Patent Application No. 62 / 783,858, filed December 21, 2018, entitled “Methods and Systems for Providing an Episode of Care,” the entire contents of which are incorporated herein by reference.

[0140] Open and Closed Digital Ecosystems

[0141] In some embodiments, the CASS 100 is designed to function as a standalone or “closed” digital ecosystem. Each component of the CASS 100 is specifically designed for use within a closed ecosystem, and devices outside the digital ecosystem typically cannot access the data. For example, in some embodiments, each component includes software or firmware implementing proprietary protocols for activities such as communication, storage, and security. The concept of a closed digital ecosystem may be ideal for companies that want to control all components of the CASS 100 to ensure compliance with certain compatibility, security, and reliability standards. For example, the CASS 100 may be designed such that new components cannot be used with the CASS without the company’s certification.

[0142] In other embodiments, CASS 100 is designed as an “open” digital ecosystem. In these embodiments, components can be manufactured by a variety of different companies according to standards for activities such as communication, storage, and security. Therefore, by using these standards, any company is free to build independent, compliant components of the CASS platform. Data can be transferred between components using publicly available application programming interfaces (APIs) and open, shareable data formats.

[0143] To illustrate one type of recommendation that can be performed using CASS 100, a technique for optimizing surgical parameters is disclosed below. In this document, the term "optimization" refers to selecting the optimal parameters based on certain specified criteria. In extreme cases, optimization can refer to selecting the optimal parameters based on data from the entire period of care (including any preoperative data, CASS data status at a given time point, and postoperative goals). Furthermore, historical data can be used to perform optimization, such as data generated during past surgeries involving, for example, the same surgeon, past patients with similar physical characteristics to the current patient, etc.

[0144] Optimized parameters can be dependent on parts of the patient's anatomy to be operated on. For example, for knee surgery, surgical parameters may include positioning information for the femoral and tibial components, including but not limited to rotational alignment (e.g., varus / valgus rotation, external rotation, flexion rotation of the femoral component, posterior tilt angle of the tibial component), resection depth (e.g., varus knee, valgus knee), and the type, size, and location of the implant. Positioning information may also include surgical parameters for combining implants, such as overall limb alignment, combined tibiofemoral hyperextension, and combined tibiofemoral resection. Other examples of parameters that CASS 100 can optimize for a given TKA femoral implant include the following:

[0145]

[0146] Other examples of parameters that CASS 100 can optimize for a given TKA tibial implant include the following:

[0147] For hip surgery, surgical parameters may include femoral neck resection location and angle, cup tilt angle, cup anteversion angle, cup depth, femoral stem design, femoral stem size, femoral stem fit within the canal, femoral offset, leg length, and femoral type of implant.

[0148] Shoulder parameters may include, but are not limited to, humeral resection depth / angle, humeral shaft type, humeral deviation, glenoid type and tilt, as well as reverse shoulder parameters, such as humeral resection depth / angle, humeral shaft type, glenoid tilt / type, glenoid ball orientation, glenoid ball deviation and deviation direction.

[0149] Various routine techniques exist for optimizing surgical parameters. However, these techniques typically require extensive computation, thus necessitating preoperative parameter determination. Consequently, surgeons' ability to modify optimized parameters based on potential problems during surgery is limited. Moreover, routine optimization techniques often operate in a "black box" manner, with little or no explanation of the recommended parameter values. Therefore, if a surgeon decides to deviate from the recommended parameter values, they often do so without fully understanding the impact of that deviation on the remainder of the surgical procedure or on the patient's postoperative quality of life.

[0150] Surgical patient care system

[0151] Using the surgical patient care system 620, the general concept of optimization can be extended to the entire period of care. This system uses surgical data, along with other data from the patient 605 and healthcare professionals 630, to optimize outcomes and patient satisfaction, such as... Figure 6 As shown in the image.

[0152] Conventionally, preoperative diagnosis, preoperative surgical planning, intraoperative execution of the established plan, and postoperative management of total joint replacement surgery are all based on personal experience, published literature, and the surgeon's training knowledge base (ultimately, the individual surgeon's tribal knowledge and their peer "network" and journal publications), as well as their instinct for accurate intraoperative tactile discrimination of "balance" using guidance and visual cues, and accurate manual execution of plane resections. This existing knowledge base and execution method are limited in their ability to optimize outcomes for patients requiring care. For example, limitations exist in: accurately diagnosing patients for appropriate, minimally invasive established care; aligning dynamic patient, medical economic, and surgeon preferences with the patient's desired outcome; executing the surgical plan to ensure proper bone alignment and balance, etc.; and receiving data from disconnected sources with varying deviations that are difficult to reconcile into the overall patient framework. Therefore, data-driven tools that more accurately simulate anatomical responses and guide surgical planning can improve existing methods.

[0153] The surgical patient care system 620 is designed to utilize patient-specific data, surgeon data, healthcare institution data, and historical outcome data to develop algorithms that suggest or recommend optimal overall treatment plans for the patient throughout their entire care period (preoperative, intraoperative, and postoperative) based on desired clinical outcomes. For example, in one embodiment, the surgical patient care system 620 tracks adherence to the suggested or recommended plan and adjusts the plan based on patient / care provider performance. Once the surgical treatment plan is finalized, the surgical patient care system 620 records the collected data in a historical database. This database is accessible for future patients and for developing future treatment plans. In addition to utilizing statistical and mathematical models, simulation tools (e.g., Based on the preliminary or recommended surgical plan, the outcome, alignment, kinematics, etc., are simulated, and the preliminary or recommended plan is reconfigured according to the patient profile or surgeon's preferences to achieve the desired or optimal outcome. The Surgical Patient Care System 620 ensures that each patient is receiving personalized surgical and rehabilitation care, thereby increasing the chances of successful clinical outcomes and reducing the financial burden on facilities associated with near-term modifications.

[0154] In some embodiments, the surgical patient care system 620 employs a data collection and management approach to provide a detailed surgical case plan, which has different steps monitored and / or performed using CASS 100. User execution is calculated upon completion of each step and used to suggest changes to subsequent steps in the case plan. The generation of the case plan relies on a series of input data stored in a local or cloud storage database. The input data may be related to the patient currently receiving treatment or to historical data from patients who have received similar treatment.

[0155] Patient 605 provides inputs such as current patient data 610 and historical patient data 615 to surgical patient care system 620. Various methods generally known in the art can be used to collect such inputs from patient 605. For example, in some embodiments, patient 605 completes a paper or digital survey parsed by surgical patient care system 620 to extract patient data. In other embodiments, surgical patient care system 620 may extract patient data from existing information sources such as electronic medical records (EMRs), health history files, and payer / provider history files. In still other embodiments, surgical patient care system 620 may provide an application programming interface (API) that allows external data sources to push data to the surgical patient care system. For example, patient 605 may have a mobile phone, wearable device, or other mobile device that collects data (e.g., heart rate, pain or discomfort level, exercise or activity level, or patient-submitted responses to any number of preoperative planning criteria or conditional compliance) and provides that data to surgical patient care system 620. Similarly, patient 605 may have a digital application on their mobile or wearable device that can collect data and transmit it to surgical patient care system 620.

[0156] Current patient data 610 may include, but is not limited to: activity level, past medical history, comorbidities, pre-rehabilitation performance, health and fitness level, preoperative expected level (related to hospital, surgery, and rehabilitation), Metropolitan Statistical Area (MSA) driven score, genetic background, previous injuries (sports, trauma, etc.), previous joint replacement surgery, previous trauma surgery, previous sports medicine surgery, treatment of contralateral joints or limbs, gait or biomechanical information (back and ankle tissues), level of pain or discomfort, nursing infrastructure information (payer coverage type, level of home medical infrastructure, etc.), and indications of the expected ideal surgical outcome.

[0157] Historical patient data 615 may include, but is not limited to: activity level, past medical history, comorbidities, pre-rehabilitation performance, health and fitness level, preoperative expected level (related to hospital, surgery, and rehabilitation), MSA-driven score, genetic background, previous injuries (sports, trauma, etc.), previous joint replacement surgery, previous trauma surgery, previous sports medicine surgery, treatment of contralateral joints or limbs, gait or biomechanical information (back and ankle tissues), pain or discomfort level, nursing infrastructure information (payer coverage type, level of home medical infrastructure, etc.), expected desired surgical outcome, actual surgical outcome (patient-reported outcome [PRO], implant survival, pain level, activity level, etc.), size of the implant used, position / orientation / alignment of the implant used, and soft tissue balance achieved, etc.

[0158] The healthcare professional 630 performing the surgery or treatment can provide various types of data 625 to the surgical patient care system 620. This healthcare professional data 625 may include, for example, descriptions of known or preferred surgical techniques (e.g., cruciate retention (CR) vs. posterior stabilization (PS), size increase vs. size decrease, with and without a tourniquet, femoral stem style, preferred THA options, etc.), the healthcare professional 630's training level (e.g., years of experience, position trained, place of training, techniques imitated), previous success levels including historical data (outcomes, patient satisfaction), and expected desired outcomes regarding range of motion, recovery days, and device survival. The healthcare professional data 625 can be obtained, for example, through paper or digital surveys provided to the healthcare professional 630, via input from the healthcare professional into a mobile application, or by extracting relevant data from the EMR. Additionally, the CASS 100 can provide data such as profile data (e.g., patient-specific knee device profile) or a historical record describing the use of the CASS during surgery.

[0159] Information related to the facility where the surgery or treatment will be performed can be included in the input data. This data may include, but is not limited to, the following: outpatient surgery center (ASC) vs. hospital, facility trauma level, Joint Replacement Comprehensive Medical Plan (CJR) or bundled candidate, MSA-driven score, community vs. metropolitan, academic vs. non-academic, postoperative network access (skilled care facilities only [SNF], family health, etc.), availability of medical professionals, availability of implants, and availability of surgical equipment.

[0160] These facility inputs can be, for example, but not limited to, surveys (paper / digital), surgical planning tools (e.g., apps, websites, electronic medical records [EMR], etc.), hospital information databases (on the Internet), etc. Input data related to the associated healthcare economics may also be obtained, including but not limited to the patient's socioeconomic profile, the level of reimbursement the patient expects to receive, and whether the treatment is patient-specific.

[0161] These healthcare economic inputs can be obtained (e.g., but not limited to) through surveys (paper / digital), direct payer information, socioeconomic databases (providing postal codes online), etc. Finally, data derived from the simulation of the program is obtained. Simulation inputs include implant size, location, and orientation. Custom or commercially available anatomical modeling software programs (e.g.) can be used. Simulations can be performed using AnyBody or OpenSIM. It should be noted that the above data inputs may not be available for every patient, and the available data will be used to generate the treatment plan.

[0162] Prior to surgery, patient data 610, 615 and healthcare professional data 625 can be acquired and stored in a cloud-based database or online database (e.g., Figure 5C The surgical data server 180 shown is used. Information related to the procedure is provided to the computing system via wireless data transmission or manually using portable media storage. The computing system is configured to generate case plans for the CASS 100. The generation of case plans will be described below. It should be noted that the system can access historical data of previously treated patients, including implant size, location, and orientation automatically generated by a computer-assisted patient-specific knee device (PSKI) selection system or by the CASS 100 itself. For this purpose, a surgical sales representative or case engineer uploads case log data to the historical database using an online portal. In some embodiments, data transmission to the online database is wireless and automated.

[0163] Historical datasets from online databases are used as input to machine learning models, such as recurrent neural networks (RNNs) or other forms of artificial neural networks. As is generally understood in the art, artificial neural networks function similarly to biological neural networks and consist of a series of nodes and connections. The machine learning model is trained to predict one or more values ​​based on the input data. For the following sections, it is assumed that the machine learning model is trained to generate predictive equations. These predictive equations can be optimized to determine the optimal size, location, and orientation of the implant for best results or satisfaction.

[0164] Once the procedure is complete, all patient data and available outcome data, including implant size, location, and orientation determined by CASS 100, are collected and stored in a historical database. Any subsequent calculations of the objective equation via RNN will, in this manner, incorporate data from previous patients, allowing for continuous improvement of the system.

[0165] In addition to or as an alternative to determining implant location, in some embodiments, the predictive equations and associated optimizations can be used to generate a resection plane for use with a PSKI system. When used with a PSKI system, the computation and optimization of the predictive equations are performed preoperatively. The patient's anatomy is estimated using medical imaging data (X-ray, CT, MRI). Global optimization of the predictive equations can provide the ideal size and location of the implant component. The Boolean intersection of the implant component and the patient's anatomy is defined as the resection volume. A PSKI can be generated to remove the optimized resection envelope. In this embodiment, the surgeon cannot change the surgical plan intraoperatively.

[0166] Surgeons may choose to change the surgical case plan at any time before or during surgery. If a surgeon chooses to deviate from the surgical case plan, the size, position, and / or orientation of the changed components are locked, and global optimization (using previously described techniques) is refreshed based on the new size, position, and / or orientation of the components to find new ideal positions for other components, and the corresponding resections required to achieve the new optimized size, position, and / or orientation of the components. For example, if the surgeon determines that the size, position, and / or orientation of the femoral implant in a TKA needs to be updated or modified intraoperatively, the position of the femoral implant will be locked relative to the anatomy, and a new optimal position for the tibia will be calculated (through global optimization) by taking into account the surgeon's changes to the size, position, and / or orientation of the femoral implant. Furthermore, if the surgical system used to perform the case plan is robot-assisted (e.g., using...), Alternatively, a device like MAKO Rio can monitor bone removal and morphology in real time during surgery. If the resection performed during the procedure deviates from the surgical plan, the processor can take the actual resections performed into account to optimize the subsequent placement of additional components.

[0167] Figure 7AThis illustrates how a surgical patient care system 620 can be adapted to perform a case plan matching service. In this example, data related to the current patient 610 is acquired and compared, in whole or in part, with a historical database of patient data and related outcomes 615. For example, the surgeon may choose to compare the current patient's plan with a subset of the historical database. The data in the historical database can be filtered to include, for example, datasets with only good outcomes, datasets corresponding to historical surgeries of patients with profiles identical or similar to the current patient's profile, datasets corresponding to specific surgeons, datasets corresponding to specific elements of the surgical plan (e.g., surgery that preserves only specific ligaments), or any other criteria chosen by the surgeon or medical professional. For example, if the current patient data matches or correlates with data from a previous patient who experienced a good outcome, the previous patient's case plan can be accessed and adapted or adopted for the current patient. Predictive equations can be used in conjunction with intraoperative algorithms that identify or determine actions associated with the case plan. Based on relevant information from the historical database and / or pre-selected information, the intraoperative algorithm determines a series of actions recommended for the surgeon to perform. Each execution of the algorithm generates the next action in the case plan. If the surgeon performs the action, the outcome is evaluated. The results of the surgeon's actions are used to refine and update the inputs to the intraoperative algorithm for generating the next step in the case plan. Once the case plan has been fully executed, all data related to the case plan (including any deviations by the surgeon from the recommended actions) is stored in a database of historical data. In some embodiments, the system uses preoperative, intraoperative, or postoperative modules in a segmented manner, rather than the entire continuous care. In other words, caregivers can specify any arrangement or combination of treatment modules, including the use of a single module. These concepts are... Figure 7B As shown in the diagram, it can be applied to any type of surgery using CASS 100.

[0168] Surgical procedure showed

[0169] As mentioned above Figure 1 and Figures 5A-5CThe various components of the CASS 100 generate detailed data logs during surgery. The CASS 100 can track and record various actions and activities of the surgeon during each step of the surgery and compare the actual activities with the preoperative or intraoperative surgical plan. In some embodiments, software tools can be used to process this data into a format that allows for efficient “replay” of the surgery. For example, in one embodiment, one or more GUIs can be used, displaying all information presented on the display 125 during surgery. This can be supplemented with graphics and images showing data collected by different tools. For example, a GUI providing a visual illustration of the knee during tissue resection can provide measured torque and displacement of the resection equipment adjacent to the visual illustration to better provide an understanding of any deviations that occur from the planned resection area. The ability to view a replay of the surgical plan or switch between different stages of the actual surgery and the surgical plan can benefit surgeons and / or surgical personnel, allowing them to identify any deficiencies or challenging phases of the surgery that can be modified in future surgeries. Similarly, in an academic setting, the aforementioned GUI can be used as a teaching tool to train future surgeons and / or surgical personnel. In addition, because the dataset effectively records many elements of a surgeon's activities, it can also be used as evidence of whether a particular surgical procedure was performed correctly or incorrectly for other reasons (e.g., legal or compliance reasons).

[0170] Over time, as more surgical data is collected, a rich database may be acquired, describing surgical procedures performed by different surgeons on various types of anatomy (knee, shoulder, hip, etc.) for different patients. Furthermore, information such as implant type and size, patient demographics, etc., can be further used to augment the overall dataset. Once the dataset is established, it can be used to train machine learning models (e.g., RNNs) to predict how surgery will proceed based on the current state of CASS 100.

[0171] The training of the machine learning model can be performed as follows. During surgery, the overall state of the CASS 100 can be sampled over multiple time periods. The machine learning model can then be trained to transform the current state of the first time period into future states for different time periods. By analyzing the entire state of the CASS 100 rather than individual data items, any causal effects of interactions between the different components of the CASS 100 can be obtained. In some embodiments, multiple machine learning models can be used instead of a single model. In some embodiments, the machine learning model can be trained not only using the state of the CASS 100, but also using patient data (e.g., obtained from EMR) and the identity of the surgeon. This allows the model to make predictions with greater specificity. Moreover, if needed, it allows surgeons to make predictions selectively based solely on their own surgical experience.

[0172] In some embodiments, predictions or recommendations made by the aforementioned machine learning model can be directly integrated into the surgical procedure. For example, in some embodiments, the surgical computer 150 can execute a machine learning model in the background to make predictions or recommendations for upcoming actions or surgical conditions. Thus, multiple states can be predicted or recommended for each period. For example, the surgical computer 150 can predict or recommend states for the next 5 minutes in 30-second increments. Using this information, the surgeon can utilize a "process display" view of the surgery to allow visualization of future states. For example, Figure 7C A series of images illustrating the implant placement interface are shown and can be displayed to the surgeon. The surgeon can navigate these images, for example, by entering a specific time in the display 125 of the CASS 100 or instructing the system to advance or rewind the display in specific time increments using tactile, verbal, or other commands. In one embodiment, the process display may be presented at the top of the surgeon's field of vision in the AR HMD. In some embodiments, the process display can be updated in real time. For example, as the surgeon moves the resection tool around the planned resection area, the process display can be updated, allowing the surgeon to see how his or her movements affect other factors of the procedure.

[0173] In some embodiments, instead of simply using the current state of CASS 100 as input to the machine learning model, the model's input can include the planned future state. For example, a surgeon may instruct that he or she is planning to perform a specific bone resection of the knee joint. This instruction can be manually entered into the surgical computer 150, or the surgeon can provide the instruction verbally. The surgical computer 150 can then generate films showing the expected effects of the incision on the surgery. Such films can show, over specific time increments, how the surgery will be affected if the expected procedures are performed, including, for example, changes in patient anatomy, changes in implant position and orientation, and changes in surgical procedures and instruments. Surgeons or medical professionals can recall or request this type of film at any time during the surgery to preview how the expected procedures will affect the surgical plan if the expected procedures are performed.

[0174] It should be further noted that using a well-trained machine learning model and the CASS robot can automate various elements of the surgery, requiring minimal intervention from the surgeon, for example, by providing approval only for each step of the procedure. For instance, over time, robotic control using arms or other means can be gradually integrated into the surgical process, with less manual interaction between the surgeon and the robot's operation. In this case, the machine learning model can learn which robotic commands are needed to achieve certain states of the CASS implementation plan. Ultimately, the machine learning model can be used to generate films or similar views or displays that can predict and preview the entire surgery from its initial state. For example, an initial state including patient information, surgical plan, implant characteristics, and surgeon preferences can be defined. Based on this information, the surgeon can preview the entire surgery to confirm that the CASS-recommended plan meets the surgeon's expectations and / or requirements. Moreover, since the output of the machine learning model is the state of the CASS 100 itself, commands can be derived to control the components of the CASS to achieve each predicted state. Therefore, in extreme cases, the entire surgery can be automated based solely on initial state information.

[0175] High-resolution imaging of critical areas is obtained using a point probe during hip surgery.

[0176] The use of point probes is described in U.S. Patent Application No. 14 / 955,742, entitled "Systems and Methods for Planning and Performing Image-Free Implant Revision Surgery," the entire contents of which are incorporated herein by reference. In short, optically tracked point probes can be used to map the actual surface of the target bone where a new implant is needed. Mapping is performed after the removal of defective or worn implants, and after the removal of any diseased or otherwise unwanted bone. Multiple points can be collected on the bone surface by brushing or scraping the remaining bone with the tip of the point probe. This is called tracking or "mapping" the bone. The collected points are used to create a three-dimensional model or surface map of the bone surface in a computer planning system. The created 3D model of the remaining bone is then used as the basis for planning the surgery and determining the necessary implant dimensions. Alternative techniques for determining 3D models using X-rays are described in U.S. Patent Application No. 16 / 387,151, filed April 17, 2019, entitled "Three-Dimensional Selective Bone Matching," and U.S. Patent Application No. 16 / 789,930, filed February 13, 2020, entitled "Three-Dimensional Selective Bone Matching," the entire contents of each of which are incorporated herein by reference.

[0177] For hip applications, point probe mapping can be used to acquire high-resolution data of key areas such as the acetabular rim and acetabular fossa. This allows the surgeon to obtain a detailed view before initiating reaming. For example, in one embodiment, the point probe can be used to identify the floor (fossa) of the acetabulum. As is well known in the art, in hip surgery, it is important to ensure that the floor of the acetabulum is not damaged during reaming to avoid destroying the medial wall. If the medial wall is unintentionally damaged, the surgery will require additional bone grafting steps. With this in mind, information from the point probe can be used to provide operational guidance for the acetabular reamer during the surgical procedure. For example, the acetabular reamer can be configured to provide tactile feedback to the surgeon when the surgeon reaches the floor or otherwise deviates from the surgical plan. Alternatively, the CASS100 can automatically stop the reamer when the floor is reached or when the reamer is within a threshold distance.

[0178] As an additional safeguard, the thickness of the area between the acetabulum and the medial wall can be estimated. For example, once the acetabular rim and acetabular fossa are drawn and registered to the preoperative 3D model, the thickness can be easily estimated by comparing the position of the acetabular surface with that of the medial wall. Using this knowledge, the CASS100 can provide alerts or other responses in case any surgical activity during reaming is anticipated to protrude through the acetabular wall.

[0179] Point probes can also be used to collect high-resolution data of common reference points used when orienting a 3D model to a patient. For example, for pelvic plane landmarks like the ASIS and pubic symphysis, surgeons can use point probes to map the bone to represent the actual pelvic plane. With a more complete view of these landmarks, the registration software will have more information to orient the 3D model.

[0180] Point probes can also be used to collect high-resolution data describing proximal femoral reference points that can be used to improve the accuracy of implant placement. For example, the relationship between the tip of the greater trochanter (GT) and the center of the femoral head is commonly used as a reference point for aligning the femoral components during hip replacement surgery. Alignment height depends on the correct location of the GT; therefore, in some embodiments, point probes are used to map the GT to provide a high-resolution view of the area. Similarly, in some embodiments, a high-resolution view of the lesser trochanter (LT) may be useful. For example, during hip replacement surgery, Dorr classification helps to select the trunk that will maximize the ability to achieve compression fit during surgery, thereby preventing micromovement of the femoral components postoperatively and ensuring optimal bone ingrowth. As understood in the art, Dorr classification measures the ratio between the canal width at the LT and the canal width 10 cm below the LT. The accuracy of classification is highly dependent on the correct location of the relevant anatomical structures. Therefore, mapping the LT to provide a high-resolution view of the area may be advantageous.

[0181] In some embodiments, a point probe is used to map the femoral neck to provide high-resolution data, allowing surgeons to better understand where to make the neck incision. A navigation system can then guide the surgeon as they make the neck incision. For example, as understood in the art, the femoral neck angle is measured by placing a line below the center of the femoral stem and a second line below the center of the femoral neck. Therefore, a high-resolution view of the femoral neck (and possibly the femoral stem) will provide a more accurate calculation of the femoral neck angle.

[0182] High-resolution femoral head and neck data can also be used to navigate resurfacing procedures, where software / hardware assists the surgeon in preparing the proximal femur and placing femoral components. As is generally understood in the art, during hip resurfacing, the femoral head and neck are not removed; instead, the head is trimmed and covered with a smooth metal cap. In this case, it is advantageous for the surgeon to map the femur and cap, allowing for a precise assessment of their respective geometries and its use to guide the trimming and placement of femoral components.

[0183] Preoperative data was registered to the patient's anatomical structures using a point probe.

[0184] As described above, in some embodiments, a 3D model is developed based on 2D or 3D images of the anatomical region of interest during the preoperative phase. In such embodiments, registration between the 3D model and the surgical site is performed prior to the surgical procedure. The registered 3D model can be used to track and measure the patient's anatomy and surgical instruments during surgery.

[0185] During the surgical procedure, landmarks are acquired to facilitate the registration of the preoperative 3D model to the patient's anatomy. For knee surgery, these points may include the femoral head center, distal femoral axis, medial and lateral epicondyles, medial and lateral malleoli, proximal tibial mechanical axis, and tibial A / P direction. For hip surgery, these points may include the anterior superior iliac spine (ASIS), pubic symphysis, points along the acetabular rim and within the hemisphere, greater trochanter (GT), and lesser trochanter (LT).

[0186] In revision surgery, the surgeon may map certain areas containing anatomical defects to better visualize and navigate implant insertion. These defects can be identified based on analysis of preoperative images. For example, in one embodiment, each preoperative image is compared to a library of images showing “healthy” anatomy (i.e., defect-free). Any significant deviation between the patient image and the healthy image can be flagged as a potential defect. During surgery, the surgeon can then be alerted to the potential defect via a visual alert on the CASS 100’s display 125. The surgeon can then map the area to provide the surgical computer 150 with more detailed information about the potential defect.

[0187] In some embodiments, surgeons may use non-contact methods to register incisions within the bone anatomy. For example, in one embodiment, laser scanning is used for registration. A laser strip is projected onto the anatomical region of interest, and changes in the height of that region are detected as changes in the line. Other non-contact optical methods, such as white light interferometry or ultrasound, may also be used alternatively for surface height measurement or registration of anatomical structures. For example, ultrasound may be beneficial where soft tissue exists between the registration point and the bone being registered (e.g., ASIS, pubic symphysis in hip surgery), providing a more precise definition of the anatomical plane.

[0188] Arthroscopic surgery video segmentation

[0189] refer to Figure 8 and 9 The system 5, including an environment 1 for segmenting arthroscopic surgical videos, can automatically segment surgical videos, such as arthroscopic surgical videos, using machine learning models. The system also generates tags associated with one or more segments of the video to allow surgeons or other users to easily navigate through the final recorded video. In one example, segmentation on the video occurs in real time, in parallel with the video display, to provide real-time automatic segmentation during the surgical procedure. System 5 includes an arthroscopic camera 10 that provides raw, unprocessed arthroscopic video data 30. The arthroscopic video data 30 is received by a central control unit (CCU) 20 and / or an artificial intelligence (AI) processing unit 25. In some examples, the AI ​​processing unit may be included as part of the CCU 20, but the AI ​​processing unit 25 may also be a separate device to provide AI processing for segmenting the video in parallel with the processing of the arthroscopic video data 30. In some examples, the arthroscopic camera 10 and the CCU 20 may be part of the same device that generates the video feed for the arthroscopic procedure. Using System 5, the operator can acquire raw, unprocessed arthroscopic video data 30 of the patient's anatomical region of interest, such as a joint, during an arthroscopic surgical procedure. However, System 5 can be used for other anatomical regions of interest. Joints can be, for example, the knee, hip, shoulder, or any other joint or structure.

[0190] In one example, the CCU 20 and / or AI processing unit 25 may be communicatively coupled to (e.g., via network 175) Figure 1 The surgical computer 150 of the CASS shown is illustrated, but the CCU 20 and / or AI processing unit 25 can be used in other CASSes. In an alternative example, the CCU 20 and / or AI processing unit 25 acts as... Figure 1The surgical computer 150 of CASS is shown. The CCU 20 and / or AI processing unit 25 may also be coupled to one or more cloud and / or local network servers or storage devices 50 via one or more communication networks 52. The cloud and / or local network servers or storage devices 50 may provide storage for the processed video 40 and the generated segments or tags as metadata, but the processed video may be stored locally on the AI ​​processing unit 25. The memory or storage devices 50 on the CCU 20 and / or the cloud and / or local network servers and / or the AI ​​processing unit 25 itself may also be used as a repository containing data that enables the machine learning models stored in the AI ​​processing unit 25 to, for example, segment the video and generate tags, as described in further detail below. By way of example only, the data may include various patient anatomy, tools used in arthroscopic surgical procedures, and / or alternative learning models.

[0191] The raw arthroscopic video data 30 is processed into a two-dimensional video feed of the anatomical region within the field of view of the arthroscopic camera 10, as referenced herein. Figure 8-17 The process of generating processed surgical video data 40 is described and illustrated in more detail. For example, a machine learning model can be used to process the video feed to identify segments in the surgical video that relate to different parts of the surgical procedure, and to generate labels associated with these segments and the anatomical structures and tools involved in these segments. These labels, along with the identified anatomical structures and tools in each segment, can be stored along with the original video in a dedicated file and / or in metadata associated with the processed video surgical data 40. Furthermore, digital markers or annotations manually inserted by the surgeon during the surgical procedure can also be stored along with the original video in a dedicated file and / or in metadata associated with the processed video surgical data 40.

[0192] The processed video surgical data 40 can be stored and / or transmitted to other devices, such as a web server or storage device 50 or a surgical computer 150. In one example, the processed video surgical data 40 can be used to display segmented video playback to the surgeon in real time on a display device 60. In one example, the generated segments provide contextual information about the procedure. For example, these segments may relate to a specific interaction between surgical instruments and a portion of the patient's anatomy.

[0193] In this example, arthroscopic camera 10 includes a camera configured to provide video feeds for arthroscopic procedures. In one example, arthroscopic camera 10 may provide high-resolution video feeds such as 4K at a rate of 60 frames per second, but other cameras may be used. In one example, the arthroscopic camera is part of an endoscopy apparatus. The endoscopy apparatus may be part of a system that also includes a CCU 20. An example of a system that includes arthroscopic camera 10 associated with an endoscopy apparatus is LENS from Smith and Nephew, Inc. TM The system. Arthroscopic camera 10 is configured to provide video feed of an anatomical region undergoing arthroscopic surgery. The anatomical region is, for example, the knee and / or hip and / or shoulder and / or limb and / or spine and / or ear, nose, and throat (ENT) applications and / or general surgical applications. In one example, arthroscopic camera 10 has spectral capability. In this example, arthroscopic camera 10 either has this capability built-in or implements this capability via an attachment to the arthroscopic camera.

[0194] To acquire arthroscopic video data 30 using system 5, a human operator positions the arthroscopic camera 10 to provide an anatomical region of interest within the field of view of the arthroscopic camera 10, for example, by placing an endoscope including the arthroscopic camera 10 near the joint. In another example, the computer-assisted surgical system disclosed herein can be used to position the arthroscopic camera 10. The arthroscopic video data 30 may include a video feed that continues throughout the entire duration of the arthroscopic surgical procedure. In this example, the arthroscopic video data 30 is a live stream relative to a static container, such as an mp4 file. The arthroscopic camera 10 may be repositioned during the procedure by a human operator or with computer assistance to obtain arthroscopic video data 30 with different fields of view.

[0195] During an arthroscopic surgical procedure, the CCU 20 and / or AI processing unit 25 receive raw, unprocessed video data 30, generate processed surgical video data 40 including identified segments and associated tags, and store and / or transmit the processed surgical video data 40. In one example, the CCU 20 provides preprocessing of the raw, unprocessed arthroscopic video data 30 before analysis by the AI ​​processing unit 25, but in another example, preprocessing may also be performed by the AI ​​processing unit 25. Using the raw, unprocessed arthroscopic video data 30, in some embodiments, the AI ​​processing unit 25 provides generated tags that include information related to the identified segments. The CCU 20 merges these generated tags with a view of the anatomical regions in the field of view of the arthroscopic camera 10 during the surgical procedure in a dedicated file and / or in metadata associated with the video to create the processed surgical video data 40. Furthermore, digital markers or annotations manually inserted by the surgeon during the surgical procedure may also be stored along with the raw video in a dedicated file and / or in metadata associated with the processed video surgical data 40. The CCU 20 or AI processing unit 25 stores the processed arthroscopic video data 40 for postoperative video processing, or transmits the processed arthroscopic video data 40 to other devices, such as a web server or storage device 50, as described in further detail below. In other examples, the CCU 20 or AI processing unit 25 provides real-time segmented video playback to the display device 60.

[0196] Now for reference Figure 10 The image shows a detailed view of CCU 20. In this example, CCU 20 includes one or more processors 21, a bus 22, a video processing module 24, an AI processing unit 25, and a video stream synthesis module 27, but CCU 20 may include other types and / or numbers of elements or components in other combinations. For example, CCU 20 may include other electronics for video processing (e.g., segmenting or merging video data, downsampling or compressing video data, providing overlay, etc.). One or more of these functions may be executed by the field-programmable gate array (FPGA) of CCU 20, but other hardware logic or programming instructions may be used for these functions.

[0197] The processor 21 of CCU 20 can execute programming instructions stored in memory or any number of functions described and illustrated herein. For example, the processor 21 of CCU 20 may include one or more central processing units (CPUs) or general-purpose processors with one or more processing cores, but other types of processors may also be used. For instance, the processor 21 of CCU 20 controls which information (e.g., generated tags) is integrated into the processed surgical video data 40 and coordinates input data to enable machine learning analysis to be performed in the AI ​​processing unit 25, but the processor 21 in CCU 20 may provide other types and / or numbers of functions. Bus 22 operatively couples the processor 21 to the various peripheral components of CCU 20.

[0198] Video processing module 24 receives raw, unprocessed arthroscopic video data 30. Video processing module 24 is configured to correct and enhance the raw resolution camera video data into a first video data stream 31, which can be provided to video synthesis module 27 for presentation on a display interface during surgery. The first video data stream 31 is a high-resolution, low-latency video stream that provides an anatomical field of view. Video processing module 24 also provides a second video data stream 32 to AI processing unit 25 for machine learning analysis as described below. In some examples, the second video data stream 32 is downsampled to achieve low-latency processing by AI processing unit 25.

[0199] Now for more specific reference Figure 10 and 11 The AI ​​processing unit 25 can perform any number of functions, including processing the second video data stream 32 to identify segments in the video data and generating tags associated with the identified segments, as described below, although the AI ​​processing unit 25 can perform other functions. The AI ​​processing unit 25 generates tag information 33 about anatomical regions and tools in the identified segments of the second video data stream 32. The tag information 33 is associated with analytical information determined using a machine learning model that relates to, for example, tools and anatomical structures in each segment, but the tag information may also include information about numerical markers or annotations associated with that segment. The tag information 33 may be associated with the video content as a dedicated file and / or as metadata associated with the video content.

[0200] Now for more specific reference Figure 11 In this example, the AI ​​processing unit 25 includes a processor 54, a memory 56, and a communication interface 58 coupled together by a bus 64. However, in other examples, the AI ​​processing unit 25 may include other types or numbers of elements in other configurations.

[0201] The processor 54 of the AI ​​processing unit 25 can execute programming instructions stored in the memory 56 of the AI ​​processing unit 25 for any number of functions described and illustrated herein. For example, the processor 54 of the AI ​​processing unit 25 may include one or more central processing units (CPUs) or general-purpose processors with one or more processing cores, but other types of processors may also be used.

[0202] The memory 56 of the AI ​​processing unit 25 stores these programming instructions for one or more parts of the invention technology as described and illustrated herein, but some or all of these programming instructions may be stored elsewhere. Various types of memory storage devices, such as random access memory (RAM), read-only memory (ROM), hard disks, solid-state drives (SSDs), flash memory, and / or any other computer-readable medium read and written by magnetic, optical, or other read-write systems coupled to the processor 54, may be used in the memory 56.

[0203] Therefore, the memory 56 of the AI ​​processing unit 25 can store one or more modules, such as machine learning modules. These modules may include computer-executable instructions that, when executed by the AI ​​processing unit 25, cause the AI ​​processing unit 25 to perform actions, such as executing one or more machine learning models on the second video data stream 32 to, for example, develop tag information 33. These modules can be implemented as components of other modules. Furthermore, these modules can be implemented as applications, operating system extensions, plugins, etc.

[0204] In this particular example, the memory 56 of the AI ​​processing unit 25 includes a video segmentation module 66. In this example, the video segmentation module 66 is configured to receive a second video stream 32, analyze the second video data stream 32, and apply a machine learning model. In one example, the second video data stream 32 is downsampled video, and the video segmentation module 66 is applied to the second video data stream 32 to allow the generation of metadata that can be associated with the real-time high-resolution first video data stream 31. The video segmentation module 66 can apply filtering and other video preprocessing techniques to analyze the second video data stream 32. The operation of the video segmentation module 66 in some examples will be discussed later. Figure 12 and 13 A more detailed description and illustration are provided.

[0205] Return to reference Figure 11In the example, the communication interface 58 of the AI ​​processing unit 25 is operatively coupled between the AI ​​processing unit 25 and the CCU 20, and communicates between them. In an example where the AI ​​processing unit 25 and the CCU 20 are separate, the AI ​​processing unit 25 may be coupled to the CCU 20 by a direct wired connection or a communication network, for example, to feed back segmented information output from the AI ​​processing unit 25 to the CCU 20, but other types of connections or configurations may also be used.

[0206] For example only, the connectivity and / or communication network may include a local area network (LAN) using TCP / IP and industry-standard protocols over Ethernet, but other types or numbers of protocols or communication networks may be used. In this example, the communication network may employ any suitable interface mechanism and network communication technology, including, for example, an Ethernet-based packet data network (PDN).

[0207] The video stream synthesis module 27 receives a high-resolution, low-latency first video data stream 31 and tag information 33 generated by the AI ​​processing unit 25. The video stream synthesis module 27 combines various inputs based on instructions received from the processor 21 of the CCU 20 to generate processed surgical video data 40, optionally including segmentation information generated by the AI ​​processing unit 25 using a machine learning model. For example, the processed surgical video data 40 provides a high-resolution, low-latency video stream including segmentation information generated by the AI ​​processing unit 25. In one example, the processed arthroscopic video data 40 represents a final combined video and / or data stream containing real-time processed full-resolution endoscopic video with additional segmentation information as a dedicated file and / or using associated metadata.

[0208] Although CCU 20 is shown as a single device in this example, in other examples, CCU 20 may include multiple devices, each having one or more processors (each processor having one or more processing cores) that implement one or more steps of this technology. In these examples, one or more of the devices may have dedicated communication interfaces or memory. Alternatively, one or more of the devices may utilize memory, communication interfaces, or other hardware or software components of one or more other devices included in CCU 20. Additionally, in other examples, one or more of the devices that make up CCU 20 (e.g., AI processing unit 25) may be standalone devices or integrated with one or more other devices or equipment. In one example, one or more functions of CCU 20 may be performed by a virtual device.

[0209] For example, one or more of the components depicted in System 5 (e.g., CCU 20) can be configured to operate as virtual instances on the same physical machine. In other words, CCU 20 and AI processing unit 25 can operate on the same physical device, rather than as separate devices communicating via a connection and / or communication network. Additionally, there can be... Figure 9 More or fewer CCU 20 as shown.

[0210] Furthermore, in any example, two or more computing systems or devices can replace any one of the systems or devices. Therefore, the principles and advantages of distributed processing, such as redundancy and replication, can be implemented as needed to improve the robustness and performance of the example devices and systems.

[0211] These examples may also be embodied as one or more non-transient computer-readable media storing instructions for one or more portions of the techniques of the present invention, as described and illustrated herein by way of example. In some examples, these instructions include executable code that, when executed by one or more processors 21 of the CCU 20, causes the processor to perform steps necessary for implementing the examples of this technique described and illustrated herein.

[0212] For more specific reference Figure 12 A flowchart illustrating an exemplary method for improving video segmentation in automated arthroscopic surgery is shown. In step 1200 of this example, the CCU 20 of system 5 obtains raw, unprocessed arthroscopic video data 30 of the arthroscopic procedure. In this example, the arthroscopic video data 30 is captured by arthroscopic camera 10 and transmitted to CCU 20 via a communication network, such as those described and illustrated in more detail herein, that is directly connected to or coupled to CCU 20.

[0213] While the arthroscopic camera 10, which is part of the endoscopic apparatus in this example, can be manually operated, in another example, the apparatus including the arthroscopic camera 10 can also be operated via a robotic arm 105A configured to locate the apparatus associated with the arthroscopic camera 10 relative to an anatomical region of interest associated with the patient and surgical procedure. Although in Figure 12 While shown as a separate step 1200, arthroscopic video data 30 is acquired continuously in the examples described and illustrated herein. Therefore, step 1200 can be performed in parallel with one or more of steps 1202-1212.

[0214] In step 1202, CCU 20 applies video preprocessing techniques to the arthroscopic video data 30 to improve the quality of the output video display, as described below. CCU 20 can apply any known preprocessing technique in this step. In other examples, video preprocessing can be performed in the AI ​​processing unit 25. In one example, one or more field-programmable gate arrays in CCU 20 can be used to perform the preprocessing in step 702. In some examples, video preprocessing techniques may include, for example, removing irrelevant backgrounds and highlighting brighter contours and areas. In one example, the arthroscopic video data 30 is split into a first video data stream 31 and a second video data stream 32 for parallel processing with the video display, wherein the second video data stream 32 is provided to the AI ​​processing unit 25. In one example, the second video data stream 32 is downsampled to achieve low-latency processing by the AI ​​processing unit 25. The AI ​​processing unit 25 of CCU 20 uses depth and / or machine learning algorithms to clean the raw, unprocessed arthroscopic video data 30. Specifically, the input arthroscopic video data can be used to train a machine learning algorithm for arthroscopic video segmentation, and the machine learning algorithm can be optimized based on feedback regarding modifications made to the arthroscopic video data to improve and / or clean the resulting video feed, optionally based on the identified anatomical structures and surgical instruments, as discussed in more detail below. In several other examples, other types of video preprocessing techniques may also be used.

[0215] In step 1204, AI processing unit 25 applies one or more machine learning models to process the second video data stream 32. However, in some examples, AI processing unit 25 may apply one or more machine learning models to unprocessed video data 30. One or more machine learning models are applied to identify segments in the video, as described in the examples herein. Machine learning models may also be applied to historical surgical data stored in AI processing unit 25 or elsewhere (e.g., on any of the servers 50) to provide relevant information about the segmented video, such as historical surgical data related to specific anatomical structures or tools involved in a surgical procedure. Any known machine learning model may be employed in step 1204.

[0216] In step 1206, the AI ​​processing unit 25 identifies segments in the arthroscopic video data. The identified segments relate to specific portions of the surgical video. For example, as described in the examples below, the identified segments may relate to specific interactions between a patient's anatomy (e.g., a portion of the knee joint) and a specific surgical instrument. In another example, the identified segments may relate to digital markers or annotations manually inserted by the surgeon during the surgical procedure. These segments can be identified based on changes in the state of the digital markers or annotations and changes in the presence of instruments in the field of view. In step 1204, the AI ​​processing unit 25 identifies the segments based on information obtained using a machine learning model.

[0217] In step 1208, AI processing unit 25 generates labels associated with the segments identified in step 1206. These labels can identify the start of a new segment, including the timestamp of the frame in which the new segment begins. Labels can be generated for any number of segments in the video, but a minimum segmentation time can be applied to avoid overly fine-grained video segmentation as described above. As described below, the generated labels also include information related to the identified segments.

[0218] exist Figure 13 An exemplary method is illustrated for applying a machine learning model and identifying segments in the surgical video data in steps 1204 through 1208. In step 1300, AI processing unit 25 analyzes frames of the surgical video using a machine learning model. In one example, frames of the surgical video contain a portion of the patient's anatomy (e.g., a portion of the patient's knee joint) and surgical instruments involved in the arthroscopic procedure. By way of example only, frames of the surgical video may also include numerical markers or annotations inserted by the surgeon during the surgical procedure. In one example, the machine learning model is trained using historical surgical data.

[0219] In step 1302, the AI ​​processing unit 25 determines one or more features in the frame being analyzed. For example, the AI ​​processing unit 25 may determine a portion of the patient's anatomical structure in the frame, surgical instruments visible in the frame, the positional relationship between the patient's anatomical structure and the surgical instruments in the frame, or the presence of digital markers or annotations. Figure 14 An exemplary still image of a video frame is shown, comprising a portion of a patient's knee and surgical instruments visible within the frame. AI processing unit 25 automatically identifies specific anatomical structures and the tools involved using a machine learning model. By way of example only, AI processing unit 25 also determines the relationship between the anatomical structures and the tools, such as whether the structures and tools are in contact with each other. In one example, AI processing unit 25 generates labels associated with frames storing the identified features, such as anatomical structures within the frame and tools used within the scene, as well as the specific portion of the anatomical structure the tool is contacting and the frame timestamp.

[0220] Refer again Figure 13 In step 1304, AI processing unit 25 compares the frame analyzed in steps 1300 and 1302 with the previous frame in the video. In one example, AI processing unit 25 may compare the features determined in step 802 between frames, but AI processing unit 25 may provide other comparisons.

[0221] In step 1306, AI processing unit 25 determines whether the state of one or more features has changed based on the comparison performed in step 1304. For example, AI processing unit 25 may determine whether there is a change in the contact between a tool or anatomical structure, or a change in a numerical marker or annotation. AI processing unit 25 applies logic to create new video segments. In one example, the logic assumes a certain degree of smoothness or a minimum number of video segment time frames to avoid producing overly fine-grained video segmentation. AI processing unit 25 may apply logic that assumes contact between a portion of an anatomical structure and a surgical tool to identify video segments, but other logic may be used to identify segments, such as when a tool and an anatomical structure are present on the same screen. If in step 1306, AI processing unit 25 determines a state change, such as a change in contact between an anatomical structure and a tool visible in the frame, then the (Yes) branch is taken to step 1308. In step 1308, AI processing unit identifies new segments in the video and generates labels to identify frames as the start of the new segments. The tag generated in step 808 also includes information related to the patient's anatomy, surgical instruments, and contact that resulted in the creation of the new segment, making the segment easily identifiable. If no state change is identified in step 1306, a No branch is taken, and at step 1300, the exemplary method restarts for the new frame.

[0222] Refer again Figure 12 In step 1210, the AI ​​processing unit 25 stores the tags generated in step 1208. These tags may be stored in a dedicated file or as metadata associated with the video feed. In one example, the video along with the processed tags may be stored on, for example, a network storage device or server 50, but the video and processed tags may be stored locally on the AI ​​processing unit 25 or CCU 20.

[0223] In step 1212, in one example, AI processing unit 25 routes the video back to display device 60 to display the processed video, including the stored metadata. The processed video can be routed back to display device 60 to display the processed video as an overlay on the main video feed. In another example, AI processing unit 25 runs the algorithm in parallel with video processing, and the video is not displayed. For example, the processed video can be later used for video playback for training purposes.

[0224] Now for reference Figure 15 The image shows an example still image of a video playback that can be used to display a processed video feed during a surgical procedure. The video playback includes a playback summary bar containing the entire video sequence. The playback summary bar provides multiple pieces of information related to the video. A current frame indicator along the playback summary bar indicates the current frame of the video being displayed.

[0225] As described above, the event state changes that generate new segments are indicated by vertical lines along the playback summary bar. The placement of event state changes along the playback summary bar indicates the automatic segmentation of the video. In one example, the segmentation of the video between event state changes primarily indicates the interaction between surgical instruments and specific parts of the patient's anatomy.

[0226] Visual indicators may also be located near the playback summary, indicating more than one part of the anatomy involving more than one tool or patient in a specific segment of the video. For example, a visual indicator may be located above or below the playback summary bar. The visual indicator provides the surgeon with a visual reference to know which tools are being used and the main anatomical structures being performed on each segment.

[0227] like Figure 15 As shown, the display may also include a tools and anatomy dashboard. In this example, the tools and anatomy dashboard is located in the upper right corner of the display, although it can be placed in other locations on the screen during video playback. This dashboard provides a catalog of major anatomical structures and tools used during the display of frames and / or segments. In this example, tools and anatomy structures are listed in the dashboard only when they are mentioned in the displayed frame. In another example, the tools and anatomy dashboard provides a comprehensive list of tools and anatomy structures involved throughout the program. In this example, visual indicators, such as color changes or icon additions, may be used to indicate specific features implicit in the currently displayed frame and / or segment.

[0228] Figure 16 An exemplary still image is shown illustrating the playback of a processed surgical video using a graphical user interface (GUI) generated for viewing the processed surgical video. The GUI is used for postoperative viewing and review of the processed surgical video. The GUI provides a set of toolbars (in this example, across the top horizontal panel) that can be used for navigation or to perform functions related to the processed video.

[0229] In this example, the left panel provides an interface that allows users to select specific features to highlight or visually annotate in the video playback display in the right panel. Users can use checkmarks or other selection tools to select specific features in the currently displayed segment. In one example, the user can only select features currently displayed in the frame. Selectable features are shown in groups based on anatomical structures, surgical instruments, or implants, but can be arranged in other ways.

[0230] The right panel of the graphical user interface displays the processed video playback. The monitor provides the actual video footage obtained during the surgical procedure. The visual monitor may also include overlay information, such as annotations or highlights of objects shown on the monitor (e.g., specific anatomical structures, instruments, or implants). In one example, the user can selectively turn the overlay information on and off.

[0231] The graphical user interface includes a playback summary bar. The graphical user interface includes standard tools for playing back video. The playback summary bar indicates when different configurations of anatomical structures and / or tools are present during video playback. The playback summary bar is segmented based on the interaction of anatomical structures and tools in each segment, but the segments can be generated in other ways, such as individually based on variations in anatomical structures or tools. In this example, the playback summary bar includes an indicator indicating the presence of a selected object in the left panel of a segment (e.g., ...). Figure 16 (See the horizontal bar shown). Indicators can be color-coded to match the colors of objects in the left panel. The absence of an indicator indicates that a specific object is not involved in that segment.

[0232] The graphical user interface may also include click and send features, allowing users to click on specific segments along the playback summary bar and providing an option to electronically send a segment of the video to another recipient. Figure 16 In the example shown, the user will select Export from the top panel, which will allow the user to then select the segments to send by clicking the Playback Summary bar. The user can choose to send multiple segments in the same message. The user can confirm the selected segments and can also save or share subsets of the segments.

[0233] Now for reference Figure 17 This image shows another exemplary still image of a search graphical user interface generated by a dedicated program for playing back surgical videos for segmentation, but other configurations are possible. This search graphical user interface provides a similar experience to the one described above. Figure 16The left-hand panel is used for user selection of specific objects. The left-center panel provides an additional set of search criteria selected by the user, but these additional search criteria can also be combined with searchable objects in the left-hand panel. For example, additional search criteria can be removed by clicking the X button associated with each object in the left-center panel. In one example, a user could search for all time periods in a patient where the FLOW50 bar was used on the meniscus of the left knee for more than 60 seconds, but any other search could be performed.

[0234] In this example, the right-center panel of the search graphical user interface indicates videos within a specific search folder or a set of folders that meet the search criteria. These videos are presented by display objects that provide information related to a particular video for ease of use. Display objects may include thumbnails from the video, the date the video was captured, or any other metadata associated with the video, such as the patient's name or the type of surgical procedure. Display objects may also provide a graphical video summary bar indicating the main actions or segments of the video. As discussed below, additional video indicators are provided for selecting a video to indicate that the video is being displayed in the display panel. In this example, the video indicator is a dark bar on the far right of the video being displayed.

[0235] The search graphical user interface also includes a detailed video viewing panel, which in this example is located on the far right of the screen. The viewing panel provides detailed metadata about the processed surgical video. The viewing panel may include a playback summary bar, such as the one mentioned above. Figure 15 and 16 The playback summary panel is discussed. In one example, a user can click and drag the video to scroll through it and view specific sequences of interest. In another example, the video may have standard playback commands for video playback. The video panel allows users to quickly determine whether to open the video file for further interaction. When a video is opened, Figure 16 The graphical user interface will be set on the monitor.

[0236] Using this technology, based on the processing of video feeds, the video output of arthroscopic surgery can be automatically segmented without surgical intervention. The use of machine learning models allows for automatic video segmentation. The technology also provides video segments with searchable parameters associated with each segment. In this way, each segment is easily accessible after surgery for quick navigation to a specific video segment. This allows for locating individual segments for viewing or sharing.

[0237] While various exemplary embodiments incorporating the principles of this teaching have been disclosed, this teaching is not limited to the disclosed embodiments. Rather, this application is intended to cover any variations, uses, or modifications of this teaching and its general principles. Furthermore, this application is intended to cover such deviations from this disclosure that fall within the scope of known or customary practices in the field to which these teachings pertain.

[0238] In the above detailed description, reference is made to the accompanying drawings, which form a part thereof. In the drawings, like symbols generally identify like parts unless the context otherwise requires. The illustrative embodiments described in this disclosure are not intended to be limiting. Other embodiments may be used, and other changes may be made without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that various features of this disclosure (as generally described herein and illustrated in the accompanying drawings) may be arranged, replaced, combined, separated, and designed into a wide variety of different configurations, all of which are expressly contemplated herein.

[0239] This disclosure is not limited to the specific embodiments described herein, which are intended as illustrations of various features. Many modifications and variations can be made without departing from the spirit and scope that will be apparent to those skilled in the art. Functionally equivalent methods and apparatuses within the scope of this disclosure (other than those listed herein) will be apparent to those skilled in the art based on the foregoing description. It should be understood that this disclosure is not limited to specific methods, reagents, compounds, compositions, or biological systems, which can certainly be varied. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.

[0240] Regarding the use of virtually any plural and / or singular terms in this document, those skilled in the art may, at their discretion, convert from plural to singular and / or from singular to plural depending on the context and / or application. For clarity, various singular / plural permutations are explicitly described herein.

[0241] Those skilled in the art will understand that, in general, the terms used herein are intended to be “open-ended” terms (e.g., the term “comprising” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” the term “including” should be interpreted as “including but not limited to,” etc.). While various compositions, methods, and apparatuses are described as “comprising” various components or steps (interpreted as meaning “including but not limited to”), compositions, methods, and apparatuses may also be “consistently composed of various components and steps” or “comprises various components and steps,” and such terms should be interpreted as defining a substantially closed group of components.

[0242] Furthermore, even when a specific number is explicitly stated, those skilled in the art will recognize that such a statement should be interpreted as referring to at least the stated number (e.g., stating "two narratives" without other modifiers means at least two narratives or two or more narratives). Additionally, in cases where terms like "at least one of A, B, and C" are used, this construction is generally intended for those skilled in the art to understand the meaning of the term (e.g., "a system having at least one of A, B, and C" includes, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). In cases where terms like "at least one of A, B, or C" are used, this construction is generally intended for those skilled in the art to understand the meaning of the term (e.g., "a system having at least one of A, B, or C" includes, but is not limited to, systems having only A, only B, only C, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). Those skilled in the art will also understand that virtually any transition words and / or phrases presenting two or more alternative terms, whether in the specification, sample embodiments, or drawings, should be understood to account for the possibility of including one, any one, or both of the terms. For example, the phrase "A or B" will be understood to include the possibility of including "A" or "B" or "A and B".

[0243] Furthermore, in the context of the features of this disclosure being described in accordance with the Markush Group, those skilled in the art will recognize that this disclosure is also based on any individual member of the Markush Group or a subgroup of its members.

[0244] Those skilled in the art will understand that, for any and all purposes, such as for providing a written description, all scopes disclosed herein also cover any possible subscopes and all possible combinations of subscopes and their subscopes. Any listed scope can be readily considered sufficiently descriptive and realized by decomposition into at least equal halves, thirds, quarters, fifths, tenths, etc., of the same scope. As a non-limiting example, each scope discussed herein can be readily decomposed into a lower third, a middle third, and an upper third, etc. Those skilled in the art will also understand that all language such as “reach,” “at least,” etc., includes the numbers stated and refers to a scope that can subsequently be decomposed into subscopes as described above. Finally, those skilled in the art will understand that a scope includes each individual member. Thus, for example, a group having 1-3 components means a group having 1, 2, or 3 components. Similarly, a group having 1-5 components means a group having 1, 2, 3, 4, or 5 components, and so on.

[0245] As used herein, the term "about" refers to a change in a numerical quantity that can occur, for example, through measurement or processing procedures in the real world, through unintentional errors in these procedures, through differences in the manufacture, origin, or purity of the composition or reagent, etc. Generally, the term "about" as used herein refers to a value or range of values ​​that is greater than or less than 1 / 10 (e.g., ±10%) of the stated value. The term "about" also refers to variations that will be understood by those skilled in the art as equivalents, provided that such variations do not contain values ​​known in prior art practice. Each value or range of values ​​following the term "about" is also intended to cover embodiments of the absolute value or range of values. Whether or not modified by the term "about," quantitative values ​​referenced in this disclosure include equivalents to the referenced values, such as possible numerical variations of such values, but those skilled in the art will recognize the equivalents.

[0246] The various features and functions disclosed above, as well as their alternatives, can be combined into many other different systems or applications. Those skilled in the art can then make various currently unforeseen or unintended alternatives, modifications, variations, or improvements, each of which is also intended to be covered by the disclosed embodiments.

Claims

1. A method for automatically segmenting surgical video data, the method comprising: The video data is obtained by a computing device from a camera configured to capture video data of a surgical procedure, wherein the video data includes a field of view of the patient’s anatomical regions during the surgical procedure; The computing device generates a first video data stream and a second video data stream; One or more machine learning models are applied to the second video data stream by an AI processing unit associated with the computing device; The AI ​​processing unit identifies one or more segments of the second video data stream based on the applied machine learning model; as well as The AI ​​processing unit generates a set of tags associated with one or more segments of the second video data. The identification of one or more segments of the second video data stream further includes: The AI ​​processing unit analyzes each of the multiple frames in the second video data stream; The AI ​​processing unit determines one or more features present in each of the plurality of video frames in the second video data stream based on the applied machine learning model; and The AI ​​processing unit identifies one or more segments of the second video data stream based on one or more features present in each of the multiple video frames in the identified second video data stream; The AI ​​processing unit compares each of the plurality of frames with the previous frame of the plurality of frames based on one or more existing features. The AI ​​processing unit identifies a state change in one of the one or more features based on the comparison; and The AI ​​processing unit identifies new segments in the second video data stream based on the identified state changes.

2. The method according to claim 1, further comprising: The computing device associates the set of tags with the first video data stream; as well as The computing device generates merged video feed data including one or more tags on the first video data stream.

3. The method according to claim 2, wherein the set of tags is stored in the video metadata of the first video data stream.

4. The method according to claim 2, further comprising: The computing device outputs the merged video feed data for display on the display interface, wherein the merged video feed data is output in real time.

5. The method according to any one of claims 1-4, wherein the surgical procedure includes an arthroscopic surgical procedure.

6. The method according to any one of claims 1-4, wherein the camera is located on the endoscope device.

7. The method of claim 1, wherein one or more features present in each of the plurality of video frames include anatomical structures associated with the surgical procedure and one or more of tools, digital markers, digital annotations, or implants associated with the surgical procedure.

8. The method of claim 1, wherein one or more features present in each of the plurality of video frames include anatomical structures associated with the surgical procedure and one or more of tools, digital markers, digital annotations, or implants associated with the surgical procedure.

9. The method of claim 8, wherein the state change includes a change in the contact between the anatomical structure and the tool associated with the surgical procedure and / or a change in the presence of the anatomical structure and the tool associated with the surgical procedure, or a change in one or more of the tool, the digital marker, the digital annotation, or the implant.

10. The method according to any one of claims 1-4, further comprising: Before the AI ​​processing unit applies the one or more machine learning models, the computing device downsamples the second video data stream.

11. A video data segmentation system, comprising: A camera, configured to capture video data; as well as A computing device, the computing device including a first processor coupled to a memory and configured to execute programming instructions stored in the first memory to perform the method according to any one of claims 1 to 10.

12. The system of claim 11, wherein the camera is located on the endoscope device.

13. A non-transient computer-readable medium storing instructions for automatically segmenting surgical video data, the instructions including executable code that, when executed by one or more processors, causes the one or more processors to perform the method according to any one of claims 1 to 10.

Images