Systems and methods for surgical procedure guidance

Abstract

Examples relate to apparatuses for guiding a surgical procedure. An example apparatus comprises: a communication interface for communicating with an artificial intelligence assistant the communication interface being adapted i. to receive input data for the artificial intelligence assistant, and ii. to receive an output data from the artificial intelligence assistant; and surgical assistance circuitry: i. to generate the input data for the artificial intelligence assistant from data associated with at least one input modality peripheral, and ii. to process the output data from the artificial intelligence assistant to generate output data associated with at least one output modality peripheral.

Description

SYSTEMS AND METHODS FOR SURGICAL PROCEDURE GUIDANCETECHNICAL FIELD

[0001] The present disclosure generally relates to methods, systems, and apparatuses for guiding surgical procedures such as, for example, computer-assisted surgical systems, as well as such procedures per se, and, in particular, to an artificial intelligence system for supporting such surgical procedures.BACKGROUND

[0002] Surgery comprises complex and delicate processes that requires a deep understanding of the medical device instruments and implants used as well as adherence to step-by-step instructions, together with precision and accuracy in execution. To ensure successful outcomes, surgeons rely on specific surgical techniques provided by medical device manufacturers and can also use feedback from various imaging technologies such as cameras, x-rays and audio communications and sensor data to navigate and perform surgeries. Performing these steps from memory, or encountering patient specific complications, can lead to mistakes that increase the risk of adverse consequences for patients. For example, surgeons practicing at a community hospital may perform a specific surgical procedure relatively infrequently. The lack of regular exposure to the technique, instruments, and patient variability increases the risk of poor outcomes or worse such as, for example, surgical injury.

[0003] For instance, patella and patella tracking problems account for a substantive number of poor total knee arthroplasty (TKA) outcomes in patients. It is well-stablished that anterior knee pain after a TKA can be a typical post-surgery patellofemoral complication that may be attributed to issues with a native or implant patella.

[0004] Currently, robot-assisted TKA or computer-assisted TKA may be used in preparing the femur and tibia such as, for example, optimizing implant placement, leg alignment, soft tissue balancing, range of motion assessment, etc. Although patella mal-positioning can result in patellar maltracking and, ultimately, pain and other significant complications, the patella is currently not addressed in conventional computer-assisted TKA.

[0005] For conventional TKA, including computer-assisted TKA, the prominent surgical technique for patellar replacement is a free-hand technique. For example, before making the patellar cut, the surgeon measures the native or pre-cut patellar thickness with a surgical caliper and then makes the cut. After the resection, the surgeon measures the post-cut patella thickness. The goal is to restore native patellar thickness, which may be approximately 25 millimeters (mm) for a male patella and 22 mm for a female patella. Surgeons may resect the patella just under the articular surface and not cut the patella to a certain minimum thickness, such as less than about 12-15 mm.

[0006] Using conventional techniques, the surgeon assumes that the patella, and patellar tracking, will be adequate post-surgery if the femoral and tibial components are placed correctly. However, misplacement of the patella can lead to various post-operative difficulties, including patella fracture, patellar clunk syndrome, patellofemoral instability, patellar maltracking (i.e., patellar tracking disorder), patella alta, patella baja, and patellar malrotation. The conventional free hand technique does not target or account for the intricate biomechanics of the patellofemoral joint.

[0007] For example, during a TKA, surgeons are required to simultaneously process complex information and make a plurality of difficult decisions in the operating room in rapid succession. A patellar cut must be planned and executed with an adequate cut thickness and an appropriate implant size, and a design must be selected for the patellar component. Additionally, a position and orientation of the patellar component must be selected in conjunction with femoral and tibial component positions and orientations. Furthermore, the biomechanics of the patellofemoral joint may be determined in an effort to restore the normal function of the knee joint. Given the complexity of the patellofemoral anatomy and biomechanics, there is a high risk of patellofemoral complications because surgeons are required to make these various planning decisions intraoperatively with minimal available information and tools.

[0008] It can be appreciated that the foregoing is a typical example of such complex and delicate processes that require such a deep understanding of the surgical tools andtechniques as well as the risks that might arise, which places a significant burden on at least one, or both, of: the surgeon and operating room staff.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] By way of illustration, specific examples will now be described, with reference to the accompanying drawings, in which:

[0010] FIG. 1 depicts an operating theatre including an illustrative computer-assisted surgical system (CASS) in accordance with one or more features of the present disclosure;

[0011] FIG. 2 illustrates a view of a surgical computer and artificial intelligence assistant;

[0012] FIG. 3 shows a view of further example of the surgical computer having the artificial intelligence system integrated;

[0013] FIG. 4 depicts a view of a pair of flowcharts for the interactions between the surgical computer and the artificial intelligence assistant; and

[0014] FIG. 5 illustrates a view of machine-readable storage storing machine-instructions for realising a surgical computer.DETAILED DESCRIPTION

[0015] Referring to FIG. 1, there is shown a view 100 of a CASS 102 according to an example. In the example depicted, the CASS 102 is arranged to aid surgeons in performing orthopedic surgical procedures such as, for example, a knee arthroplasty (e.g., total knee arthroplasty (TKA)) or total hip arthroplasty (THA). An effector platform 104 positions surgical tools relative to a patient during surgery. For example, for a knee surgery, the effector platform 104 may include a robotic arm 104A and / or an end effector 104B that holds surgical tools or instruments during their use. Effector platform 104 can include a limb positioner 104C for positioning the patient’s limbs during surgery. Resection equipment (not shown in FIG. 1) can be provided to perform bone or tissue resection using, for example, mechanical, ultrasonic, or laser techniques. The effector platform 104 can also include a cutting guide or jig 104D that is used to guide saws or drills used to resect tissue during surgery. Such cutting guides 104D can be formed integrally as part of theeffector platform 104 or as part of the robotic arm 104A, or cutting guides can be separate structures that can be matingly and / or removably attached to the effector platform 104 or robotic arm 104A.

[0016] The CASS 102 comprises an optical tracking system 106 that uses one or more sensors to collect real-time position data that locates the patient’s anatomy and surgical instruments. Any suitable tracking system can be used for tracking surgical objects and patient anatomy in the surgical theatre. For example, a combination of infrared (IR) and visible light cameras can be used in an array. Such an optical tracking system 106 can use the EMR retro-reflected from any of the retro-reflectors to determine real-time position data that locates at least one, or both, of: the patient’s anatomy and surgical instruments.

[0017] Accordingly, the CASS 102 shown in FIG. 1 depicts a number of markers and / or marker arrays comprising respective retro-reflectors. The marker arrays can be placed on objects or body parts to be tracked or for which respective positions and / or orientations are to be determined. Therefore, a set of retro-reflectors can be used to determine at least one, or both, of: position and orientation of a respective item. As indicated above such a set of retro-reflectors are formed as a marker array or as a marker. The respective item can be an object or body part. A set of retro-reflectors can comprise one retro-reflector or a number of retro-reflectors. Examples can be realised in which a set of retro-reflectors comprises at least three retro-reflectors. For example, a first marker array 114 is situated on the robotic arm 104 A. Knowing the position of the first marker 114 can allow, for example, the position and orientation of the actuator 116 of the robotic arm 104A to be determined. A second marker array 118 is placed on the handheld tool 104B to allow the position and orientation of the tool 104B to be determined and / or tracked in 3D space. A third marker array 120 can be situated relative to the jig 104D to allow the position and orientation of the jig 104D to be determined. A fourth marker array 122 can be placed on the limb positioner 104C to assist in determining the position of a respective distal actuator 124 for holding a limb. A fifth marker array 126 can be placed on eye-wear 128 of a surgeon 130. A sixth marker array 127 can be located on the operating table 104E.

[0018] Although the CASS 102 has been described with reference to a set of marker arrays comprising six marker arrays, examples are not limited thereto. Examples can be realised in which such a set of marker arrays comprises one or more than one marker array to suit the needs of the operation to be performed. Still further, the deployment of the marker arrays can be realised other than in relation to the robotic arm 104A, the handheld tool 104B, the jig 104D and the limb positioner 104C.

[0019] Although the above has described a one to one relationship between an object or body part and a respective marker array, examples can be realised in which an object or body part can comprise a set of markers arrays. Such a set of marker arrays per object or body part can be used to improve or increase the accuracy with which at least one, or both, of: position and orientation can be determined.

[0020] The registration process that registers the CASS 102 to the relevant anatomy of the patient can also involve the use of anatomical landmarks, such as landmarks on a bone or cartilage. For example, the CASS 102 can include a 3D model of the relevant bone or joint and the surgeon can intraoperatively collect data regarding the location of bony landmarks on the patient’s actual bone using a probe that is connected to the CASS. Alternatively, the CASS 102 can construct a 3D model of the bone or joint without pre-operative image data by using location data of bony landmarks and the bone surface that are collected by the surgeon using a CASS probe or other means.

[0021] A tissue navigation system (not shown in FIG. 1) provides the surgeon with intraoperative, real-time visualization for the patient’s bone, cartilage, muscle, nervous, and / or vascular tissues surrounding the surgical area.

[0022] The CASS 102 can also comprise a display 108 to provide graphical user interfaces (GUIs) that display images collected by the tissue navigation system as well other information relevant to the surgery to the surgeon 130 or other operating threatre staff. For example, the display 108 can overlay image information collected from various modalities (e.g., CT, MRI, X-ray, fluorescent, ultrasound, etc.) collected pre-operatively or intraoperatively to give the surgeon various views of the patient’s anatomy as well as real-timeconditions. A surgical computer 150 provides control instructions to various components of the CASS 102, collects data from those components, and provides general processing for various data needed during surgery. In the example depicted in FIG. 1 , the surgeon 130 is shown as wearing the protective eye- wear or an augmented reality headset 128, which can also comprise or bear the respective marker 126. The surgeon also has a microphone 132 and a set of wireless earphones 134 or the like.

[0023] Although the retro-reflective entities described with reference to FIG. 1 above were marker arrays, examples are not limited thereto. Examples can be realised in which one or more, or all, of the marker arrays are markers instead,

[0024] The surgical computer 150 provides control instructions to various components of the CASS 100, collects data from those components, and provides general processing for various data needed during surgery. The surgical computer 150 is also responsible to managing communications with an artificial intelligence assistant (AIA) 152, which will be described in greater detail with reference to FIGs. 2 to 5 taken jointly and severally.

[0025] Referring to FIG. 2, there is shown a view 200 of the surgical computer 150 and the artificial intelligence system 152. The surgical computer 150 comprises software 154 for managing communications and data exchanges with the artificial intelligence assistant 152. The software 154 is an examples of “surgical assistance surgery”. Examples can be realised in which the artificial intelligence assistant 152 is remote from the surgical computer 150, that is, the artificial intelligence assistant 152 is accessible via, for example, a network 204. The surgical computer 150 comprises communication hardware and software 206, which is an example of communication circuitry. The term “communication circuitry” is used herein refers to such a combination of hardware and software for realising networked communications. The communication circuitry 206 communicates with the artificial intelligence assistant 152 via a respective input / output interface 208.

[0026] The surgical computer 150 comprises a further input output interface 210. The further input / output interface 210, in practice, represents a collection or set of input output interfaces arranged for interfacing, or otherwise exchanging data with, respective entitiesof a set of peripherals 212. The set of peripherals 212 can comprise one, or more than one, peripheral. In the example depicted in FIG. 2, the set of peripherals comprises 12 peripherals. Examples can be realised in which the set of peripherals 212 comprises at least one, or more, of the following taken jointly and severally in any and all permutations:

[0027] a stereoscopic camera 214, at least one audio interface 216, and augmented reality headset 218, at least one accelerometer 220, at least one pressure sensor 222, at least one thermometer 224, an optical tracking system 226, at least one surgical tool 228, a display 230, a robotic arm 232, a limb holder 234, an operating table 236 and an audio output device such as, for example, earphones or similar headset 237.

[0028] In general, the surgical computer 150 is arranged, via the software 154, to send a query 238 to the artificial intelligence assistant 152 in response to a set of input data 240 received from the set of peripherals 212. The set of input data 240 can comprise data from any of the above described peripherals taken jointly and severally in any and all permutations. Therefore, the set of input data 240 is an example of fused data, that is, a fusion of data from one or multiple peripherals, which can be known or referred to as unimodality data that forms a uni-modality data set or multi-modality data that forms a multimodality input data set. Accordingly, the set of input data 240 can be uni-modal or multimodal. The query 238 is an example of output data from the perspective of the surgical computer 150, or, more particularly, from the software 154 of the surgical computer 150. The query 238 is derived from the set of input data 240. Therefore, the query 238 can be realised as a uni-modal query or a multi-modal query according to the set of input data 240 being uni-modal data or multi-modal data respectively.

[0029] The artificial intelligence assistant 152 comprises artificial intelligence circuitry 242. Again, the term “circuitry” as used herein refers to either hardware or software, or to a combination of hardware and software. The artificial intelligence circuitry 242 can be realised as at least one, or both, of hardware and software. The artificial intelligence assistant 152 comprises communication circuitry 244. The communication circuitry 244 is arranged to manage communications and data exchanges, via a respective input output interface 246, with the surgical computer 150. Examples of the artificial intelligenceassistant 152 can be realised that comprise a large language model 248. Examples of such an LLM 248 comprise, for example, ChatGPT3, ChatGPT4, Google Bard, or the like. The large language model 248 can be tailored, or otherwise configured, to respond to queries in the medical field. Examples can be realised in which the large language model 248 is arranged to respond to queries relating to activities or actions arising from the operating room. Such activities or actions are represented in the fused data 240, in particular from the query that comprises data 250 derived from, or representing, the set of input data 240.

[0030] The artificial intelligence assistant 152 is arranged to receive the query 238. The large language model 248 is arranged to process the received query 238 to generate a respective response 252. The response 252 is output or otherwise sent, via the communications circuitry 244 and input / output interface 246, to the surgical computer 150. The response 252 will have, or can have, taken into account one or more than one of the data modalities, that is, the types of data, presented within or contained within data 250 derived from the set of input data 240. Examples of such different modalities of input data within the set of input data 240 and respective responses generated by the large language model 248 within a given context will be described below.

[0031] The software 154 of the surgical computer 150 is arranged to receive, via the communication circuitry 206 and input / output interface 208, the response 252. The response comprises respective response data 254 that reflects the response to the LLM 248 to the query, in particular, to the data 250 contained within the query. The software 154 is arranged to generate a set of output data output data 256. The output data 256 comprises data suitable for processing by, or otherwise being output to, a respective set of peripherals of the set of peripherals 212. The respective set of peripherals of the set of peripherals 212 can comprise one peripheral or multiple peripherals of the set of peripherals 212. The output data 256 can be output to respective peripherals of the set of peripherals 212. The set of peripherals 212 used to process the output data 256 will depend on the contents of the response 252, in particular, the content of the response data 254. Examples of the output data 256 being directed to respective peripherals of the set of peripherals 212 will be described below. For example, the data 254 of the response 252 may contain text baseddirections for performing a surgical action. The text based directions are intended to be output to the surgeon. Outputting the text based directions to the surgeon can be realised using one or more than one modality. For instance, the text based directions can be displayed on the display 230. Alternatively, or additionally, the text based directions can be converted to speech that is output via one or more than one audio device. Examples can be realised in which the text data can be displayed on, for example, a display of the augmented reality headset 218 or on a more generic display such as the above mentioned display 230, or converted to speech for output via the earphones 237.

[0032] The set of input data 240 can comprise data representing a query generated by the surgeon. That query may have been issued, for example, using the microphone 216. The microphone 216 will generate audio or speech data (not shown). The speech data can be converted to a format suitable for processing by, or submission to, the large language model 248. Examples can be realised in which that format is text. Alternatively, or additionally, that format can comprise a digital representation of the speech uttered by, for example, the surgeon.

[0033] Referring to figure 3, there is shown a view 300 of the surgical computer 150 having the artificial intelligence assistant 152 formed as an integral part thereof. Reference numerals common to FIGs. 2 and 3 refer to the same entity and have the same operation. Therefore, the surgical computer 150 comprises the above described software 154, the communication circuitry 206, and the input output interface 210, as well as the artificial intelligence assistant 152 and associated software 242.

[0034] The interaction between the software 154, the set of peripherals 212 and the artificial intelligence assistant is substantially the same as described above with reference to FIG. 2.

[0035] Referring to figure 4, there is shown a view 400 of a pair of flowcharts 402a and 402b describing processing undertaken by the software 154 and the artificial intelligence system 152 respectively.

[0036] Referring to the left-hand flowchart 402a, at 404, the set of input data 240 is received. The set of input data 240 is processed, at 406, to create the query 238 in a format that is suitable for processing by the artificial intelligence assistant 152 and, more particularly, by the large language model 248. More specifically, the set of input data 240 is expressed in the form of data 250 that is suitable for processing by the LLM 248. Examples can be realised in which the LLM 248 is multi-modal such that the set of input data 240 can merely be copied to data 250 for the LLM 248. The query 238 is output or otherwise transmitted or communicated to the artificial intelligence assistant 152 at 408. At 410, the response 252, comprising the response data 254, is received by the surgical computer 152. The software 154 of the surgical computer 152 processes the response 250, in particular the response data 254, at 412, to generate the set of output data 256. The set of output data 256, as indicated above, may comprise uni-modal data intended to be output via a single peripheral of the set of peripherals 212 or multi-modal data intended to be output by a respective set, or subset, of the set of peripherals 212. The set of output data 256 is output, via the communication circuitry 206 and input output interface 210, to a respective peripheral of the set of peripherals 212 or to a respective set or subset of the set of peripherals 212 at 414.

[0037] Referring to the right-hand flowchart 402b, the query 238 is received by the artificial intelligence assistant 152 at 416. At 418, the received query 238 is submitted to, or is otherwise processed by, the large language model 248. The large language model 248 is arranged to generate, at 420, the response 252 to the query 240. The response 252 comprises the response data 254. The response 252 is output by the artificial intelligence assistant 152 for transmission to the surgical computer 150 at 422.

[0038] The functionality of the surgical computer 150, the surgical computer software 154, the artificial intelligence assistant 152 and the large language model 248 can be realised using machine instructions that can be processed by a machine comprising, or having access, to the instructions. The machine can comprise a computer, processor, processor core, DSP, a special purpose processor implementing the instructions such as, for example, an LPGA or an ASIC, circuitry or other logic, a compiler, a translator, an interpreter or anyother instruction processor. Processing the instructions can comprise interpreting, executing, converting, translating or otherwise giving effect to the instructions. The instructions can be stored using a machine readable medium, which is an example of machine-readable storage. The machine-readable medium can store the instructions in a non-volatile, non-transient, manner or in a volatile, transient, manner. The instructions can be arranged to give effect to any and all operations described herein taken jointly and severally in any and all permutations. The instructions can be arranged to give effect to any and all of the operations, devices, systems, flowcharts, and methods described herein taken jointly and severally in any and all permutations. In particular, the machine instructions can give effect to, or otherwise implement, the operations of the flowcharts, taken jointly and severally, depicted in, or described with reference to, figure 4, or any of the entities shown in any of the figures.

[0039] Therefore, FIG. 5 shows a view 500 of machine instructions 502 stored using machine readable storage 504 for implementing the examples described herein. The machine instructions 502 can be processed by, for example, a processor 506 or other processing entity, such as, for example, an interpreter, as indicated above.

[0040] The machine instructions 502 comprise at least one, or more than one, of the following taken jointly and severally in any and all permutations:

[0041] machine instructions 508 for receiving the set of input data 240 from the set of peripherals 212;

[0042] machine instructions 510 to process the set of input data to generate the query 238 to be submitted to the artificial intelligence assistant 152;

[0043] machine instructions 512 to submit the query 238 to the artificial intelligence assistant 152;

[0044] machine instructions 514 to receive the response 252 from the artificial intelligence assistant 152;

[0045] machine instructions 516 to generate the output data 256 to be output to the set of peripherals 212, or to a subset of the set of peripherals; and

[0046] machine instructions to 518 to output the output data 256 to the set, or subset, of peripherals 212.

[0047] Examples can be realised in which the machine readable instructions also, or alternatively, comprise at least one or more than one of the following taken jointly and severally in any and all permutations:

[0048] machine instructions 520 to receive the query 238 from the surgical computer 150;

[0049] machine instructions 522 to process the received query 238;

[0050] machine instructions 524 to generate the response 252 to the received query 238; and

[0051] machine instructions 526 to output the response 252 to the surgical computer 150.

[0052] It can be appreciated that the systems and methods presented herein at least reduce, or eliminate, surgical error, which improve patient outcomes through enhanced information access and creative problem solving by the surgeon, or other operating room staff, involved in preoperative and intraoperative surgical planning and execution though access to natural language text, voice and vision systems in conjunction with the artificial intelligence assistant 152.

[0053] As indicated above, surgery is a complex and delicate process that requires a deep understanding of the medical device instruments and implants, adherence to a step-by- step instruction, precision and accuracy. To ensure successful outcomes, surgeons rely on specific surgical techniques provided by the medical device manufacturer and feedback from various imaging technologies such as cameras, x-rays and audio communications and sensor data to navigate and perform surgeries.

[0054] Numerous examples will now be given below of the use of the systems and methods described above.

[0055] Example 1

[0056] The first example will be described with reference to interrogatories issued by the surgeon and responses issued by the artificial intelligence system 152.

[0057] The surgeon 130 speaks into the microphone 216: “I’m about to implant the IM Nail in the intramedullary canal, what are some tips to ensure a safe and effective implantation?” The microphone 216 can be considered to be an Al-enabled microphone or an LLM-enabled microphone, that is, the output of the microphone can form an input into a large language model, either directly or via the agency of an intermediary system such as, for example, the surgical computer 150.

[0058] The surgical computer 150 will convert the surgeon’s recorded utterance into a format suitable for processing by the LLM 248. In the example described, the surgical computer 150 can convert the speech into corresponding text. The text will form the basis of the query 238, more particularly the data 250 within the query 238. The surgical computer 150 will send the query 238 to the artificial intelligence assistant 152.

[0059] Upon receiving the query 238, the artificial intelligence assistant 152 processes, via the LLM 248, the data 250 contained within the query 238 and generates a response 252 containing the response data 254. The response 252 is sent to the surgical computer 150, where it is processed or otherwise converted to output data 256 in a format suitable for output using a subset one or a plurality of the set of peripherals 212. Having performed that conversion, the surgical computer 150 outputs the response to the subset of peripherals.

[0060] The conversion process can use a modality conversion table 258. The modality conversion table 258 can be configured with mapping data 260 to contain mappings 262 to 266 between one or more data formats or modalities contained within the response data 254 output by the LLM 248 and one or more data formats or modalities of the data formats or modalities associated with the set, or a subset, of the set of peripherals 212. In the example depicted in FIG. 2, the mappings 262 to 266 have been shown as comprising three mappings 262 to 266. The mappings 262 represents a response data format, on the right-hand side, and an output data format on the left-hand sode for converting one ormore than one set of data (not shown) within the response data 254 into one or more than set of data to form the output data 256.

[0061] For example, the response data 254 might contain text. The first mapping 262 may reflect a mapping between any received text data and at least one, or both, of: a text output data format and an audio output data format. In such a case, the output data 256 will comprise: (1) output data reflecting the conversion of the received text data of the response data 254 to a text format suitable for output by one of the peripherals that can display text and (2) output data in the form of speech or audio data that is derived from the received text data of the response data 254.

[0062] Assuming that the subset of peripherals comprises the earphones 237. The format of the output data 256 will be audio data to be played through the earphones 237.Therefore, the surgical computer 152 outputs audio data as the output data 256 for output via the earphones 237. Consequently, the surgeon 130 might hear an audio response.The audio response might be, for example, the following:

[0063] “During inserting an intramedullary nail, several complications can arise. Some of the common complications include:

[0064] 1. Fracture: The insertion of the nail can cause a fracture in the bone. This can happen if the nail is too large or if too much force is used during the insertion process.

[0065] 2. Malpositioning of the nail: Improper placement of the nail can lead to malalignment of the bone, which can result in poor healing or a non-union of the fracture.

[0066] 3. Nerve or blood vessel damage: The insertion of the nail can damage nerves or blood vessels around the bone, leading to numbness, weakness, or even loss of sensation or function in the affected limb.

[0067] 4. Infection: The surgery can introduce bacteria into the bone or surrounding tissue, leading to infection.

[0068] 5. Non-union or delayed healing: In some cases, the bone may not heal properly or may take longer than expected to heal after the nail is inserted.”

[0069] Alternatively, or additionally, the response can also be displayed on the display 230. The surgical computer 150 can be configured to construct the output data 256 in the form of multi-modality output data that outputs the response 252 in various different formats using different peripherals of the set of peripherals 212. For example, the response 252 might comprise the above text detailing points 1 to 5 as the response data 254. The surgical computer 150 can convert the response data 254 into an audio format that is suitable for output via an audio output device such as, for example, the earphones 237 or other type of speaker, and into a visual for such as, for example, text or graphics suitable for display on the at least one, or both, of: the display 230 or augmented reality headset 218, or any other type of display. The mapping of the response data 254 into one or more output modality formats can be reflected by the above described modality conversion table 258.

[0070] Having heard and / or seen the response, the surgeon 130 may choose to continue the dialogue with a further query, whereupon the query-response cycle repeats. For instance, the surgeon might ask “What can I do to minimise embolism during nail insertion?”

[0071] The surgical computer 150 formats the set of input data 240 representing the question “What can I do to minimise embolism during nail insertion?” into the data 250 for the query 238. The query 238 is sent to the artificial intelligence assistant 152 where the data 250 is processed by the LLM 248. The response data 254 that represents the response to processing the data 250 by the LLM 248 is output via the response 252 for processing by the surgical computer 150, in particular, the software 154. Again, the modalities conversion table 258 is used to determine a suitable output format or multiple suitable formats for the output data 256 as expressed by the set, or a subset, of the mappings 262 to 266. The set, or subset, of the mappings 262 to 266 is used to generate the output data 256 in a format suitable for output by the set, or a subset, of the set of peripherals 212. Assume in the present example that the second mapping is arranged to map text of the response data 254 to audio or speech data to be output by the earphones237. Therefore, the output data 254 will comprise audio or speech data that is directed for output via the earphones 237.

[0072] Therefore, the surgeon 130 might hear audio data to the effect of the following:

[0073] “ There are several steps that can be taken to minimize the risk of embolism during intramedullary nailing of the femur:

[0074] 6. Careful technique: The surgeon should use careful and precise techniques during the procedure to minimize the amount of debris generated and avoid damage to the medullary canal.

[0075] 7. Reaming: The surgeon should use proper reaming techniques, including careful selection of the size of the reamer and the speed of reaming, to minimize the amount of debris generated.

[0076] 8. Minimally invasive surgery: Minimally invasive techniques can be used to reduce the risk of embolism. For example, small incisions and specialized tools can be used to minimize tissue damage and reduce the amount of debris generated during the procedure.

[0077] 9. Anti-embolism stockings: The patient may be fitted with anti-embolism stockings to help prevent blood clots from forming in the legs.

[0078] 10. Pharmacological prophylaxis: The patient may be given medications, such as anticoagulants, to help prevent blood clots from forming.

[0079] 11. Early mobilization: Early mobilization after the procedure can help reduce the risk of blood clots by promoting blood flow and preventing stasis.”

[0080] The above response can, again, be converted to any format suitable for output via one or more of a respective subset of peripherals of the set of peripherals 212.

[0081] Example 2

[0082] A further example will be described within the context of at least one video feed being supplied to the artificial intelligence assistant 152 that solicits a response from the LLM 248.

[0083] The operating room can be equipped with one or more than one camera such as, for example, the above described camera 214. The camera 214 can be a stereoscopiccamera. The camera 214 can be arranged to generate the set of input data 240 in the form of a video stream. The video stream can be converted to a format suitable for processing by the artificial intelligence assistant 152 and output as the data 250 within the query 238. In this case, the query 238 may comprise instructions associated with the video stream. The instructions can be instructions from, for example, the surgeon 130 to “Monitor this knee arthroplasty operation for anomalies”.

[0084] The data 250 is received and processed by the LLM 248 to monitor for anomalies. The anomalies can comprise at least one, or more than one, of the following taken jointly and severally in any and all permutations: a biological anomaly, a physiological anomaly, a surgical procedural anomaly, an instrument selection error, an instrument use error and the like.

[0085] Upon detecting any anomaly, the LLM 248 is arranged to generate the response 252 containing the response data 254. The response data 254 can be any response associated with the detected anomaly. For example, unremoved damaged bone or damaged articular cartilage might still be visible in the video stream due to the surgeon inadvertently overlooking or otherwise failing to spot the damaged bone or articular cartilage. Upon detecting such an anomaly, the LLM 248 can generate the response data 254 for sending to the surgical computer 150. The response data 254 can be an audible alert in the form of speech such as, for example, “We recommend checking whether or not all damaged bone has been removed since we observed some remaining damaged bone in the video stream” . The surgical computer 150 can generate the output data 256 by converting the response data 254 into a format for output via the set, or a subset, of the peripherals 212 according to the table 258 containing the mappings 262 to 266.

[0086] Following such a conversion, the output data 256 is output via a respective one, or a respective subset, of the set of peripherals 212. The output data 256 can be output via, for example, the earphones 237 or some other audio output device.

[0087] The response data 254 may contain video data. The video data can represent an augmented form of the original video stream. Such an augmented video stream can be arranged to highlight or otherwise draw attention to the remaining damaged bone ordamaged articular cartilage. In such a case, the received response data 254 can be processed by the surgical computer 150, using the table 258, to output a video feed, with or without an alert or other warning, to the surgeon 130 via, for example, a visual display device. The visual display device can comprise at least one, or both, of: the augmented reality headset 218 or the display 230. The warning might be an audible warning in the form of a speech output such as, for example, “Warning: Remaining damaged bone!” .

[0088] Accordingly, the examples can provide systems and methods that can assist surgeons by at least one, or more than one, of the following taken jointly and severally in any and all permutations: monitoring the surgery, coaching manufacturer recommended surgical steps, detecting and alerting surgical error, processing and answering real time questions about the procedure specific to the patient and the surgical situation, providing monitoring and enhanced interpretation of medical imaging, instrument navigation and activation, and implant navigation and positioning.

[0089] Although the examples described herein have made reference to using a Large Language Model, examples are not limited thereto. Examples can be realised in which some other form of generative artificial intelligence assistant is used.

[0090] Examples can be realised according to the following clauses:

[0091] Clause 1 : An apparatus for guiding a surgical procedure, the apparatus comprising:

[0092] a communication interface for communicating with an artificial intelligence assistant, the communication interface being adapted

[0093] to receive input data for the artificial intelligence assistant, and

[0094] to receive an output data from the artificial intelligence assistant; and

[0095] surgical assistance circuitry:

[0096] to generate the input data for the artificial intelligence assistant from data associated with at least one input modality peripheral, and

[0097] to process the output data from the artificial intelligence assistant to generate output data associated with at least one output modality peripheral.

[0098] Clause 2: The apparatus of clause 1, comprising at least one, or both, of: the at least one input modality sensor and the at least one output modality sensor.

[0099] Clause 3: The apparatus of any preceding clause, in which the input data is associated with a surgical procedure.

[0100] Clause 4: The apparatus of clause 3, in which the input data is associated with a request for surgical information or an image relating to the surgical procedure.

[0101] Clause 5: The apparatus of any preceding clause, in which the output data from the artificial intelligence assistant is associated with a surgical procedure.

[0102] Clause 6: The apparatus of clause 5, in which the output data from the artificial assistant associated with the surgical procedure comprises surgical guidance relating to the surgical procedure.

[0103] Clause 7: The apparatus of any preceding clause, in which at least one, or both, of the input data and the output data for the artificial intelligence assistant comprises at least one or more than one of: audio data, visual data, position data, haptic data, pressure data, temperature data, acceleration data, or control data, taken jointly and severally in any and all permutations.

[0104] Clause 8: The apparatus of any preceding clause, comprising a surgical instrument interface for coupling to a surgical instrument.

[0105] Clause 9: The apparatus of clause 8, in which the surgical instrument interface is responsive to respective control data to actuate or control the surgical instrument.

[0106] Clause 10: The apparatus of any preceding clause, in which the input data for the artificial intelligence assistant is video data derived from at least one, or more than one, video camera.

[0107] Clause 11 : The apparatus of any preceding clause comprising the artificial intelligence assistant.

[0108] Clause 12: Machine-instructions for influencing a surgical procedure, the machine-instructions comprising:

[0109] instructions for a communication interface for communicating with an artificial intelligence assistant; the communication interface being adapted

[0110] to receive input data for the artificial intelligence assistant, and

[0111] to receive an output data from the artificial intelligence assistant;

[0112] surgical assistance instructions comprising:

[0113] instructions to generate the input data for the artificial intelligence assistant from data associated with at least one input modality sensor, and

[0114] instructions to process the output data from the artificial intelligence assistant to generate output data associated with at least one output modality peripheral.

[0115] Clause 13: The machine-instructions of clause 12, in which the input data is associated with a surgical procedure.

[0116] Clause 14: The machine-instructions of clause 13, in which the input data is associated with a request for surgical information relating to the surgical procedure.

[0117] Clause 15: The machine-instructions of any of clauses 12 to 13, in which the output data from the artificial intelligence assistant is associated with a surgical procedure.

[0118] Clause 16: The machine-instructions of clause 15, in which the output data from the artificial assistant associated with the surgical procedure comprises surgical guidance relating to the surgical procedure.

[0119] Clause 17: The machine-instructions of any of clauses 12 to 16, in which at least one, or both, of: the input data and the output data for the artificial intelligence assistant comprises at least one or more than one of: audio data, visual data, position data, haptic data, pressure data, temperature data, acceleration data, or control data, taken jointly and severally in any and all permutations.

[0120] Clause 18: The machine-instructions of any of clauses 12 to 17, comprising instructions associated with a surgical instrument interface for coupling to a surgical instrument.

[0121] Clause 19: The machine-instructions of clause 18, in which the instructions associated with the surgical instrument interface are responsive to respective control data to actuate or control the surgical instrument.

[0122] Clause 20: The machine-instructions of any of clauses 12 to 19, in which the input data for the artificial intelligence assistant is visual data derived from at least one, or more than one, imaging sensor.

[0123] Clause 21: Machine-readable storage storing machine-instructions of any of clauses 12 to 20.

[0124] Clause 22: A real-time, multi-modality, surgical robot comprising an apparatus of any of clauses 1 to 11.

Claims

CLAIMSWhat is claimed is:

1. An apparatus for guiding a surgical procedure, the apparatus comprising: a. a communication interface for communicating with an artificial intelligence assistant, the communication interface being adapted i. to receive input data for the artificial intelligence assistant, and ii. to receive an output data from the artificial intelligence assistant; and b. surgical assistance circuitry: i. to generate the input data for the artificial intelligence assistant from data associated with at least one input modality sensor, and ii. to process the output data from the artificial intelligence assistant to generate output data associated with at least one output modality sensor.

2. The apparatus of claim 1, comprising at least one, or both, of: a. the at least one input modality sensor, and b. the at least one output modality sensor.

3. The apparatus of claim 1, in which the input data is associated with a surgical procedure.

4. The apparatus of claim 3, in which the input data is associated with a request for surgical information or an image relating to the surgical procedure.

5. The apparatus of claim 1, in which the output data from the artificial intelligence assistant is associated with a surgical procedure.

6. The apparatus of claim 5, in which the output data from the artificial assistant associated with the surgical procedure comprises surgical guidance relating to the surgical procedure.

7. The apparatus of claim 1, in which at least one, or both, of: the input data and the output data for the artificial intelligence assistant comprises at least one or more than one of:a. audio data, b. visual data, c. position data, d. haptic data, e. pressure data, f. temperature data, g. acceleration data, or h. control data, taken jointly and severally in any and all permutations.

8. The apparatus of claim 1, comprising a surgical instrument interface for coupling to a surgical instrument.

9. The apparatus of claim 8, in which the surgical instrument interface is responsive to respective control data to actuate or control the surgical instrument.

10. The apparatus of claim 1, in which the input data for the artificial intelligence assistant is video data derived from at least one, or more than one, video camera.

11. The apparatus of any preceding claim comprising the artificial intelligence assistant.

12. Machine-readable storage storing machine-instructions for influencing a surgical procedure, the machine-instructions comprising: a. Instructions for a communication interface for communicating with an artificial intelligence assistant; the communication interface being adapted i. to receive input data for the artificial intelligence assistant, and ii. to receive output data from the artificial intelligence assistant; and b. surgical assistance instructions comprising: i. instructions to generate the input data for the artificial intelligence assistant from data associated with at least one input modality peripheral, andii. instructions to process the output data from the artificial intelligence assistant to generate output data associated with at least one output modality peripheral.

13. The machine-readable storage of claim 12, in which the input data is associated with a surgical procedure.

14. The machine-readable storage of claim 13, in which the input data is associated with a request for surgical information relating to the surgical procedure.

15. The machine-readable storage of claim 12, in which the output data from the artificial intelligence assistant is associated with a surgical procedure.

16. The machine-readable storage of claim 15, in which the output data from the artificial assistant associated with the surgical procedure comprises surgical guidance relating to the surgical procedure.

17. The machine-readable storage of claim 12, in which at least one, or both, of the input data and the output data for the artificial intelligence assistant comprises at least one or more than one of: a. audio data, b. visual data, c. position data, d. haptic data, e. pressure data, f. temperature data, g. acceleration data, or h. control data, taken jointly and severally in any and all permutations.

18. The machine-readable storage of claim 12, comprising instructions associated with a surgical instrument interface for coupling to a surgical instrument.

19. The machine-readable storage of claim 18, in which the instructions associated with the surgical instrument interface are responsive to respective control data to actuate or control the surgical instrument.

20. The machine-readable storage of claim 12, in which the input data for the artificial intelligence assistant is visual data derived from at least one, or more than one, imaging sensor.

21. A real-time, multi-modality, surgical robot comprising an apparatus as claimed in any of claims 1 to 11.

Images