Integrated Digital Surgical System
The integrated surgical support system addresses the limitations of existing systems by integrating energy devices, imaging, and AI/ML for enhanced surgical precision and cost-effective data-driven navigation.
Patent Information
- Application Number
- JP2024526969
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-10
- Filing Date
- 2022-11-10
- Publication Date
- 2024-12-05
AI Technical Summary
Existing surgical systems lack integration of energy systems, imaging interfaces, and data acquisition, limiting data integration, imaging fusion, and critical information exchange, and are hindered by high capital costs and limited intelligence.
An integrated surgical support system with a housing that connects energy devices, imaging scopes, and AI/ML modules, featuring a SOC-based substrate design for data processing and real-time surgical navigation, and intraoperative augmented reality.
Enhances surgical precision and decision-making with real-time data integration, imaging fusion, and intelligent control, reducing complications and costs through advanced data processing and AI-driven navigation.
Smart Images

Figure 2024544868000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates generally to an integrated surgical support system, and more particularly to an integrated surgical support system for simplifying and enhancing medical procedures and analyses. [Background technology]
[0002] It should not be assumed that the subject matter discussed in the Background section is prior art merely as a result of being mentioned in the Background section. Similarly, it should not be assumed that the problems mentioned in the Background section or associated with the subject matter of the Background section have been previously recognized in the prior art. The subject matter in the Background section also merely represents different ways in which the claimed technology may be implemented.
[0003] Mechanical instruments have been used in various open surgical procedures for centuries. Such instruments have become a natural extension of the surgeon's hands to perform specific functions to treat diseased tissues and organs. There are different types of commonly performed surgical procedures, including laparoscopy, endoscopy, arthroscopy, bronchoscopy, gastroscopy, etc. Laparoscopy or laparoscopic procedures are commonly used as it is a minimally invasive solution for a wide range of procedures such as cholecystectomy, appendectomy, hernia, and other more complex systemic procedures / colorectal procedures / GYN procedures / bariatric procedures, etc. Extensive improvements in mechanical instruments have been made to aid the surgeon's work in performing surgical cutting, dissection, coagulation, tissue treatment and management. Thus, energy-driven devices such as advanced RF bipolar and ultrasonic scalpel systems have gained popularity due to this improvement.
[0004] Currently, as a new addition to surgical instruments, robotic-assisted surgery is becoming more prevalent in urology and other surgical procedures. Even with decades of development, robotic systems still function as an extension of the surgeon's arm. In these systems, a robotic arm is used to hold an instrument for performing a surgical procedure, and a control system is used separately to control the movement of the arm and its instrument according to user manipulation of a master manipulator. The control system includes a filter in its forward path to attenuate master input commands that may cause vibrations of the instrument tip, and an inverse filter in a feedback path to the master manipulator configured to compensate for the delay introduced by the forward path filter. To enhance control, a master command and slave joint observer is also inserted into the control system to estimate slave joint position, velocity and acceleration commands using received slave joint position commands and torque feedback, and to estimate actual slave joint position, velocity and acceleration using sensed slave joint position and commanded slave joint motor torque. However, these systems may only provide limited capabilities or limited information of multiple energy devices via a screen.
[0005] Moreover, the robotic system offers the possibility of an integrated endoscope with mechanical and energy surgical instruments. Through this integration and electronic drive system, it is not only possible to provide individual instruments for cutting and coagulation, but also allows the integration of data for each instrument, including procedure imaging, tissue interaction and instrument interdependence recognition, to assist the surgeon in decision making. As data accumulates over time, this helps human surgeons effectively tackle complex procedures, make appropriate decisions and reduce complications during surgery. In traditional non-robotic laparoscopic surgery, individual instruments such as energy and mechanical instruments do not have this advanced functionality and the ability to integrate data for surgical assistance. Even surgical robots still lack this effective integration due to the historical individualized instrument design mindset. It only offers limited integration and intelligence based on existing instruments. Also, the high capital cost of the robotic system limits its benefits to specific procedures and specific hospitals.
[0006] Given the differences in the prior art, there is a need for a surgical system that is integrated with energy systems, imaging interfaces, and other data collection interfaces to enable data integration, imaging fusion, and critical information exchange. Summary of the Invention
[0007] According to one aspect, a surgical integrated support system is disclosed. The surgical integrated support system comprises an integrated surgical device. The integrated surgical device comprises a housing having one or more ports integrated on one side of the housing. The one or more ports are configured to couple the integrated surgical device with one or more energy instruments. The integrated surgical device has at least one external image input port, at least one output port, a power source port, and at least one external intelligence module port on the other side of the housing. Herein, the one or more energy instruments include at least one of bipolar and advanced bipolar shears, monopolar shears, ultrasonic shears, microwave ablation devices, laser ablation devices, laparoscopic devices, robotic control units, and endoscopes. It is noted that each of the one or more energy instruments is controlled between a high voltage energy output and a radio frequency energy output of the integrated surgical device. Furthermore, the integrated surgical device comprises an imaging scope connection assembled on one side of the housing and configured to input at least one optical or ultrasonic image. In one embodiment, the display unit is detachably connected to the housing and configured to provide an aggregated output related to the one or more energy devices. The display unit is constructed to tilt at one or more angles to provide convenience to the operator. It is noted that the display unit may be wirelessly connected to the housing using Ethernet. Further, a display control unit is disposed on a side of the housing, coupled to the display unit, and configured to control the operation mechanisms of each of the one or more energy devices. Furthermore, the surgical integrated support system comprises an artificial intelligence and machine learning (AI / ML) enabled module integrated within the housing to automatically optimize multiple parameters of the one or more energy devices. Herein, the AI / ML enabled module is communicatively coupled to a cloud network to train a model of the AI / ML enabled module and to collect real-time information related to the one or more energy devices.
[0008] In one embodiment, the AI / ML-enabled module is configured to generate a three-dimensional (3D) reconstruction of the multi-organ model using preoperative magnetic resonance imaging (MRI) images or computed tomography (CT) scan images as reference points. Furthermore, the AI / ML-enabled module collects real-time inferences using a live video feed from the laparoscope to highlight the location of the diagnosis. In another embodiment, the AI / ML-enabled module calculates an operating curve for each of the one or more energy devices for display on the display unit. The curve provides information regarding the progress of the diagnosis. Furthermore, the AI / ML-enabled module displays a reminder of the operating status of the one or more energy devices for the surgeon's reference.
[0009] In another embodiment, the AI / ML-enabled module is configured to retrieve real-time streaming video from each of the one or more energy devices in operation, compute machine learning inferences for lesion localization and surgical navigation, process intraoperative real-time surgical video, and upload and store within a cloud network the surgical videos and surgical data associated with each of the one or more energy devices utilized in the surgery.
[0010] According to a second aspect, the SOC-based board design is integrated into an integrated surgical device. The SOC-based board design comprises one or more primary connection ports fabricated on the SOC-based board design. The one or more primary connection ports establish a connection with one or more energy devices. Herein, the one or more primary connection ports are fabricated to establish a high-speed connection with an input device including one or more energy devices. Furthermore, the SOC-based board design comprises a processing unit fabricated on the SOC-based board design. The processing unit separates input data obtained from one or more energy devices into small packets and processes them. It is noted that the input data includes voice, image, video, and / or signal data.
[0011] The SOC-based board design further comprises a video processing unit (VPU) communicatively coupled to the processing unit. The VPU takes input data and converts it into a high-resolution output signal. The SOC-based board design further comprises one or more secondary connection ports fabricated on the SOC-based board design. The one or more secondary connection ports connect with a display unit for displaying the high-resolution output signal taken from the processing unit and / or the VPU. Here, the one or more secondary connection ports are fabricated to establish a high-speed connection with an output device including a display unit. [Brief description of the drawings]
[0012] The accompanying drawings illustrate various embodiments of the system, method, and various other aspects of the present disclosure. Those skilled in the art will appreciate that the illustrated element boundaries (e.g., boxes, groups of boxes, or other shapes) in the drawings represent an example of a boundary. In some examples, an element may be designed as multiple elements, or multiple elements may be designed as one element. In some examples, an element shown as an internal component of one element may be implemented as an external component of another element, and vice versa. Furthermore, elements may not be drawn to scale. A non-limiting and non-exhaustive description is provided with reference to the following drawings. The components in the figures are not necessarily to scale, with emphasis instead being placed on illustrating the principles.
[0013] [Figure 1] FIG. 2 is a front view of an integrated surgical device according to the present embodiment.
[0014] [Diagram 2] FIG. 1 illustrates a display integrated with an integrated surgical device according to an embodiment.
[0015] [Diagram 3] FIG. 2 is a rear view of the integrated surgical device according to the present embodiment.
[0016] [Figure 4] FIG. 1 illustrates an integrated surgical support system according to an embodiment.
[0017] [Diagram 5] FIG. 1 is a block diagram illustrating intraoperative augmented reality and navigation according to the present embodiment.
[0018] [Figure 6] 1 illustrates a motherboard-based design of an integrated surgical device according to an embodiment.
[0019] [Figure 7] FIG. 2 illustrates a cart layout of the integrated surgical support system according to the present embodiment.
[0020] [Figure 8A] FIG. 1 illustrates a system-on-chip (SOC) based board design architecture for an integrated surgical device according to an embodiment. [Figure 8B] FIG. 1 illustrates a system-on-chip (SOC) based board design architecture for an integrated surgical device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0021] Consider now in detail several embodiments of the present disclosure illustrating all its features. The words "comprising," "having," "including," and "including," as well as other forms thereof, are equivalent in meaning and are intended to be open ended in that the item or items following any one of these words are not meant to be an exhaustive listing of such item or items, nor are they meant to be limited to only the item or items listed.
[0022] It should also be noted that as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Although any systems and methods similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present disclosure, the preferred systems and methods are described herein.
[0023] Embodiments of the present disclosure will be described more fully below with reference to the accompanying drawings, in which like numerals represent like elements throughout the several views, and in which exemplary embodiments are shown. However, the claimed embodiments may be embodied in many different forms and should not be construed as being limited to the embodiments described herein. The examples described herein are non-limiting examples and are merely examples among other possible examples.
[0024] FIG 1 illustrates a front view of an integrated surgical device 100, according to one embodiment. FIG 1 will be described in conjunction with FIGs 2-3.
[0025] The integrated surgical device 100 may include a housing 102 having one or more ports 104, an imaging scope connection 106, a display control unit 108, a display unit 110, a light / sensory source indicator 112, and an artificial intelligence and machine learning (AI / ML) enabled module (not shown). The housing 102 may be a central or base station for each device or apparatus or instrument connected thereto. The one or more ports 104 are configured to couple the integrated surgical device 100 with one or more energy instruments (energy instrument-1, energy instrument-2, energy instrument-3), as shown in FIG. 1. In one embodiment, the one or more energy instruments may include, but are not limited to, at least one of bipolar and advanced bipolar shears, monopolar shears, ultrasonic shears, microwave ablation devices, laser ablation devices, laparoscopic devices, robotic control units, and endoscopes. In one embodiment, the imaging scope connection 106, the display control unit 108, and the light / sensory source indicator 112 may be any type of mechanical or electrical connection.
[0026] In one embodiment, the integrated surgical device 100 may include an energy device control module (not shown) configured to control multiple parameters of one or more energy devices using multiple control means. It is noted that the energy device control module may include selectable options that correspond to the control functions of one or more energy devices or that, when selected, execute the control functions of one or more energy devices. In one embodiment, a user may control one or more energy devices by manipulating the control means. The control means may include selecting a drill speed, a rod length, a current or voltage power supply, and the like. The energy device control module may include control functions such as buttons, balls, foot pedals, and wheels that allow navigation through the options of one or more energy devices. Thus, the energy device control module may have a simplified layout or a reduced set of functions when compared to a virtual device. For example, the energy device control module may be limited to navigation controls and selection buttons, thereby allowing the user to navigate one or more energy devices and select virtual controls to activate desired functions of one or more energy devices. It is noted that the user may refer to a physician, a specialist, an operator, or a laboratory technician.
[0027] Furthermore, the imaging scope connection 106 may be configured to detect image information within the patient's body. The imaging scope connection 106 may be configured to input at least one optical imaging device or ultrasound imaging device. The at least one optical imaging device or ultrasound imaging device may include a visual head member (not shown) and an elongated connector (not shown) having a handheld operation portion, an insertion portion, and a first contact element. The at least one optical imaging device or ultrasound imaging device establishes a connection with the housing 102 when the elongated connector is plugged into the imaging scope connection 106.
[0028] Further, the display control unit 108 may be capable of driving one or more energy devices connected to the housing 102. The display control unit 108 may communicate with the housing 102, and the housing may communicate with the one or more energy devices. The display control unit 108 may send and receive signals between the housing 102 and the one or more energy devices. In one example, the display control unit 108 sends a control command as a positioning signal to the housing 102 when selecting an option from the one or more energy devices projected on the display unit 110 or otherwise interacting with such one or more energy devices. It is noted that the display control unit 108 may include buttons (up, down, left and right) that allow the user to scroll through the one or more energy devices. The up button, the down button, the left button, and the right button may allow the user to scroll to a desired selection on the one or more energy devices. It is noted that the display control unit 108 need not have buttons and may be any type of device that allows the user to navigate through the one or more energy devices. In one embodiment, the display control unit 108 may be a handheld device incorporating a touchpad and / or trackwheel / ball, thereby allowing a user to view one or more energy devices on a computing touchpad or monitor on an associated screen. The touchpad or trackwheel / ball may allow a user to navigate to the energy devices and may allow the user to select a desired control device.
[0029] In one embodiment, the display unit 110 may be mounted on the housing 102 via an arm 114. The arm 114 may include a hinge mechanism (not shown) for lifting the display unit 110 at a certain angle from the housing 102. The angle of the display unit 110 may be set by the user at a predetermined angle between 0 degrees and 90 degrees. It is noted that the display unit 110 may be configured to fit on the top surface of the housing 102. The arm 114 may not be visible when the display unit 110 is in a closed position on the top surface of the housing 102. In one embodiment, the display unit 110 may be detachably connected to the arm 114 such that the display unit 110 can be detached or attached from the housing. In another embodiment, the display unit 110 may be wirelessly connected to the housing 102 and may be installed in a different location. It is noted that a portion of the arm 114 is visible when the foldable display unit 110 is extended as in FIG. 1. It should be noted that the arm 114 also functions as a mechanism for preventing the display unit 110 from falling off.
[0030] Additionally, the display unit 110 is configured to show integrated key information that may be important to assist the user / doctor / surgeon to optimize procedure flow and decision making during the surgical procedure. The integrated key information includes information regarding one or more energy devices, such as, but not limited to, power levels, cutting and sealing times, instrument temperature, resonant frequency of the ultrasonic device, tissue properties such as tissue impedance, tissue temperature, and the rate of change of these properties. It is noted that the information data may be integrated with image data or other images from the endoscope to provide the user with additional insights to guide the procedure and avoid potential complications during the procedure, thus reducing the risk to the patient.
[0031] Additionally, the housing 102 may include a light / sensory source indicator 112. The light / sensory source indicator 112 may include a light source (not shown), such as a light emitting diode (LED), attached to the imaging scope interface 106 or one or more energy devices. The light source is provided directly on the visual head member (not shown) to provide improved illumination capabilities. In one embodiment, the LED may be a white light LED, or an LED with a narrow spectrum around a preferred wavelength. In another embodiment, the light source is positioned at a distance from the objective opening of the imaging scope interface 106. For example, when the light source is positioned from the objective opening, the visual head member is provided with a means for collecting, reflecting, and / or projecting at least a portion of the light generated by the light source toward a target on the patient's body. The collecting, reflecting, and / or projecting means may be a reflector (not shown) having a deployable configuration. In one embodiment, the reflector may be expandable and / or contractible between a smaller diameter and a larger diameter of an iris design comprising, for example, multiple rigid or semi-rigid members. Additionally, the light source may be coupled to a number of optical fibers (not shown) disposed across and along the length of the visual head member. In one embodiment, the multiple optical fibers may be positioned on the expandable member, thereby allowing for a cone-shaped projection of light. The light source is configured to transmit a signal to the light / sensory source indicator 112. In one embodiment, the light / sensory source indicator 112 may have an active area size equal to or greater than the outer diameter of the elongated connector.
[0032] FIG. 3 shows a rear view of the integrated surgical device 100 according to an embodiment. The integrated surgical device 100 comprises an external image input port 302, a power port 304, an external intelligence module port 306, and at least one output port 308 integrated on the other side of the housing 102. The external image input port 302 allows one or more energy devices to be connected to the housing 102. Furthermore, the external image input port 302 may be configured to receive inputs, such as images or sensor data, from one or more energy devices. Furthermore, the external image input port 302 may be configured to receive at least one optical or ultrasound image from an imaging device connected to the integrated surgical device 100. The at least one output port 308 may be an equipment information output port. The equipment information output port may include an input unit data (not shown) and an output interface (not shown). The input unit data and the output interface allow for the integration of data from other treatment devices, diagnostic devices, or navigation devices. Additionally, the external intelligence module port 306 may be configured to connect auxiliary devices and energy devices.
[0033] In an exemplary embodiment, when an activation button or pedal on the energy device is pressed via the interface connection of the imaging scope connection 106 or an additional data interface, the energy device sends a trigger signal to the external imaging system and the laparoscopic insufflator. The connected laparoscopic imaging system is triggered to enhance the imaging process and reduce the effect from smoke or fog from tissue interaction with the energy device. This signal can also be implemented as an on / off signal to automatically control the insufflator to reduce smoke, fog for better images.
[0034] In one embodiment, the integrated surgical device 100 can also be connected to a human / computer interface, such as a touch screen, mouse, or control wheel. This feature gives the user the option to input or label on the image. Thus, the user may control the function of one or more energy devices and label one or more energy devices and tissue interactions during the procedure via the computer interface. This may also be done when the energy device is off to simply label tissue or anatomical structures during the procedure for training, navigation, or data collection purposes.
[0035] FIG. 4 illustrates an integrated surgical support system 400 according to one embodiment.
[0036] The surgical integrated support system 400 may include the integrated surgical device 100. As discussed, the integrated surgical device 100 may be connected to one or more energy devices. The one or more energy devices may include an energy source unit 402, a stapler 404, a laparoscope 406, and a robot control unit 408. The energy source unit 402 may include bipolar shears, monopolar shears, and ultrasonic shears that may be powered and controlled by the energy source unit 402. The energy source unit 402 includes electronic components (not shown) for powering the one or more energy devices between a high voltage energy output and a radio frequency energy output.
[0037] Furthermore, the surgical integrated support system 400 may include a stapler 404 for automatically stapling tissue during / after surgery. The laparoscope 406 may include a camera system (not shown), a light source box (not shown), and a handheld scope (not shown). Furthermore, the robot control unit 408 may be configured with several instruments in a group of robotic arms. It will be clear to those skilled in the art that the surgical integrated support system 400 is not limited to the above-mentioned instruments, and any kind of energy device / instrument may be present, such as an endoscope, microwave or laser cauterization device. In one embodiment, the surgical integrated support system 400 may be coupled to a cloud network 410 to exchange information related to one or more energy instruments, and vice versa.
[0038] The surgical integrated support system 400 may process information in real time through inference of the machine learning model. It should be noted that the machine learning model may be trained by collecting operational data from one or more energy devices. Furthermore, the surgical integrated support system 400 may provide an aggregated visualization output from one or more energy devices. Furthermore, each of the one or more energy devices may have an instance of a unique intelligence function, as shown in FIG. 5.
[0039] FIG. 5 illustrates a block diagram showing an example image 500 of intraoperative augmented reality and navigation. The example image 500 shows a three-dimensional (3D) reconstructed view 502. The 3D reconstructed view 502 may be a 3D reconstruction of a preoperative image showing a computed tomography (CT) scan or a magnetic resonance imaging (MRI) image. Additionally, a live surgical view 504 is shown. The live surgical view 504 may include live streaming or live images of a surgical procedure, such as a laparoscopy. Additionally, a 3D mesh view 506 is displayed. The 3D mesh view 506 may be a three-dimensional view of the patient's organs. Additionally, a tracking and navigation view 508 is displayed. The tracking and navigation view 508 may be real-time tissue tracking and navigation. The real-time tissue tracking and navigation may include augmented reality, localization, and navigation of lesions and nodules.
[0040] FIG. 6 illustrates a motherboard-based design of the integrated surgical device 100 according to the present embodiment. The motherboard-based design may accommodate an artificial intelligence and machine learning (AI / ML)-enabled module 600. The AI / ML-enabled module 600 includes an energy source unit 402 and a motherboard 602 having a system memory 604, an industrial central processing unit (CPU) 606, a non-volatile memory express (NVMe) storage 608, and a graphics processing unit (GPU) 610. Furthermore, the AI / ML-enabled module 600 includes a network interface card 612, an I / O port 614, and a video processing unit 616. Furthermore, the energy source unit 402 is configured to provide input and power that may connect to the laparoscope 406, one or more energy appliances, a controller 618, an external device 620, a cloud data stream 622, and a display unit 110 via USB or wirelessly. Furthermore, the laparoscope 406, the controller 618, and the display unit 110 are coupled to different elements of the motherboard 602. In one embodiment, the external device 620 may include one or more energy devices connected to the integrated surgical device 100 via one or more ports 104 and an imaging device connected via the imaging scope connection 106.
[0041] Further, a system memory 604 may be coupled to the industrial CPU 606. The system memory 604 may include appropriate logic, circuitry, and / or interfaces that may be configured to store machine code and / or computer programs having at least one code section executable by the processor. Examples of memory implementations may include, but are not limited to, random access memory (RAM), read only memory (ROM), hard disk drive (HDD), and / or secure digital (SD) cards.
[0042] Further, the industrial CPU 606 is coupled to the system memory 604, the network interface card 612, the GPU, and the Nvme storage 608. Further, the industrial CPU 606 comprises a processor that may include suitable logic, circuits, interfaces, and / or code that may be configured to execute a set of instructions stored in the system memory 604. The processor may be configured to generate a display image or a display video. The processor may further be configured to receive data of the energy source unit 402 via a cloud data stream 622. Examples of the processor may be an X86-based processor, a reduced instruction set computing (RISC) processor, an application specific integrated circuit (ASIC) processor, a complex instruction set computing (CISC) processor, and / or other processors.
[0043] In one implementation, the Nvme storage 608 may be integrated with the industrial CPU 606. The Nvme storage 608 may be configured to store information of input data received from the energy source unit 402. The Nvme storage 608 may be configured to store details of the input signal received via the industrial CPU 606. Those skilled in the art will appreciate that the data stored in the above-mentioned database may be stored in a structured data format or an unstructured data format. Implementations of the above-mentioned Nvme storage 608 include, but are not limited to, secure databases such as Amazon Web Services Cloud (AWS®), Microsoft Azure®, Cosmos DB®, Oracle® Database, Sybase®, MySQL®, Microsoft Access®, Microsoft SQL Server®, FileMaker Pro™, and dBASE®. Those skilled in the art will appreciate that in alternative implementations, the above-mentioned databases may be implemented as separate entities from the Nvme storage 608 without limiting the scope of the present disclosure. Additionally, the GPU 610 may be coupled to the Nvme storage 608 and the industrial CPU 606, and is configured to provide additional computational power required for the generation of display data. Note that the GPU coupled to the storage reduces buffer talk.
[0044] An I / O port 614 coupled to the motherboard 602 may include appropriate logic, circuitry, interfaces, and / or code that may be configured to receive signals from an energy device. The I / O port 614 may include various input / output devices that may be configured to facilitate communication between an external device and the display unit 110. In one embodiment, the display unit 110 may be referred to as a foldable additional display. Such one or more electronic devices may be configured in conjunction with the I / O port 614 to present display data of one or more interfaces on an additional display in an instance. Display screens include, but are not limited to, liquid crystal display (LCD) displays, light emitting diode (LED) displays, organic LED (OLED) display technologies, and the like.
[0045] The network interface card 612 may include appropriate logic, circuitry, interfaces, and / or code that may be configured to facilitate communication between multiple energy devices on the energy source unit via the cloud data stream 622. The network interface card 612 may be implemented based on known technologies to support wired or wireless communication. The network interface card 612 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a coder / decoder (CODEC) chipset, a subscriber identity module (SIM) card, and / or a local buffer. The network interface card 612 may communicate with a network via wireless communication using a network such as the Internet, an intranet, and / or using a wireless network such as a cellular network, a wireless local area network (LAN), and / or a metropolitan area network (MAN). The wireless communications may use any of a number of communications standards, protocols, and technologies, such as Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Long Term Evolution (LTE), Time Division Multiple Access (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email, instant messaging, and / or short message service (SMS), etc.
[0046] The video processing unit 616 may be configured to process and convert low quality images and / or video footage into high quality footage. Additionally, the industrial CPU 606 may be configured in conjunction with the video processing unit 616 to generate a display of multimedia content on the display unit 110. It should be noted that the video processing unit 616 may also be referred to as a video recorder.
[0047] Further, the AI / ML-enabled module 600 is configured to acquire data from the energy source unit 402. Further, the AI / ML-enabled module 600 computes intraoperative real-time processing of the surgical video, thus improving localization and surgical navigation. Further, the AI / ML-enabled module 600 is configured to upload data to the cloud network 410. Further, the AI / ML-enabled module 600 uses preoperative MRI or CT images as input to generate a 3D reconstruction of the multi-organ model. The AI / ML-enabled module 600 is configured to generate real-time inference with a live video feed from the laparoscope to highlight the location of the lesion, node, or lymph node along with the contours of important nerves, veins, and arteries. A guided path for operating on the lesion or node is illustrated.
[0048] Further, the AI / ML-enabled module 600 may be configured to train a deep learning model. In one embodiment, the deep learning model in an initial step is trained using information collected by one or more energy appliances. It is noted that the deep learning model is configured to process input data from one or more energy appliances to predict the output condition of the one or more energy appliances.
[0049] In one embodiment, the AI / ML-enabled module 600 is connected with a deep learning model to process input data from one or more sensors integrated with one or more energy devices. The AI / ML-enabled module 600 may be connected to the housing 102 to form the surgical integrated support system 400. The AI / ML-enabled module 600 is trained to enhance imaging processing to reduce effects from parameters such as smoke or fog from tissue interaction with one or more energy devices. Furthermore, the AI / ML-enabled module 600 is trained to implement on / off signals to one or more energy devices, for example to automatically control an insufflator to reduce smoke, fog, to obtain better images. Furthermore, the AI / ML-enabled module 600 uses quantum bit computation to generate multiple inputs with multiple possibilities of accurate output. In one embodiment, the AI / ML-enabled module 600 is trained using a deep learning model, using one layer of input data as input to another layer to generate a predicted output.
[0050] FIG. 7 illustrates a cart layout 700 of the surgical integrated support system 400 according to an embodiment. The cart layout includes one or more energy devices as shown in FIG. 4. The one or more energy devices are connected to the AI / ML-enabled module 600 and send digital control signals to the energy device control module of the one or more energy devices via a Peripheral Component Interconnect Express (PCIE). The one or more energy devices may include a laparoscope 406, an energy source unit 402, a robot control unit 408, a robot controller 702, and an insufflator 704. Furthermore, the cart layout 700 may include a display screen 706 for monitoring parameters of the one or more energy devices while performing a surgical procedure. Thus, the surgeon 708 may access the required data from the AI / ML-enabled module 600. In an embodiment, the surgeon 708 may provide commands to the one or more energy devices while receiving suggestions from the AI / ML-enabled module 600. It should be noted that the surgeon 708 may refer to a doctor or specialist, an operator, or a laboratory technician.
[0051] The AI / ML-enabled module 600 is configured to generate a 3D reconstruction of the multi-organ model using pre-operative MRI or CT scan images as reference points. Additionally, the AI / ML-enabled module 600 is configured to collect real-time inferences using live video feeds from the one or more energy devices to highlight the location of the diagnosis. In one embodiment, the AI / ML-enabled module 600 is configured to calculate an operating curve for each of the one or more energy devices for display on the display unit 110. The curves provide information regarding the progress of the diagnosis and display operating status reminders of the one or more energy devices for the surgeon's reference.
[0052] In one embodiment, the AI / ML-enabled module 600 is configured to retrieve real-time streaming video from each of the one or more energy devices in operation. Additionally, the AI / ML-enabled module 600 is configured to compute machine learning inferences for lesion localization and surgical navigation. Additionally, the AI / ML-enabled module 600 is configured to process intraoperative real-time surgical video and upload and store surgical videos and data associated with each of the one or more energy devices in operation within the cloud network 410.
[0053] 8A and 8B show a system-on-chip (SOC) based board design 800 architecture for an integrated surgical device, according to one embodiment.
[0054] The SOC-based board design 800 may also be referred to as a system-on-chip (SoC) design architecture. The SOC-based board design 800 allows for hardware-oriented user programming to improve the stability of the surgical integrated assistance system 400, the coding level of the drivers, and the interference of one or more energy devices that may cause problems such as delays. Furthermore, the SOC-based board design 800 may be configured to provide complete user customization, including form factors. The SOC-based board design 800 includes one or more primary connection ports, a processing unit, and one or more secondary connection ports.
[0055] Further, the one or more primary connection ports include a general purpose input / output (GPIO) 802, a video input / output 804. The GPIO 802 and the video input / output 804 are used to connect one or more energy devices to perform surgical activities. Furthermore, the SOC-based board design 800 includes a processing unit configured to receive and process input data from the one or more energy devices. The processing unit includes an application processing unit 806, a real-time processing unit 808, a signal processing unit 810, and an audio processing unit 812. Furthermore, the application processing unit 806 provides general-purpose computing in a standard programming environment based on the SOC-based board design 800.
[0056] In one embodiment, the processing unit separates the input data received from one or more energy devices into small packets and processes the input data independently to eliminate lag, delay, or interference between the input data of one or more energy devices. Furthermore, the real-time processing unit 808 is configured to perform real-time processing, in which the surgical integrated support system 400 may input the rapidly changing data received from the GPIO 802 and the video input / output 804, and then provide an output instantly so that the changes over time can be seen very quickly. It is noted that the real-time processing unit 808 performs instant processing of the data when the data input request needs to be addressed quickly. Furthermore, the signal processing unit 810 is configured to manipulate the information content in the signal to facilitate automatic speech recognition (ASR). It is noted that ASR helps to extract information from the voice signal and then convert the extracted information into recognizable words. Furthermore, the voice processing unit 812 is configured to convert between analog and digital formats, cut or boost selected frequency ranges, remove unwanted noise, add effects, and obtain many other desired results.
[0057] Furthermore, the SOC-based board design 800 is configured with a video processing unit (VPU) 814 to take input data and convert it into a high-resolution output signal. The VPU 814 reduces the need for an external capture card system. In an exemplary embodiment, the VPU 814 may have programmable logic as an H.265 encoder and decoder to input and output 4K60 video. It should be noted that the SOC-based board design 800 is a printed circuit board (PCB) design, and it is feasible to combine the VPU 814 with different video inputs and outputs 804, such as High-Definition Multimedia Interface (HDMI), Serial Digital Interface (SDI), etc.
[0058] Further, one or more secondary connection ports are fabricated on the SOC-based board design 800. The one or more secondary connection ports connect to a display unit for displaying high resolution output signals retrieved from the processing unit and / or the VPU 814. Further, the SOC-based board design 800 includes a system controller 816, a memory unit 818, a graphical processing unit (GPU) 820, a platform management unit 822, a security unit 824, and a storage 826. Herein, the storage 826 is configured to store and / or input data related to a surgical procedure or data for one or more energy devices. The system controller 816, the memory unit 818, the GPU 820, the platform management unit 822, and the security unit 824 operate in synchronization with the processing unit. The one or more secondary connection ports include a high speed connection unit 828 and a universal connection 830.
[0059] Additionally, the SOC-based board design 800 may be connected to the display unit 110 via a high-speed connection unit 828. It should be noted that the display unit 110 may also be referred to as a screen, as shown in FIGS. 8A-8B. In one embodiment, the high-speed connection unit 828 may enable connection with multiple devices used in minimally invasive procedures. The multiple devices may include a laparoscope 406, a stapling device 404, and an energy device 402. Additionally, the high-speed connection unit 828 may enable connection of a laparoscope with internal data access 832 to the SOC-based board design 800 using a capture card over Ethernet. In one embodiment, the laparoscope with internal data access 832 refers to a laparoscope that shares a data architecture that allows access of image data at various breakout points of the data collection and transfer path. It should be noted that the laparoscope with internal data access 832 may store information in a hard disk drive (HDD) format. Here, the laparoscope with internal data access 832 may alternatively connect with USB 3.0 and the high-speed connection unit 828. Additionally, a laparoscope with internal data access 832 may be connected via Ethernet to 100G Ethernet and MIPI PHY as shown in FIG. 8A.
[0060] Additionally, the high speed connection unit 828 may enable the connection of the laparoscope with external data access 836 to the SOC based board design 800 using a capture card over Ethernet. In one embodiment, the laparoscope with external data access 836 refers to a third party laparoscope, and the access of image data can only be made through the capture card. Alternatively, the laparoscope with external data access 836 may be connected to the PCIe via the capture card and through the video input / output unit 804 through the HDMI TX or SDI TX port. It should be noted that the laparoscope with internal data access 832 and the laparoscope with external data access 836 may be referred to as an internal laparoscopy and an external laparoscopy. Additionally, the GPIO 802 may be connected with a touch screen 834 for receiving instructions or commands from the surgeon 708 related to various surgical activities. It should be noted that the GPIO 802 may store and execute instructions in a solid state drive (SSD) format.
[0061] Additionally, the general purpose connectivity 830 may include Gigabit Ethernet (GigE), an embedded multimedia card (SD / eMMC), a serial peripheral interface (SPI), and a universal asynchronous receiver / transmitter (UART). Additionally, the GigE may be a tethering protocol for data transfer based on a wide range of Ethernet standards and may be connected to a server or network 838. Additionally, the SPI may be connected to a power stapler 840 and the SD / eMMC may be connected to an SD storage card 842 via a Bluetooth connection.
[0062] Additionally, the memory unit 818 may be connected to a dual in-line memory module (DIMM) 844 for transferring data associated with each of one or more devices.
[0063] In one embodiment, the processing unit may directly program and process input data, thus reducing latency in reading and transmitting data without modifying drivers or the operating system. For example, the surgeon 708 may directly process any port (e.g., USB) data without accessing the motherboard's operating system (OS) and firmware. The hardware characteristics of the SoC have several important features to improve performance. The real-time processing unit 808 may accelerate machine learning algorithms. One feature is that the processing unit requires less power so that it does not have a large thermal mitigation problem. The processing unit also has a large hardware capacity expandability. In one embodiment, the processing unit may have a PCIE slot for connecting additional high-speed components. In another embodiment, the processing unit may have a SATA port to support large-capacity storage.
[0064] Furthermore, one or more computer-readable storage media may be utilized in implementing embodiments consistent with the present disclosure. A computer-readable storage medium refers to any type of physical memory in which information or data readable by a processor may be stored. Thus, a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing the processor to execute steps or stages consistent with the embodiments described herein. The term "computer-readable medium" should be understood to include tangible products and exclude carrier waves and transient signals, i.e., non-transient signals. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, and any other known physical storage medium.
[0065] Features of the present invention will become apparent from the following description and the accompanying drawings. In the description and drawings, certain embodiments of the invention are disclosed in detail as illustrating some of the ways in which the principles of the invention may be used, but it will be understood that the invention is not correspondingly limited in scope. Features described and / or illustrated with respect to one embodiment may be used in the same or similar manner in one or more other embodiments and / or may be used in combination with or instead of features of the other embodiments.
[0066] While a preferred embodiment of the invention has been illustrated and described, as noted above, many changes can be made without departing from the spirit and scope of the invention. For example, aspects of the invention may be employed in alternative operating systems. Accordingly, the scope of the invention is not limited by the disclosure of the preferred embodiment. Instead, the invention should be determined entirely by reference to the appended claims. [Explanation of symbols]
[0067] Integrated surgical support system 100 Integrated Surgical Devices 102 Housing 104 One or more ports 106 Imaging scope connection part 108 Display control unit 110 Display unit 112 Light / sensory source 114 Arm 302 External image input port 304 Power Port 306 External Intelligence Module Port 308 at least one output port 400 Integrated Surgical Support System 402 Energy Source Unit 404 Stapler 406 Laparoscopy 408 Robot Control Unit 410 Cloud Server 500 Intraoperative Augmented Reality and Navigation Screen Mockups 600 Artificial Intelligence and Machine Learning (AI / ML) enabled modules 602 Motherboard 604 System Memory 606 Industrial Central Processing Unit (CPU) 608 Storage 610 Graphical Processing Unit (GPU) 612 Network Interface Card 614 I / O ports 616 Video Processing Unit 618 Controller 620 External Devices 622 Cloud Data Stream 700 Cart Layout 702 Robot Controller 704 Insufflator 706 Display screen 708 Surgeon 800 SOC based board design 802 General Purpose Input / Output (GPIO) 804 Video I / O 806 Application Processing Unit 808 Real-time Processing Unit 810 Signal Processing Unit 812 Audio Processing Unit 814 Video Processing Unit (VPU) 816 System Control Unit 818 Memory Unit 820 Graphical Processing Unit (GPU) 822 Platform Management Unit 824 Security Unit 826 Storage Unit 828 High Speed Connection Unit 830 General Purpose Connection 832 Laparoscope with internal data access 834 Touchscreen 836 Laparoscope with external data access 838 Server or Network 840 Power Stapler 842 SD storage card 844 Dual In-line Memory Module (DIMM)
Claims
1. a housing having one or more ports integrated into one side of the housing, the one or more ports configured to couple the integrated surgical device with one or more energy instruments; A display control unit disposed on a side surface of the housing and configured to control an operating mechanism of each of the one or more energy devices; an artificial intelligence and machine learning (AI / ML) enabled module integrated within the housing for automatically optimizing a plurality of parameters of the one or more energy devices; An integrated surgical support system comprising an integrated surgical device having:
2. 10. The surgical integrated assistance system of claim 1, wherein the one or more energy instruments include at least one of bipolar and advanced bipolar shears, monopolar shears, ultrasonic shears, microwave ablation devices, laser ablation devices, laparoscopic devices, robotic control units, and endoscopes.
3. The surgical integrated assistance system of claim 1 , wherein each of the one or more energy instruments is controlled between a high voltage energy output and a radio frequency energy output of the integrated surgical device.
4. The surgical integrated support system of claim 1 , wherein a display unit is removably attached to the housing and configured to provide an aggregated output related to the one or more energy devices.
5. The surgical integrated support system of claim 1, wherein the integrated surgical device has at least one external image input port, at least one output port, a power supply port, and at least one external intelligence module port on the other side of the housing.
6. The surgical integrated support system of claim 5, wherein the at least one output port is configured to transmit output energy device data, surgical equipment image data, equipment information, and organizational intelligence.
7. The surgical integrated support system according to claim 4 , wherein the display unit is configured to display integrated display device information, tissue intelligence information, and at least one optical or ultrasound image.
8. The surgical integrated support system of claim 1 , wherein the AI / ML-enabled module is configured for data acquisition, calculation, automation, multi-modality device connectivity and data storage.
9. The surgical integrated support system of claim 1, wherein the AI / ML-enabled module is communicatively coupled to a cloud network for training a model of the AI / ML-enabled module and for collecting real-time information related to the one or more energy devices.
10. The AI / ML compatible module is generating a three-dimensional (3D) reconstruction of the multi-organ model using preoperative magnetic resonance imaging (MRI) images or computed tomography (CT) scan images as reference points; collecting real-time inferences using live video feeds from the one or more energy devices to highlight locations of diagnostics; calculating an operating curve for each of said one or more energy devices for display on said display unit, said operating curve providing information regarding the progress of said diagnosis; The surgical integrated support system of claim 1 , configured to display operational status reminders of the one or more energy devices for a surgeon's reference.
11. The AI / ML compatible module is Retrieving real-time streaming video from each of the one or more energy devices in operation; Compute machine learning inference for lesion localization and surgical navigation; Processes real-time intraoperative surgical video; The surgical integrated support system of claim 10, configured to upload and store in a cloud network the surgical videos and data associated with each of the one or more energy devices utilized in a procedure.
12. 1. A system-on-chip (SOC) based board design integrated into an integrated surgical device, the SOC based board design comprising: one or more primary connection ports fabricated on the SOC-based board design, the one or more primary connection ports establishing connections with one or more energy devices; a processing unit fabricated on the SOC-based board design and configured to separate and process input data from the one or more energy devices into small packets; a video processing unit (VPU) communicatively coupled to the processing unit and configured to take the input data and convert it into a high definition output signal; and one or more secondary connection ports fabricated on the SOC-based board design for connecting to a display unit for displaying the high resolution output signal extracted from the processing unit and / or VPU.
13. 13. The SOC-based board design of claim 12, wherein the processing unit is configured to provide stability while processing the input data by eliminating interference and delays that may arise due to the connected one or more energy devices.
14. 13. The SOC-based board design of claim 12, wherein the processing unit separates and processes the input data into small packets to eliminate delays and interference of the input data between the one or more energy devices.
15. The SOC based board design of claim 12 , wherein the processing unit is configured to provide customization to an operator to enhance usability and form factor during execution of operations.
16. The SOC based board design of claim 12 , wherein the input data includes audio, image, video and / or signal data.
17. 13. The SOC-based board design of claim 12, wherein the VPU is configured to input and output 4K 60Hz video.
18. The SOC-based board design of claim 12 , wherein the one or more primary connection ports are fabricated to establish a high speed connection with the one or more energy devices.
19. The SOC-based board design of claim 12 , wherein the one or more secondary connection ports are fabricated to establish a high speed connection with the display unit.
Citation Information
Patent Citations
Positioning indicator system for a remotely controllable arm and related methods
JP2019528878A
Control system for surgical robots
JP2020509847A
System and method for controlling delivery of electrosurgical energy
JP2021058585A
Surgery system and control method
JP2021074242A
Method of HUB communication with surgical instrument systems
US20190125455A1
