Robotic systems and methods for precision organ resection techniques
A modular robotic system with integrated machine learning automates surgical tasks in scientific laboratories, reducing labor costs and enhancing accuracy in sample preparation and labeling, addressing the limitations of existing automation solutions.
Patent Information
- Application Number
- JP2024527739
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-11
- Filing Date
- 2022-11-11
- Publication Date
- 2024-12-05
AI Technical Summary
Current laboratory automation solutions for sample preparation and labeling in scientific research are costly, require significant infrastructure changes, and do not allow trained scientists to perform surgical tasks with the same accuracy as manual methods, leading to time-consuming and labor-intensive processes.
A modular robotic system equipped with a processor, articulated robotic arm, sensors, and a surgical platform, capable of performing automated surgical procedures, including anesthesia, tissue resection, and sample handling, with integrated machine learning for precision and automation.
The system reduces labor costs and time spent on repetitive tasks, enhances accuracy, and allows trained staff to focus on core research by automating sample preparation and labeling with precision comparable to manual methods, while minimizing human error.
Smart Images

Figure 2024544958000001_ABST
Abstract
Description
[Technical field]
[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 278,478, filed November 11, 2021, the entire contents of which are incorporated by reference herein.
[0002] [Technical field] The present disclosure relates generally to automated equipment systems in scientific laboratories, and more specifically to automated systems and methods for administering anesthetic agents, collecting tissues and samples from model organisms, collecting data, and organizing the tissues and samples into tubes, racks, and liquid nitrogen canisters. [Background technology]
[0003] [Background of the invention] Scientific research facilities, such as academic laboratories and pharmaceutical companies, perform preclinical studies on model organisms such as rodents, birds, amphibians, and nematodes to study how small molecules, vaccines, or treatments affect diseases. Studies to determine how these treatments work at a molecular level or how organs respond to treatments require the precise removal of different tissues, such as kidneys in the case of kidney disease studies or ovaries in the case of polycystic ovarian syndrome studies.
[0004] In academic laboratories conducting research with animals, scientists often collect as many tissues as possible in order to store these biological samples for future studies or to study the effects of drugs across multiple tissues. When a scientist needs to conduct one of these studies, they first carefully create a study design to test their hypothesis, followed by dietary, oral, IV, IP, subcutaneous, or various other drug delivery methods. After the study design is successfully completed, the scientist may spend hours meticulously labeling sample tubes / boxes and organizing tissue cassettes for specimen collection during the surgical procedure to harvest the tissues, i.e., during the necropsy.
[0005] During necropsy, animals are usually anesthetized and then euthanasia is performed according to local advisory committee protocols. Typically, animals are anesthetized by injection of ketamine / xylazine or other barbiturates or gas anesthesia such as isoflurane for short-term anesthesia. Once the animal is anesthetized, scientists may perform two euthanasia methods, such as exsanguination (removal of blood), decapitation, cervical dislocation, or removal of vital internal organs, in accordance with IACUC protocols to ensure that animals used in research receive humane care.
[0006] There are several common issues associated with administering anesthesia in a research environment, including: 1) standardization of flow rate: researchers may administer oxygen-isoflurane at various flow rates, resulting in widely varying levels of anesthesia between animals in the same study; 2) lack of training: properly setting up anesthesia equipment requires training by DVM or PHD staff to ensure the equipment functions properly; 3) inter-user variability: in academic fields, new students are often performing these tasks each semester, so there is typically a large degree of variability in where untrained students inject animals, which can result in unwanted premature deaths prior to euthanasia; 4) injection volume variability: when using common injectable anesthetic cocktails such as ketamine / xylazine, different volumes are needed for different animals to provide the same dose, necessitating different volumes, and sometimes multiple injections.
[0007] Following euthanasia, a typical metabolic study may involve the removal of certain organs, depending on the study design. For example, if the animal is participating in a study looking at kidney disease, the kidneys may be removed. After removal from the body, the kidneys may be transferred to weighing paper and placed on an analytical scale, and the organs may be weighed. The weight of the tissue is recorded manually with pen and paper, and the tissue is then: 1) placed in a labeled tissue cassette and then immersed in formalin, 2) placed in a labeled small microtube and immediately "flash" frozen in liquid nitrogen, or 3) placed in a labeled small microtube with a small amount of a chemical, such as RnaLater, to preserve the mRNA. Other studies may perform a process called perfusion, which removes blood from the organ, such as by cutting open the atria of the heart and pumping fluid into the circulatory system. Immediately after the total perfusion is completed, the process is completed, and the tissue is then fixed in a formalin buffer for histology examination.
[0008] One fundamental problem with this process, which occurs during or before the autopsy, is that scientists must individually label tubes for each specimen they intend to collect from each animal. Some scientific studies may be conducted on 3-50 rodents per treatment group, but it is common to have 2-10 different treatment groups. If scientists are performing multiple types of analyses on each tissue collected, they may need 3-6 separate labeling workflows for each tissue (blood tubes, tissue cassettes, protein tubes, RNA tubes, frozen tissues, biohazard bags, weighing papers, etc.). This highly manual process therefore requires hours of preparation before the autopsy and sometimes takes place during the surgical procedure, slowing down the procedure significantly. While academic studies often collect only tissues necessary for the current study, many industrial laboratories collect samples of nearly every organ / tissue available for future studies, further increasing the labeling workflow required for the autopsy.
[0009] Because this sample collection process is very time-consuming, several automated labeling tools have been developed, such as the SCINOMIX Print Solo System for individually labeling tubes of various sizes, and the TubeWriter® for labeling large numbers of identical tubes. However, all of these types of automated tube labeling machines require connection to a PC or require the user to type / pre-configure the content of each label. Other approaches, such as the IntelliXmark® Printer, offer large desktop solutions for high-throughput laboratories labeling large volumes of tubes. However, none of these automated solutions currently offer labeling tools for marking tissue cassettes in addition to labeling other types of tubes, such as 1.5 mL microtubes, 3 mL blood tubes, 8 mL blood tubes, etc.
[0010] The two main drawbacks to automating these surgical tasks are: 1) the anticipated costs are too high, and 2) automation would not be able to perform the surgical tasks as accurately as trained scientists and research veterinarians. Thus, to date, most companies have relied primarily on manual labor to perform these tasks, requiring up to four scientists in some cases, each of whom spends 40 hours performing rodent necropsies and additional hours labeling tubes for each scientific study.
[0011] Therefore, a solution that can perform sample preparation, collection, and storage with a level of precision equal to or greater than that of a trained scientist would allow these facilities to utilize such equipment for surgical tasks that require highly precise movements. Furthermore, since most current automated solutions require infrastructure modifications, providing a benchtop device would allow for a reduction in expected initial costs. Further cost savings can be expected since personnel may spend less time performing autopsies, allowing scientists to spend more time performing their core competencies rather than performing repetitive surgical tasks. Summary of the Invention
[0012] The disclosed invention overcomes most, if not all, of the shortcomings of prior art laboratory automation solutions, devices, and methods for sample labeling, sample handling, and animal anesthesia. The disclosed robotic system provides a modular system for automating necropsies and performing common preclinical research techniques such as injections and tissue collection. The disclosed system and method allows existing scientific laboratories to utilize both robotic surgical applications and manual surgery simultaneously, thus reducing the time trained staff spend on repetitive surgical tasks.
[0013] Accordingly, and briefly, the present invention provides a system for automated surgery comprising a processor and memory, an articulating robotic arm including an end effector, a plurality of sensors, and a surgical platform. The processor may be coupled to the plurality of sensors and to the articulating robotic arm, where one or more of the plurality of sensors provide signals associated with detecting and identifying characteristics of a surgical specimen. Additionally, the memory stores program instructions executable by the processor to process data received from the plurality of sensors and to output control signals to the articulating robotic arm to perform an automated surgical procedure.
[0014] The system may further include a remote communications interface, in which case the system may be configured to transmit data collected during the automated surgical procedure to a remote server, receive data related to the automated surgical procedure from the remote server, store program instructions received from the remote server in the memory, or any combination thereof. The remote server may be a cloud-based server or a server on a local area network.
[0015] The end effector may include a surgical instrument or a cleaning instrument. The articulating robotic arm may be configured to autonomously exchange the end effector with a second end effector, for example, by removing the end effector and inserting the second end effector. Alternatively or additionally, the end effector may include multiple surgical instruments and / or multiple cleaning instruments, which may be moved individually into a working position.
[0016] The surgical platform may have a breakaway plate configured to open into a downwardly positioned collection chamber that may be configured as part of a drawer on the system that collects waste in a separate, closable container when the breakaway plate is open.
[0017] The sensors may include any combination of color cameras, 3D cameras, LIDAR systems, infrared sensors, magnetic sensors, structured light sensors, proximity sensors, force sensors, and time-of-flight sensors. The sensors may include temperature sensors, humidity sensors, pressure sensors, gas sensors, and any combination thereof. The sensors may include any combination of color cameras, 3D cameras, LIDAR systems, infrared sensors, magnetic sensors, structured light sensors, proximity sensors, force sensors, and time-of-flight sensors, and any combination of temperature sensors, humidity sensors, pressure sensors, and gas sensors.
[0018] The articulating robotic arm, the plurality of sensors, and the surgical platform may be positioned within a housing having an inner shell, an outer shell, and an access door, and the articulating robotic arm may be mounted to a wall of the housing on the inner shell. Access to the interior of the housing may be through the access door, and the processor may output a control signal to lock the access door during the automated surgical procedure.
[0019] The system may further include an anesthesia unit having a positive airflow chamber, e.g., an induction chamber, an anesthetic supply, and a flow regulator in fluid communication with the anesthetic supply and the positive airflow chamber. The anesthesia unit may provide sufficient anesthetic to anesthetize or euthanize a model animal. The flow regulator may include a flow detector and a solenoid controller. The processor may be coupled to the flow regulator and may output a control signal to the solenoid controller. The memory may store program instructions executable by the processor to process data received from the flow detector and the plurality of sensors to track progress of the automated surgical procedure, vary a flow rate of anesthetic from the anesthetic supply to the induction chamber, or both.
[0020] The system may include a conveyor system for moving the surgical specimen from the positive air flow chamber to the surgical platform.
[0021] The system may further include a sample handling unit including a tube labeling system and a tube handling robot. The tube handling robot may be configured to collect sample tubes from a tube container, place the sample tubes in a predetermined location within the tube labeling system to provide a labeled sample tube, and position the labeled sample tube for collection of a sample from the surgical specimen. Additionally, the tube labeling system may be configured to mark the sample tubes with a barcode or text including a sample name, a sample collection date, a sample collection time, a protocol name, a protocol number, user identification information, or any combination thereof. The sample tubes may be of various sizes and shapes.
[0022] The sample handling unit may include a sample storage area configured as a drawer on the system. The temperature of the sample storage area may be above or below ambient temperature. When below ambient temperature, e.g. below -20°C, the sample handling unit may include a freezing system including a removable liquid nitrogen canister.
[0023] The freezing system may further include an integrated liquid nitrogen replenishment and dispensing station configured to provide liquid nitrogen replenishment.In lieu of a liquid nitrogen quick-freeze station, the freezing system may include a freezing unit, which may have multiple freezing units with a variety of different temperature ranges.
[0024] One or more of the plurality of sensors may include a camera configured to capture an image. Either the camera or the processor may use the image to generate image data representative of at least a portion of the image, and may transmit the image data in real time to the processor, the memory, the remote server (i.e., local or cloud-based), or any combination thereof. The processor is generally configured to determine a real-time spatial position of the end effector of the articulating robotic arm based on the image data. Additionally, the processor may indicate real-time positions of the articulating robotic arm and the end effector attached thereto during the surgical procedure based on machine learning applied to a dataset of previously performed surgical procedures.
[0025] The end effector may include a calibration position, and the processor may be configured to perform a calibration of the real-time spatial position of the end effector.
[0026] Aspects, features, benefits, and advantages of embodiments herein will become apparent with reference to the following description, the appended claims, and the accompanying drawings, in which like numerals represent like features in the various drawings. It should be noted that features and components in these drawings illustrating diagrams of embodiments of the invention are not necessarily drawn to scale unless otherwise noted. The illustrations in the accompanying drawings are not intended to be limiting, and other embodiments may be utilized and other changes may be made without departing from the spirit or scope of the subject matter presented herein. [Brief description of the drawings]
[0027] [Figure 1A] FIG. 1A shows a perspective view of a modular surgical system according to an embodiment of the present disclosure.
[0028] [Figure 1B]FIG. 1B illustrates a front view of the modular surgical system of FIG. 1A, showing certain internal components.
[0029] [Figure 1C] FIG. 1C illustrates a top view of the modular surgical system of FIG. 1A, showing certain internal components.
[0030] [Figure 2A] 2A and 2B show right and left side views, respectively, of the modular surgical system of FIG. 1A, showing certain internal components. [Figure 2B] 2A and 2B show right and left side views, respectively, of the modular surgical system of FIG. 1A, showing certain internal components.
[0031] [Diagram 3] FIG. 3 shows a transparent top perspective view of the modular surgical system of FIG. 1A, highlighting the internal and external components of the system.
[0032] [Figure 4] FIG. 4 is a diagram illustrating a communication workflow of a cloud server connected to an edge device according to an embodiment of the present disclosure.
[0033] [Diagram 5] FIG. 5 illustrates a privacy-aware machine learning infrastructure for communicating in a federated environment in accordance with an embodiment of the present invention.
[0034] [Figure 6] FIG. 6 illustrates a robotic operation utilizing a vision calibration system and the steps outlined in the vision calibration workflow according to an embodiment of the present invention.
[0035] [Figure 7A]FIG. 7A shows an image of a model animal captured by the disclosed system.
[0036] [Figure 7B] FIG. 7B shows a visual example of machine learning detection and extraction of foreground objects along with pre-segmented regions for surgical applications according to an aspect of the present disclosure based on the image of FIG. 7A.
[0037] [Figure 8] FIG. 8 illustrates an example of a computer vision system for detecting surgical landmarks in an exemplary rodent, according to an embodiment of the present disclosure.
[0038] [Figure 9] FIG. 9 illustrates a flow diagram of a workflow process for pixel-level and color-level analysis of real-time video to perform surgical object detection and localization, according to an embodiment of the present disclosure.
[0039] [Figure 10A] 10A-10F illustrate application of a custom algorithm for image analysis to identify specific surgical landmarks to detect organs in accordance with an embodiment of the present disclosure, where FIG. 10A is an image of a model animal captured by a system of the present disclosure, FIGS. 10B-10D illustrate identification of features in the image of FIG. 10A by a custom algorithm of the present disclosure, and FIGS. 10E and 10F illustrate edge detection in the image of FIG. 10A by a custom algorithm of the present disclosure. [Figure 10B] 10A-10F illustrate application of a custom algorithm for image analysis to identify specific surgical landmarks to detect organs in accordance with an embodiment of the present disclosure, where FIG. 10A is an image of a model animal captured by a system of the present disclosure, FIGS. 10B-10D illustrate identification of features in the image of FIG. 10A by a custom algorithm of the present disclosure, and FIGS. 10E and 10F illustrate edge detection in the image of FIG. 10A by a custom algorithm of the present disclosure. [Figure 10C] 10A-10F illustrate application of a custom algorithm for image analysis to identify specific surgical landmarks to detect organs in accordance with an embodiment of the present disclosure, where FIG. 10A is an image of a model animal captured by a system of the present disclosure, FIGS. 10B-10D illustrate identification of features in the image of FIG. 10A by a custom algorithm of the present disclosure, and FIGS. 10E and 10F illustrate edge detection in the image of FIG. 10A by a custom algorithm of the present disclosure. [Figure 10D] 10A-10F illustrate application of a custom algorithm for image analysis to identify specific surgical landmarks to detect organs in accordance with an embodiment of the present disclosure, where FIG. 10A is an image of a model animal captured by a system of the present disclosure, FIGS. 10B-10D illustrate identification of features in the image of FIG. 10A by a custom algorithm of the present disclosure, and FIGS. 10E and 10F illustrate edge detection in the image of FIG. 10A by a custom algorithm of the present disclosure. [Figure 10E] 10A-10F illustrate application of a custom algorithm for image analysis to identify specific surgical landmarks to detect organs in accordance with an embodiment of the present disclosure, where FIG. 10A is an image of a model animal captured by a system of the present disclosure, FIGS. 10B-10D illustrate identification of features in the image of FIG. 10A by a custom algorithm of the present disclosure, and FIGS. 10E and 10F illustrate edge detection in the image of FIG. 10A by a custom algorithm of the present disclosure. [Figure 10F] 10A-10F illustrate application of a custom algorithm for image analysis to identify specific surgical landmarks to detect organs in accordance with an embodiment of the present disclosure, where FIG. 10A is an image of a model animal captured by a system of the present disclosure, FIGS. 10B-10D illustrate identification of features in the image of FIG. 10A by a custom algorithm of the present disclosure, and FIGS. 10E and 10F illustrate edge detection in the image of FIG. 10A by a custom algorithm of the present disclosure.
[0040] [Figure 11] FIG. 11 shows an example involving the detection of multiple organs in a single frame.
[0041] [Figure 12A] 12A and 12B show a demonstration of the user interface of the present disclosure for labeling custom training data on an edge device, where FIG. 12A shows an image of a model animal captured by the system of the present disclosure and FIG. 12B shows the labeling of the organs and / or tissues found in FIG. 12A via a customized training process. [Figure 12B] 12A and 12B show a demonstration of the user interface of the present disclosure for labeling custom training data on an edge device, where FIG. 12A shows an image of a model animal captured by the system of the present disclosure and FIG. 12B shows the labeling of the organs and / or tissues found in FIG. 12A via a customized training process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0042] The disclosed robotic systems and methods address several aspects of animal necropsy that remain challenging and unsolved in the prior art. Specifically, the disclosed systems enable a robotic arm with high degrees of freedom to manipulate common laboratory tools to perform tissue resection, collection, and storage. Each of these systems includes one or more components that are individually configured to automate specific tasks related to tissue resection, collection, and storage, such as small animal anesthesia, small animal surgery (e.g., tissue resection), and labeling sample tubes.
[0043] Thus, the system may include a number of sensors, a remote communication interface, a memory configuration, a processing component, one or more machine learning computing nodes, an articulating manipulator arm, a vision coordination system, a sample handling system, a freezing system, a biohazard waste system, and a human interface system.
[0044] Prior to describing the systems and methods disclosed herein, certain terms and abbreviations will first be explained and / or defined.
[0045] <Abbreviations and definitions>
[0046] As used herein, including the appended claims, the singular forms "a," "an," and "the" include the plural and references to a particular numerical value include at least that particular value unless the context clearly indicates otherwise (i.e., plural). By way of example, "a" (a) processor may include one or more processors, "an" (a) end effector may include one or more end effectors, "the" (a) gas cylinder may include one or more gas cylinders, while "the plurality of" (a) sensors includes two or more sensors.
[0047] As used in this specification, including the appended claims, the word "comprising" and variations of the word "comprising" do not limit the invention and do not exclude any modifications or additions. Additionally, although the invention is described in terms of comprising, the devices, assemblies and methods detailed herein may also be described as consisting essentially of, or consisting of. For example, although the invention is described in terms of a system comprising a number of components, additional components unrelated to the function of the system may be included, and systems consisting essentially of, or consisting of, the same components are also within the scope of the invention.
[0048] Use of the word "or" means "and / or" unless expressly stated otherwise.
[0049] As used herein, the term "substantially" may be interpreted as representing the inherent degree of uncertainty that may result from any quantitative comparison, value, measurement, or other representation. Thus, the term "substantially" may generally mean an amount that is at least about 80%, about 90%, about 95%, about 98%, or about 99% of the recited comparison, value, measurement, or other representation.
[0050] Various aspects of the systems and methods disclosed herein may be illustrated by describing multiple components as being coupled, attached, and / or joined. The terms "coupled," "attached," and / or "joined" are used interchangeably in this disclosure and refer to either a direct connection between two components or, where appropriate, an indirect connection to each other through an intervening or intermediate component. In contrast, when a component is referred to as being "directly coupled," "directly attached," and / or "directly joined" to another component, there are no intervening elements or components present. (001) Relative terms such as "lower" or "bottom" and "upper" or "top" are used herein to describe the relationship of one element to other elements shown in the figures. It may be understood that the relative terms are intended to encompass different orientations of aspects of the system in addition to the orientation depicted in the figures. As an example, if an aspect of a system shown in the figures were flipped over, an element described as being on the "bottom" side of the other element would be oriented on the "top" side of the other element as shown in the associated figure. Thus, the term "bottom" may encompass both an orientation of "bottom" and "top," depending on the particular orientation of the figure.
[0051] In any operating example, or unless otherwise indicated, all numerical values expressing quantities of ingredients used in the specification and claims are understood to be modified in all instances by the term "about". Thus, unless indicated to the contrary, the numerical parameters set forth in the following specification and appended claims are approximations that may vary depending at least on the substrate used, the type and format of the touch-sensitive display surface, and the size of the assembly or device including the assembly. At the very least, and not as an attempt to limit the application of the doctrine of equivalents to the scope of interpretation of the claims, each numerical parameter should at least be construed by applying ordinary rounding techniques in light of the number of reported significant digits.
[0052] Notwithstanding that the numerical ranges and parameters setting forth the broad scope of the invention are approximations, the numerical values set forth in the specific examples are reported as precisely as possible, however, any numerical values inherently contain certain errors necessarily resulting from the standard deviation found in their respective testing measurements.
[0053] Terms such as "including" should be understood to mean "including, but not limited to." When ranges are given, any of the endpoints of those ranges and / or numerical values within those ranges may be combined within the scope of the invention.
[0054] As used herein, the term "digital information" may be understood to include any textual or graphical information that may be transmitted by a processor, such as a central processing unit (CPU) or graphical processing unit (GPU), and displayed on an output device, such as a computer display or screen, of the systems of the present disclosure.
[0055] The terms "storage," "data storage," and "memory," when used in the context of a computer system or method, may be used interchangeably and may be interpreted as referring to both temporary and permanent storage of data on a computer (i.e., on non-volatile memory hardware devices).
[0056] As used herein, the term "surgical instrument" is used to refer to a surgical tool suitable for performing a surgical operation on tissue, such as graspers, suture graspers, cutters, sealers, staplers, clip appliers, dissectors, scissors, scissors, suction instruments, clamping instruments, electrodes, coagulators, curettes, ablators, scalpels, needle holders, needle drivers, spatulas, forceps, biopsy and retraction instruments, or combinations thereof.
[0057] As used herein, the term "sensor" is used to denote a device that detects or measures a physical property and records, displays, or otherwise responds to it. Exemplary sensors include at least color cameras, 3D cameras, LIDAR systems, infrared sensors, time-of-flight sensors, magnetic sensors, force sensors, pressure sensors, structured light sensors, temperature sensors, humidity sensors, pressure sensors, gas sensors, and any other sensor known in the art that is useful for robotic manipulation and calibration, climate control, automated surgery, etc.
[0058] As used herein, the term "edge device" may be understood to include devices that provide an entry point into an enterprise's or service provider's core network. Examples include routers, routing switches, integrated access devices, multiplexers, and various metropolitan and wide area network access devices.
[0059] As used herein, the term "real-time" is used to mean the time required for a computer to receive and process constantly changing data, optionally in combination with other data such as predetermined, reference, estimated, etc. (which may be non-real-time data such as constant data or data that changes more frequently than one minute), to return real-time information to a user. "Real-time" may include short delays, such as up to 5 seconds after occurrence, preferably within 1 second, and more preferably within 0.1 seconds.
[0060] As used herein, the term "system" may refer to individual units of the disclosed invention, such as the surgical unit including the articulating robotic arm and surgical platform, the sample handling unit including the tube labeling component and the tube / sample freezing and / or storage component, the anesthesia unit including the anesthesia chamber and an anesthesia source, and the waste removal unit including means for removal of solid and liquid waste. Other components are possible, some of which are described herein, as well as additional units. Furthermore, the term system may be understood to refer to a group of units or their components. The system of the present invention further includes software products and databases useful for visualization and localization of relevant tissues and organs of surgical specimens, such as rats and mice, for the purpose of informing robotic surgery. Additional systems of the present invention include software products for informing specific robotic surgery protocols, including surgical procedures and sample preparation / storage.
[0061] The inventions detailed in this disclosure are not limited to the specific devices, methods, processes, elements or parameters described and / or illustrated herein, and the terminology used herein is for the purpose of describing particular embodiments, is for illustrative purposes only, and is not intended to limit the invention as claimed. All patents and other publications identified in this specification are incorporated herein by reference as if fully set forth herein.
[0062] In the following description, specific details are set forth to provide a better understanding of various embodiments of the disclosed robotic systems and methods. However, those skilled in the art will understand that the embodiments may be practiced without these details and / or without any details not described herein. In other instances, well-known structures, methods and / or techniques related to how the various embodiments are practiced may not be shown or described in detail to avoid unnecessarily obscuring the description of other details of the various embodiments.
[0063] <Components of the Disclosed System>
[0064] The present disclosure provides systems and methods for automating procedures such as administration of anesthetics for anesthesia or euthanasia, performing surgery on model organisms, collecting tissues and samples from model organisms, receiving data regarding the samples collected and stored, organizing tissues and samples into tubes, racks, and liquid nitrogen canisters, and collecting data during each of these procedures. These procedures can be performed by scientific laboratories using a variety of methods, including: Dramatically improve the productivity of the drug discovery process Allowing researchers to spend more time focusing on their core activities and less time performing repetitive surgical tasks; By paying low wages to the workers who operate these automated devices, the processes they implement (e.g., sample collection and storage) take less time to perform, reducing labor costs, one of the largest components of a laboratory's total costs; Maximize the accuracy of research results and minimize human error, Ensure data longevity, allowing protection and tracking of the entire history of surgical data; Improve accuracy of drug delivery to reduce supply costs caused by waste, Overcoming the shortage of skilled scientists, especially in developed countries; It utilizes novel system designs and software methods that enable:
[0065] The disclosed system provides a self-contained research platform that includes sensors combined with robotic assistance devices, tube handling / storage systems, anesthesia / euthanasia chambers, and robotic arms for performing surgical procedures. The disclosed system also provides computer vision algorithms and coordinates those algorithms with the robotic hardware to perform tissue sampling and extraction, among other surgical procedures.
[0066] 1A-1C, modular system 10 may include several internal hardware components surrounded by a frame 12 and enclosed by an exterior cover 14a. The bottom of frame 12 may include stands or feet 16 to ensure that system 10 is level prior to performing a surgical procedure.
[0067] Exemplary internal hardware components include a freezer component 19, a sample tube handling and labeling component 23, a robotic arm (two shown: 21, 22), an analytical scale 27, an anesthesia / euthanasia chamber 29, a surgical platform 33 with a waste disposal system, and a hidden area housing the electronic components 30. Exemplary external hardware components include a sample storage drawer 18a, a biohazard waste storage drawer 18b, and specimen input drawers (18c-18d), a user interface such as a touch screen panel 14b (see also FIG. 4), and dual side doors (41a, 41b) for material input (e.g., sample tubes, anesthesia, liquid nitrogen).
[0068] These exemplary components may be housed in separate systems or in a single system. For example, these exemplary components may be compartmentalized such that they may be provided as separate units that may be combined to form the system 10. With reference to FIG. 3, the components may be compartmentalized into multiple units, such as a sample handling unit 51, a robotic surgical unit 52, an anesthesia unit 53, etc., and certain additional functions, such as a sample disposal unit 40 and a vision adjustment unit 26, may be included with one or more of these units. Together, these components and / or units may perform common rodent surgical procedures, such as drug implantation, injections, tissue resection, device implantation, etc., with increased precision and throughput compared to unassisted surgery.
[0069] Referring to FIG. 1C showing a top view of the modular surgical system 10, a rigid outer frame 12 is configured to surround the inner frame 13. The back of the outer frame 12 is substantially flat and may have benchtop accessibility with removable or retractable outer covers (14a and 41a, 41b shown in FIGS. 2A and 2B), allowing for easy replacement of the system's internal components. Furthermore, such a design allows the modular surgical system 10 to be installed along a wall in a laboratory, such as by being installed along a wall while resting on a benchtop. An inner layer, i.e., the inner frame 13, may be formed as a second wall spaced apart from and located inside the outer frame 12. The space between the inner frame 13 and the outer frame 12 may be configured to accommodate wiring and other components, such that the system 10 provides a clean operating area free of excess wiring and an aesthetically pleasing appearance inside.
[0070] The internal frame 13 provides support for the robotic arms (21, 22) and the laboratory sample handling device 23. Additional support members, such as steel frames or panels, may be provided to provide a watertight area that can be easily disinfected, at least in the area of the robotic arms (21, 22). For example, additional panels may be provided on the back wall of the system 10, such as behind the robotic arms (two are shown in the figures: 21, 22), or on the back wall of the robotic surgical unit 52, or over the entire length of the back wall, including part of the side walls.
[0071] FIG. 2A shows the right side of the disclosed invention, and FIG. 2B shows the left side. Each of these sides can include doors (41a, 41b, respectively) that can be opened to replace internal consumable components for the anesthesia unit, the tube labeling unit (i.e., sample handling unit), the biohazard waste unit, and the surgical accessories. On the right side of the frame (FIG. 2A), the user interface devices and other robotic controllers can be positioned behind another removable plate to allow easy access to the devices for troubleshooting and repair. For example, the portion along the right roof area of the system can include an additional panel that houses electronic components. The panel can be configured as an access panel or door. Alternatively or additionally, the user interface devices and other robotic controllers can be positioned behind the user interface panel 14b, which can be configured as a panel that opens to provide access to the interior.
[0072] Referring to Figure 3, a see-through perspective view of the system is shown from above. Doors (14a, 41a, 41b) on the front and side exterior of the system 10 allow easy access to the internal surgical surfaces of the device and / or reloading of certain consumable surgical tool accessories. The front door 14a may be configured to open away from the device (upward), as shown in Figure 1A, to allow a user to interact with the internal components as needed. According to certain embodiments, the door may be locked at the start of a surgical procedure and cannot be opened while a surgical protocol is active.
[0073] <Robotic Surgery Unit>
[0074] The robotic surgical system 52 may include three main components: an articulating robotic arm (two shown: 21, 22) and a surgical platform 33. The surgical platform 33 may include a heating element to maintain a stable and constant temperature of the animal. The surgical platform 33 may include a motorized component, i.e., a separation plate, that allows it to be opened and closed, such as into a waste disposal area (area defined by drawer 18b).
[0075] The robotic surgical unit 52 typically has the largest area within the center of the system 10. Each of the articulating robotic arms may include a self-contained controller system and interchangeable end effectors for manipulating different materials or performing different procedures. That is, a first end of an articulating robotic arm may have an end effector that includes multiple interchangeable devices, tools or other surgical instruments. Each robotic arm may be equipped with an end effector that includes multiple "tools" that are interchangeable with new tools (i.e., in the case of disposable tools such as scalpel blades) or that are interchangeable with different or clean tools during a surgical procedure.
[0076] Alternatively or additionally, the first end of the articulating robotic arm may be configured to receive and exchange end effectors, such as tools, cleaning instruments or other surgical instruments. This exchange may be manual, semi-automatic (such as via instructions from a user interface), or autonomous (programmed with a protocol or based on data from one or more sensors). Exemplary end effectors include graspers, scalpels, scissors, loops, electromagnetic end effectors, electro-adhesive end effectors, surgical instruments as otherwise defined above, or combinations thereof, i.e., the end effector may include multiple surgical instruments and / or cleaning instruments that may be moved individually to a working position. Additionally, there may be end effector depots, such as positioned on either side of the surgical platform 33, on the wall (e.g., the back wall) to which the robotic arm is attached, or the like.
[0077] The robotic surgical unit 52 may further include a conveyor system within the large main chamber that may move specimens to desired locations within the surgical chamber, such as from the induction chamber of the anesthesia unit 53 to the surgical platform 33.
[0078] The robotic surgical unit can undergo self-calibration before each surgical procedure (i.e., pipeline), and the unit can be "digitally" actuated during surgery with better than 0.005 mm accuracy using colorimetric, spatial, and vision machine learning models. Calibration can be done via specific references such as visual markers (color, size, shape, etc.) used by cameras and visual sensors, weight used by scales, secondary thermometers, etc. Additionally, certain surgical platforms can be equipped with a warming center (e.g., heating elements and sensors) that monitors the body temperature of the animal during surgery and provides real-time heating to prevent hypothermia during these surgical techniques.
[0079] To run the system, the system 10 may include a controller having a memory component, a storage component, and a computing component. Together, these components accelerate calculations and workloads and provide local storage of data. In certain configurations, the memory component may be integrated into a single board computer (SBC) such as an edge device and may include DDR3, DDR4, or DDR5 RAM. The computing component may include up to three separate SBCs, each with its own processor for performing tasks within the system 10, including displaying a graphical user interface on an output device (user interface 14b). The storage component may include, but is not limited to, physical storage devices such as solid state hard drives, SD cards, removable flash storage devices, etc. Together, these components may enable local processing on the device and communication with a cloud interface.
[0080] <Visual Adjustment System>
[0081] Throughout this disclosure, the vision coordination system is described as including multiple sensors, such as a multi-level camera tracking grid, multiple cameras, and integrated lighting. The vision coordination system may actively monitor the progress of the surgical procedure during and optionally after the movement of the animal through various stages of the procedure (i.e., on a conveyor system) and collection of the organs, including weighing each organ, to ensure the success of each step of the surgery. The multi-level tracking grid may include multiple cameras that calculate the exact geospatial coordinates of the animal within the device in real time. This multi-level system may be integrated with a robotic surgical unit to provide input coordinates to facilitate the movement of an articulating robotic arm. The vision coordination system may include certain features as described and shown in Figures 6-10F. It should be understood that the vision coordination system of the present disclosure is not limited to the exemplary features specifically shown herein.
[0082] The computer vision workflow of the present disclosure may follow, but is not limited to, the steps outlined in Figure 6. The computer vision system may follow a process in which images are collected via a vision coordination system and real-time data streams are processed independently for each procedure, allowing precise control of a robotic coordination unit (e.g., vision coordination unit 26).
[0083] The computer vision workflow can be decomposed into three main aspects (features): 1) color-based workflow 610, 2) machine learning workflow 620, and 3) coordinate mapping workflow 630. The color-based workflow, i.e., image pre-processing pipeline 610, can be described as multiple consecutive steps with optional intermediate operations, where traditional steps of image capture, color filtering (e.g., light filtering), light enhancement (e.g., color adjustment), region cropping (e.g., edge detection), and masking (e.g., feature selection, database mapping, and region cropping) can be applied to each frame to accurately identify each organ system and its corresponding location in the surgical plane. Each frame is mapped to a corresponding physical location in the surgical plane and converted from pixel-level values to coordinates using a unique mapping strategy.
[0084] A machine learning workflow, i.e., computer vision pipeline 620, may run in parallel or serially with the color-based workflow, where pre-trained computer vision models trained on a unique customized surgical dataset may analyze images and provide multiple results including, but not limited to, image segmentation, object detection, object localization, image classification, instance segmentation, etc. There may also be a coordinate mapping workflow, i.e., vision and machine alignment pipeline 630, which uniquely brings together the results of the color-based workflow and the computer vision workflow to map pixels to physical animals / organs to translate the steps required for the articulating robotic arm to perform the surgical application.
[0085] The computer vision workflow may include multiple intermediate steps, as shown in Figures 7A and 7B, whereby the image is subjected to foreground segmentation to extract the most important image details before passing through the color-based workflow. Additionally, the computer vision workflow may also include custom trained computer vision models that can be run on external features, as shown in Figure 8. Prior to surgical application, these external features may be used to guide the robotic arm for positioning, such as before instantiation of a surgical procedure, and these features may also be communicated through multiple sensors in the surgical field. Multiple sensors provide signals for locating, detecting, classifying, and identifying the model organism, and signals to the robotic manipulator to control the articulating arm. The sensors also provide inherent information regarding depth, color, and information necessary for communication with the visual communication system. Exemplary sensors include cameras (3D and color), time-of-flight sensors, LIDAR systems, infrared sensors, magnetic sensors, structured light sensors, or any other sensors and combinations thereof known in the art.
[0086] The computer vision workflow may follow a sequential step-by-step protocol as shown in the flow diagram of FIG. 9. This step-by-step process may be performed in some aspects of the device or may follow a combination of the steps outlined in the figure. As shown in FIG. 9, the first step of the protocol (step 1) may include imaging and its capture of image data (i.e., a continuous stream of raw images and / or video capture by a camera). This data is used to detect and extract any foreground objects in the image data as shown in step 2 of FIG. 9. Optionally, disconnected components found in the image data may be cropped or masked (step 2.5). Step 3 of the protocol includes the generation of pixel features, followed by multi-organ detection (step 4) using a pre-trained computer vision model. Optionally, after step 3, objects may be located in the image (step 3.33) and cropped (step 3.66) and this data may be provided as input to step 4 (multi-organ detection). Finally, in step 5, calculations are made regarding the use of the articulating robotic arm to grasp the organ or tissue, and the process is continuously monitored during grasping of the organ / tissue by the articulating robotic arm.
[0087] Additional steps are performed in the computer vision workflow when the machine learning model is further trained with new surgical data under various conditions. During the computer vision workflow, features are generated from each individual frame to create several custom algorithms that help select image features to be sent to the computer vision model, as shown in Figures 10A-10F. The resulting features generated by applying these algorithms can create a numerical vector of data that can be passed to the machine learning model to identify specific regions on each frame of the video that need to be converted to pixel coordinates to control the articulating robotic arm. An example of an input image for feature extraction is shown in Figure 11.
[0088] The performance of the computer vision model can be selectively enhanced by providing additional labeled training data to the computer vision system by participating in the labeling process on the user's local device. This process can be completed during surgery or any time suitable data exists on the user's edge device. The customized training process can be displayed on the system's user interface device (such as touch screen 14b) or can be interacted with via a remote cloud platform. An example of a user interface for providing customized feedback is shown in Figures 12A and 12B.
[0089] Thus, the present disclosure further provides a method for optimizing future surgical procedures. The method may include providing a selection of an initial plan for a surgical procedure and surgical specimen, such as any of the procedures and specimens shown herein (e.g., rat necropsy), via a user interface on a remote server or system, and providing the selected initial plan to the system 10 (e.g., a controller of at least an articulating robotic arm). The method may further include transmitting information about the surgical procedure from the user and / or from the system to a processor on the remote server or system during and / or after the surgical procedure. This information about the actual procedure may be used to generate initial plans for subsequent surgical procedures that share common characteristics (i.e., type of surgery, surgical specimen).
[0090] The disclosed algorithms can generate an initial plan for a subsequent surgical procedure, which can include a collection of initial plans, surgical measurements, and numerical vectors of data stored on a server (local or cloud-based) from previous surgical procedures. Analysis of the aggregated data can be performed via known machine learning, reinforcement learning, or artificial intelligence methods and algorithms.
[0091] <Biohazard waste treatment system>
[0092] The biohazard waste disposal system may include a series of manual doors, waste bins, rest chambers, motorized bagger accessories, biohazard waste containers, and a sanitation system. Based on user selection, such as via a graphical user interface (e.g., touch screen 14b) at the beginning of a surgical protocol, at the end of each procedure, animals may be placed in the biohazard waste container if a terminal procedure is to be performed, or moved into the rest chamber for collection after waking up from anesthesia. The waste bin may include a box with a removable drawer that has an opening on top of the surgical platform 33 (e.g., a separation plate) that is directly connected to the sanitation system. The sanitation system may include a network of tubes for spraying a sanitation spray on the surgical platform to clean the instruments. The sanitation system may include a refillable bottle that the user can insert into the system with clean water and / or other cleaning agents, as well as a removable bottle for disposing of biohazard waste. The powered bagging accessory may include a powered component for moving a plastic waste disposal bag into position for collection of waste from a surgical procedure.
[0093] The waste disposal system on the exterior may be configured with a large drawer that slides out toward the user and away from the device (e.g., 18b). The waste disposal system may allow automated disposal of carcasses or biohazardous waste collected during a surgical protocol into clean biohazardous waste bags that may be collected in a waste area so that multiple surgical procedures may be performed consecutively. Although the biohazardous waste disposal system is described herein as a drawer configured to receive liquid and solid biohazardous waste, other configurations are within the scope of the presently disclosed invention. Additionally, different configurations of the biohazardous waste disposal system may be included in different versions of the presently disclosed system, such as a system that includes only one unit (i.e., surgical unit, anesthesia unit, sample handling unit, etc.) or a system that includes multiple units.
[0094] <Anesthesia Coordination Unit>
[0095] An anesthesia system or anesthesia conditioning unit may have a positive air flow chamber, i.e. an induction unit, a conveyor system, sensors, a gas regulation controller, and an automatic door. An induction chamber is used to contain the animal in an enclosed space during anesthesia procedures. An exemplary chamber includes an automatic lid or door that opens to allow the animal in and out and closes to contain the animal and anesthesia gas, and also includes two separate inlets and outlets for the entry of fresh anesthesia gas and collection of waste gas material for cleaning.
[0096] The anesthesia unit may provide multiple options so that the user can select an approved anesthesia based on the institution's guidelines to provide anesthesia / euthanasia via the induction chamber. The anesthesia conditioning unit may allow the user to provide interchangeable bottles of anesthesia, such as isoflurane. When configured as an induction chamber, the gas regulation controller may provide a positive air flow to provide the anesthetic into the induction chamber and exclude other gases to facilitate the animal's submission to the anesthetic. The gas may pass through the gas regulation controller and the gas chamber may be connected to a conveyor system that may allow the animal to be transported into the main surgical area.
[0097] According to certain embodiments, the system 10 may be configured to receive animals for surgery via a drawer (18c, 18d) positioned adjacent to the anesthesia / euthanasia chamber (i.e., anesthesia unit 29) allowing a user to place an animal inside the anesthesia / euthanasia chamber. The drawer operates on a sliding mechanism allowing a user to pull the drawer towards themselves, away from the surgical platform and lift the lid so that a rodent can be placed inside the drawer. Once the rodent is securely inside the drawer, the lid can be closed and pushed towards the device so that the drawer is flush with the surface of the surgical platform. Additionally, a button on the front of the device may allow the drawer to automatically unlock and eject, also allowing a user to remotely monitor and track when the drawer has been opened as part of an integrated cloud platform. The anesthesia unit 53 may include an anesthesia / euthanasia chamber 29 (eg, an induction chamber), an anesthesia agent supply 25 (eg, a gas cylinder), and a flow regulator 24 that includes a flow detector and a solenoid controller.
[0098] A number of sensors may be integrated within the gas chamber and throughout the rest of the device to regulate proper gas exchange. The sensors may allow real-time communication and visualization of the surgical platform 33 (e.g., separation plate) remotely within an online cloud platform or locally via a user interface (such as touch screen 14b). The sensors may include visual, acoustic, tactile, pressure, temperature, and gas sensors. For example, the system 10 may include sensors for real-time monitoring of the following: surgical platform temperature, cryounit temperature, anesthesia / euthanasia chamber gas flow rate, anesthesia / euthanasia chamber gas composition, visible light (sensor), and infrared (sensor). These sensors throughout the system 10 may be controlled remotely via a cloud platform or locally via a user interface. Each sensor may include a specialized cover and housing that allows the device to withstand the surgical environment inside the surgical platform and still operate within normal limits. Although particular sensors and configurations have been described herein, other sensors and configurations are within the scope of the present disclosure. Furthermore, different configurations of sensors may be included in different versions of the systems of the present disclosure, such as systems that include one or more units.
[0099] <Software and User Interface>
[0100] Implementations of the disclosed systems, including the surgical platform, sample handling unit, and anesthesia unit, have been described in the context of a device configured to perform various steps, methods, and / or functions in accordance with aspects of the described subject matter. It should be understood that a system, including a computing device or computer system, may be implemented by one or more computing devices. Implementations of the system and its various components or units may be described in the context of a "device configured to." Here, the term "configured to" may be interpreted to mean that the device may implement computer-executable instructions that are executed to perform various steps, methods, and / or functions in accordance with aspects of the described subject matter.
[0101] In general, a computer system or computing device may include one or more processors and storage devices (e.g., memory and disk drives), and may also include various input devices, output devices, communication interfaces, and / or other types of devices (see FIG. 4). A computer system or computing device may also include a combination of hardware and software. Thus, it should be understood that various types of computer-readable storage media may be part of a computer system or computing device. As used herein, the terms "memory," "computer-readable storage medium" and "computer-readable storage media" do not mean and specifically exclude propagated signals, modulated data signals, carrier waves, or any other type of transitory computer-readable media. In various implementations, the robotic system and surgical platform may include a processor configured to execute computer-executable instructions and a computer-readable storage medium (e.g., memory and / or additional hardware storage) that stores computer-executable instructions configured to perform various steps, methods, and / or functions in accordance with aspects of the subject matter that have been described.
[0102] Computer-executable instructions may be embodied and / or implemented in various ways as computer programs (e.g., client programs and / or server programs), software applications (e.g., client applications and / or server applications), software code, application code, source code, executable files, executable components, routines, application programming interfaces (APIs), functions, methods, objects, properties, data structures, data types, etc. Computer-executable instructions may be stored on one or more computer-readable storage media and executed by one or more processors, computing devices, and / or computer systems to perform particular tasks or implement particular data types in accordance with aspects of the subject matter that have been described.
[0103] The systems of the present disclosure may implement and utilize one or more program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. For example, the systems of the present disclosure include an image processing module (see FIG. 6) that includes computer-executable instructions for interpreting sensor signals to locate, detect, classify, and identify a model organism and to send signals to a robotic manipulator to control an articulating arm.
[0104] The disclosed system may be implemented as a distributed computing system or environment in which components are located on different computing devices connected to each other via a network (e.g., wired and / or wireless) and / or other direct and / or indirect connection forms. In such a distributed computing system or environment, tasks may be performed by one or more remote processing devices or in a cloud of one or more devices linked through one or more communication networks. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including media storage devices. Furthermore, the instructions described above may be implemented in part or in whole as hardware logic circuitry, which may include a processor.
[0105] The systems of the present disclosure may be implemented by one or more computing devices, such as computers, PCs, server computers, etc., configured to provide various types of services and / or data stores in accordance with aspects of the described subject matter. Exemplary server computers may include, but are not limited to, web servers, front-end servers, application servers, database servers, domain controllers, domain name servers, directory servers, and / or other suitable computers.
[0106] Components of the systems of the present disclosure may be implemented by software, hardware, firmware, or combinations thereof. For example, a system may include components implemented by computer-executable instructions stored on one or more computer-readable storage media and executed to perform various steps, methods and / or functions in accordance with aspects of the surgical procedures, visual detection, or machine learning that have been described.
[0107] The systems of the present disclosure may include a controller, memory, additional hardware storage, input devices, and output devices. The input devices may include one or more of the aforementioned example input devices, such as a user interface, a cloud platform, etc. The output devices may include data stored in a database, a GUI output on a user interface, electronic or physically printed readings of the results of a surgical procedure or sensor values, etc.
[0108] The systems of the present disclosure may include one or more communication interfaces that allow the system to communicate with other computing devices and / or other computer systems. For example, with reference to FIG. 4, the system may provide communication between a user interface accessed by a user and a database on a local network (LAN) or over the Internet. Thus, the system may be provided locally, cloud-based, or any combination thereof (i.e., cloud-based data may be used to run executables locally, or vice versa).
[0109] The systems of the present disclosure may include and / or execute one or more computer programs, for example implemented by software, firmware, hardware, logic and / or circuitry of a sensor and / or robot controller. The computer programs may include an operating system, for example implemented by one or more exemplary operating systems suitable for execution on a computing device. The computer programs may include one or more applications.
[0110] The software for controlling the system of the present disclosure may be displayed using a graphical user interface (GUI) to facilitate communication with the device (e.g., user interface). The GUI may include, but is not limited to, multiple screens for selecting an organization, managing user accounts and logging into the device, selecting protocols, and settings for controlling the surgical process. The GUI may be displayed on a user input device such as a touch screen, or may be controlled via a remote computer via HTTP communications. Local device input on the touch screen may be integrated with various other hardware components to provide an additional layer of security before accessing the surgical data on the machine, which may include integration with key cards, USB password devices such as UbiKey, biometric scanners, or multi-factor authentication devices that communicate with other personal electronic devices.
[0111] In addition to accepting user input, the GUI may also integrate with sensors contained throughout the device. The user interface may be a touch screen panel, USB input, card reader, and buttons to accept user input and control the physical device. A user may be able to control the device by following on-screen prompts via the touch screen and manipulating the buttons as instructed. The GUI on the touch screen may be configured to allow a user to monitor the status of the internal components locally or remotely via a cloud platform. To enhance the security of the device, locking mechanisms and user access control privileges may be available via password protection, user account access control, physical key card or USB key lock, or multi-factor authentication protocols such as mobile phone authentication.
[0112] The system is described as being equipped with hardware that allows communication with a remote cloud interface, for example as shown in Figure 4. The remote cloud communication interface may include multiple software and hardware combined to accomplish this function. Users may interact with the system in a variety of ways, including but not limited to directly via a touch screen interface, connecting to a cloud platform via either a desktop or mobile version, or remotely using the provider's software in a third party electronic lab notebook / lab information management system.
[0113] The remote cloud communication interface may be described as including three main components: 1) a centralized cloud system, 2) a local hardware interface including edge devices and an Ethernet connection, and 3) a remote access platform. The centralized cloud system may consist of a system designed to provision, deploy, update, and distribute firmware, software, or model updates remotely to edge devices via an Ethernet connection. This centralized cloud system may be maintained at a separate site from the end user's hardware and may allow remote updates without requiring user interaction with the devices.
[0114] The local hardware and edge devices may include, but are not limited to, electronics for video capture, computers, memory, storage, user interfaces, and visual displays that are physically connected or can communicate in a distributed manner over a local network. The remote access platform may be described as a software suite that allows a user to send commands to the hardware interfaces, control, download, upload, view, edit, or manage data / protocols remotely over the Internet. In this disclosure, the remote access platform may include multiple screens and websites that allow a user to interact with the device and examine the data, as shown in FIG. 4.
[0115] The disclosed system is described as being equipped with hardware and software that allows a privacy-aware machine learning architecture to be utilized to ensure that data privacy is maintained across multiple devices. As shown in FIG. 5, users 1-3 can maintain their data on local servers and the global model design in a central cloud hub can coordinate deployment to local edge devices. Each individual user can accept firmware updates from the central coordination hub and retrain models based on their individual data, preventing unnecessary network traffic of large video streams outside the network when necessary. In this privacy-aware architecture, users can accept models updated based on their data and selectively provide data to enhance the global model through contributing to a managed cloud database of anonymous data contributors across appropriate edge devices.
[0116] <Sample handling and cryogenic system>
[0117] The sample handling unit may include a multi-tiered storage rack, sample boxes, a labeling system, user-supplied sample tubes, a tube conveyor, and a Cartesian robotic sample handling stage. The sample handling unit may be mounted to the wall of the housing and may provide a platform for sample acquisition and a queue-based barcode sample tracking system. As shown in Figures 1-3, the sample handling unit 23 may be responsible for manipulating the storage tubes that are the result of a surgical procedure. These tubes may be subject to labeling and a unique barcode may be affixed to the outside of the tube, and each sample may be organized into a sample handling stage.
[0118] Each sample can be moved throughout the robotic system by a Cartesian robot, which can open and close the tubes. Once the tubes are filled with samples, they can be closed and moved to the appropriate storage location as defined in the surgical protocol. The sample handling unit can move in a three-dimensional field of XYZ and can be tightly interconnected with tube conveyors and other external accessories that may be available. Additionally, integration with an analytical scale 27 is possible to automatically record the weight of the sample immediately after cutting before being placed into the 1.5 ml tubes.
[0119] The sample handling unit may include additional accessories including, but not limited to, a tube hopper, a tube labeling device, a tube rack, a tube opening device, a tube scanning device, and a tube organizing device. Each of these extensions may be integrated with sensors for digital tracking of the sample processing pipeline. The tube hopper may include an exterior shell and a mounting bracket that may facilitate connection between the exterior wall of the device to automate tube sorting. The tube hopper may include an internal auger that may allow a user to insert an empty 1.5 ml tube and feed the tube into the sample handling unit. The tube labeling device may include a device that allows labeling of multiple tubes or individual tubes that are placed in the device. The tube rack may include a frame with holes that allows samples to be organized in the device, and the tube opening device may connect to the tube rack to open and close the tubes. The tube scanning device may connect directly to a computing device and scan the unique identifier on each tube for data management.
[0120] The cryogenic system may include, but is not limited to, a freezer component 19, a removable liquid nitrogen container, and / or a sample freezing stage. The freezer component may be described as a compartment with a door that allows sample boxes to be placed inside and kept cold. The liquid nitrogen container may allow the user to provide liquid nitrogen used during surgery to rapidly freeze tissue, and the sample freezing stage may be utilized to rapidly cool samples before freezing or to prepare them to be placed in tubes before freezing. The cryogenic system may be mounted directly to a wall of the sample handling unit 51, and the user may open a door (e.g., 41b) to add liquid nitrogen or remove samples upon completion of a run. Thus, the freezer component may include a door 41b on the exterior for opening to remove samples. The user may be able to remove individual samples or groups of samples, and may also allow the user to fill the sample storage / freezing area with liquid nitrogen.
[0121] <Aspects of the present disclosure>
[0122] The disclosed robotic systems and methods enable a high degree of freedom robotic arm to manipulate common laboratory tools (i.e., surgical instruments, graspers, cleaning tools, etc.) to perform tissue resection, collection, and storage. Each of these systems includes one or more components that are individually configured to automate specific tasks related to tissue resection, collection, and storage, such as small animal anesthesia / euthanasia, small animal surgery (e.g., tissue resection), and labeling sample tubes.
[0123] The disclosed system may include a processor, a memory, a surgical unit having a plurality of sensors, an articulating robotic arm including an end effector, and a surgical platform. The processor may be coupled to a plurality of sensors and to the articulating robotic arm, the plurality of sensors providing signals related to detection and identification of features of a surgical specimen. Additionally, the memory stores program instructions executable by the processor to process data received from the plurality of sensors and to output control signals to the articulating robotic arm to perform an automated surgical procedure. The system's memory may also include a configuration file for running the robotic arm, allowing for proper calibration before execution of each process.
[0124] The surgical platform may be integrated into the biohazard waste system or may be separate for enhanced functionality. The surgical platform may have a breakaway plate configured to open into a collection chamber positioned below. The collection chamber may be configured as part of a drawer on the system that collects waste in a separate, closable container when the breakaway plate opens. The biohazard waste system may include a chamber in which samples can be bagged and from which a user can remove completed samples from the system. Additionally, the biohazard waste system may include a handling device for moving a biohazard bag to capture samples to be discarded from the main surgical platform. The biohazard waste system may include a single movable drawer for manual removal of samples by a user without automated sample disposal.
[0125] The articulating manipulator arm may be capable of performing very precise surgical tasks with a high degree of freedom. Additionally, the first end of the articulating manipulator arm may have an end effector that includes multiple interchangeable devices, tools or other surgical instruments, or that is configured to receive one tool or other surgical instrument. Thus, the end effector may include a grasper, scalpel, scissors, loop, electromagnetic end effector, electro-adhesive end effector, surgical instrument as otherwise defined above, cleaning tool, or combinations thereof. Alternatively, the end effector may be configured to receive a grasper, scalpel, scissors, loop, electromagnetic end effector, electro-adhesive end effector, surgical instrument as otherwise defined above, cleaning tool, or combinations thereof. Additionally or alternatively, the end effector may have a swap mechanism that allows for changing tools. According to certain embodiments, the end effector may have two or more tools or surgical instruments, and may swap a first tool for a second tool, such as during surgery. The exchange of tools can be manual, such as via real-time commands from a user interface, or can be autonomous. According to certain aspects, the articulating robotic arm can be configured to exchange tools autonomously, i.e., the articulating robotic arm can remove a first tool from an end effector and insert a second tool into the end effector.
[0126] The disclosed system may include a processor, a memory, a number of sensors, and an anesthesia unit having a positive airflow chamber or induction chamber, an anesthetic supply, e.g., a gas cylinder, and a flow regulator in fluid communication with the anesthetic supply and the positive airflow chamber. The flow regulator may include a flow detector and a solenoid controller. The processor may be coupled to the flow regulator and may output a control signal to the solenoid controller. The memory may store program instructions executable by the processor to process data received from the flow detector and the number of sensors to track anesthesia progress, vary the flow rate of anesthetic from the anesthetic supply to the induction chamber, or both. When the system is included in a larger system including a surgical unit and associated components, the system may include a conveyor system for moving a surgical specimen from the induction chamber to a surgical platform. Additionally, the anesthesia unit may be configured to have replaceable components, such as components that allow for the use of either gaseous or liquid anesthetics, and / or components that adjust the anesthetic and replenish liquid chemicals for low-flow anesthesia applications.
[0127] The disclosed system may include a processor, a memory, a plurality of sensors, and a sample handling unit including a tube labeling system and a tube handling robot configured to collect sample tubes from a tube container, place the sample tubes in a predetermined location within the tube labeling system, label the sample tubes to provide labeled sample tubes, and position the labeled sample tubes for collection of samples from the surgical specimen. The tube labeling system may be configured to mark the sample tubes with a barcode or text including a sample name, a sample collection date, a sample collection time, a protocol name, a protocol number, user identification information, or any combination thereof. The sample handling unit may further include a sample storage area configured as a drawer on the system, the temperature of the sample storage area may be less than -20°C, and the sample handling unit may include a freezing system including a removable liquid nitrogen canister and / or an integrated liquid nitrogen replenishment and dispensing station configured to provide liquid nitrogen replenishment. The sample storage area may also have a cooling unit configured to maintain samples placed therein at a user selected temperature. The memory may store program instructions executable by the processor to process data received from a plurality of sensors to track the progress of sample tube labeling and storage and to output control signals to a tube handling robot.
[0128] The system of the present disclosure may include any combination of two of the systems previously described herein. For example, the system may include a memory, a processor, a number of sensors, and any combination of two of a surgical unit, an anesthesia unit, and a sample handling unit.
[0129] A system of the present disclosure may include each of the systems previously described herein. For example, a system may include a memory, a processor, a number of sensors, a surgical unit, an anesthesia unit, and a sample handling unit. Such a system may include a transport device, such as a conveyor belt, capable of transporting samples through the system, such as from the anesthesia unit to the surgical unit.
[0130] Systems of the present disclosure may include one or more sensors, a remote communication interface, a memory component, a processing component, one or more machine learning computing nodes, an articulating manipulator arm, a vision coordination system, a sample handling system, a freezing system, a biohazard waste system, and a human interface system.
[0131] Each of the systems may be positioned individually or collectively within a housing that includes an inner shell, an outer shell, and an access door.
[0132] Each of the systems may include one or more sensors individually or collectively, with example sensors including one or more of color cameras, 3D cameras, LIDAR systems, infrared sensors, time-of-flight sensors, magnetic sensors, force sensors, pressure sensors, structured light sensors, temperature sensors, humidity sensors, pressure sensors, gas sensors, and any other sensor known in the art useful for robotic operation and calibration, climate control, automated surgery, and the like.
[0133] Each of the systems may include memory and processing components, individually or collectively, that may use system-specific information, such as calibration data to ensure proper performance of the system, i.e., data received from sensors, computer programming instructions on the memory that are executed by a processor to operate the system, sensor data regarding the system environment, and the state of various processes.
[0134] Each of the systems may individually or collectively include a remote communication interface for communicating data collected by the various systems with a local server and / or a cloud computing environment. The remote communication interface may include a database for recording real-time data that may be collected from the various systems. The database may be a local database, a cloud-based database, or both a local database and a cloud-based database. The remote communication interface may be linked to a user input device and provide communication with the server / database (local and / or cloud-based), thereby improving security of the system and allowing shared use of the system across multiple laboratories and badge-based security systems to allow or deny access to various aspects of the device to certain users. Additionally, the remote communication interface may eliminate the need for users to input labels for tubes that may be processed remotely, and may allow processing of each system to be managed remotely. A small configuration file may be periodically shared with the database via the remote communication interface to track version configurations between processes to ensure reproducibility. Communication between the systems and the server / database may include wired communication, wireless communication, or a combination thereof.
[0135] Each of the systems may individually or collectively comprise a human interface system. The human interface system may include a graphical display monitor and an input device. The input device may be a touch screen or a voice command interface where the user can see progress and information displayed that is intended to guide the robot or display error information. Additionally, the remote interface may include additional functionality within the interface system that provides the user with controls to access the cloud platform.
[0136] Each of the systems may individually or collectively comprise one or more machine learning computing nodes. The machine learning computing nodes may include trained machine learning models and corresponding tensor processing units with a central processing unit for executing the machine learning models in real time. These computing nodes may include separate physical hardware for performing the computations or integrated electronic platforms capable of processing the machine learning computations while simultaneously running the device. The computing nodes may include cloud computing nodes that do not require a physical device connected to the machine.
[0137] The vision adjustment system may include multiple vision sensors / tools including color cameras, 3D cameras, LIDAR systems, infrared sensors, time-of-flight sensors, structured light sensors, etc. The vision adjustment system may include lights to enable communication of visual information. Additionally, the vision adjustment system may communicate with a central server via a wired network, a wireless network, or a combination thereof. Additionally, the vision communication system may include replaceable parts in a modular format to enable hardware updates along with future firmware upgrades.
Claims
1. 1. A system for automated surgery, comprising: A processor and a memory; an articulating robotic arm including an end effector; A plurality of sensors; A surgical platform; Equipped with the processor is coupled to the plurality of sensors and to the articulating robotic arm; the plurality of sensors providing signals related to detection and identification of characteristics of a surgical specimen; The memory stores program instructions executable by the processor to process data received from the plurality of sensors and to output control signals to the articulating robotic arm to perform an automated surgical procedure. A system characterized by:
2. The end effector includes a surgical tool or a cleaning tool.
2. The system of claim 1 .
3. the articulating robotic arm is configured to autonomously exchange the end effector with a second end effector; The autonomous replacement of the end effector includes removing the end effector and inserting the second end effector.
2. The system of claim 1 .
4. Two Articulating Robot Arms The system of claim 1, further comprising:
5. The sensors include one or more of a color camera, a 3D camera, a LIDAR system, an infrared sensor, and a time-of-flight sensor, and one or more of a temperature sensor, a humidity sensor, a pressure sensor, and a gas sensor.
2. The system of claim 1 .
6. Anesthesia Unit Further comprising: The anesthesia unit comprises: An induction chamber; an anesthesia supply department; a flow regulator in fluid communication with the anesthetic supply and the induction chamber; It has The flow regulator includes a flow detector and a solenoid controller. The processor is coupled to the flow regulator and outputs a control signal to the solenoid controller.
2. The system of claim 1 .
7. The memory stores program instructions executable by the processor to process data received from the flow detector and the plurality of sensors to track progress of the automated surgical procedure, vary a flow rate of anesthetic agent from the anesthetic agent supply to the induction chamber, or both.
7. The system of claim 6.
8. A conveyor system for moving the surgical specimen from the induction chamber to the surgical platform.
7. The system of claim 6, further comprising:
9. The anaesthetic supply is a gas cylinder.
7. The system of claim 6.
10. The surgical platform has a release plate configured to open into a collection chamber positioned below.
2. The system of claim 1 .
11. The collection chamber is configured as part of a drawer on the system that collects waste in a separate, closable container when the breakaway plate is open. The system of claim 10.
12. A sample handling unit including a tube labeling system and a tube handling robot. Further comprising: The tube handling robot is configured to collect sample tubes from a tube container, place the sample tubes in positions within the tube labeling system to provide labeled sample tubes, and position the labeled sample tubes for collection of samples from the surgical specimen.
2. The system of claim 1 .
13. The tube labeling system is configured to mark the sample tubes with a barcode or text including a sample name, a sample collection date, a sample collection time, a protocol name, a protocol number, a user identification, or any combination thereof.
13. The system of claim 12.
14. A sample storage area configured as a drawer on the system 13. The system of claim 12, further comprising:
15. the temperature of the sample storage area is less than −20° C.; The sample handling unit includes a freezing system that includes a removable liquid nitrogen canister.
15. The system of claim 14.
16. The freezing system further includes an integrated liquid nitrogen replenishment and dispensing station configured to provide replenishment of liquid nitrogen.
16. The system of claim 15.
17. The sample storage area includes a cooling unit configured to maintain samples placed therein at a user-selected temperature.
15. The system of claim 14.
18. Remote Communication Interface Further comprising: The system is configured to transmit data collected during the automated surgical procedure to a remote server, receive data related to the automated surgical procedure from the remote server, store program instructions received from the remote server in the memory, or any combination thereof.
2. The system of claim 1 .
19. one or more of the plurality of sensors includes a camera configured to acquire images during the automated surgical procedure and transmit the images in real time to the processor; The processor is configured to generate image data representative of at least a portion of the image.
2. The system of claim 1 .
20. The processor is configured to determine a real-time spatial position of the end effector of the articulating robot arm based on the image data.
20. The system of claim 19.
21. the end effector includes a calibration position; The processor is configured to perform a calibration of the real-time spatial position of the end effector.
21. The system of claim 20.
22. The processor directs real-time positions of the articulating robotic arm and the end effector attached thereto during the surgical procedure based on machine learning applied to a dataset of previously performed surgical procedures.
20. The system of claim 19.
23. the articulating robotic arm, the plurality of sensors, and the surgical platform are positioned within a housing having an inner shell, an outer shell, and an access door; the articulating robot arm is attached to a wall of the housing on the inner shell; access to the interior of the housing is provided via the access door; The processor outputs a control signal to lock the access door during the automated surgical procedure.
2. The system of claim 1 .
Citation Information
Patent Citations
Multi-output anesthesia system
JP2005517507A
Freezing machines with containers used for frozen samples
JP2018500548A
Systems and methods for automated intubation
JP2020505170A
Operation field monitor system
JP2021142175A
Detection of unintentional movement of a user interface device
US20200046439A1
