Medical instrument classification and tracking for sterile processing

Abstract

Methods, systems, and apparatus, including medium-encoded computer program products, for classifying and tracking medical instruments. The process includes receiving an image generated by an imaging device; providing the image as input to a medical instrument detection model; detecting, by the medical instrument detection model, a medical instrument in the image; in response to detecting the medical instrument in the image, providing the image as input to a medical instrument classification model; determining, by the medical instrument classification model, a classification of the medical instrument as a particular type of medical instrument; annotating the image with a label indicating the particular type of medical instrument to generate an annotated image; and providing the annotated image for presentation by an electronic display.

Description

MEDICAL INSTRUMENT CLASSIFICATION AND TRACKINGFOR STERILE PROCESSINGCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 638,829, filed April 25. 2024, the entirety of which is incorporated by reference.TECHNICAL FIELD

[0002] The present disclosure relates to various computing devices, computing management systems, and computing environments that automate the management of medical instruments.BACKGROUND

[0003] Medical instruments are used in medical procedures such as surgeries. A tray or group of medical instruments is provided in an operating room in order to provide care for a patient. Medical personnel use the medical instruments for surgical procedures or other procedures. Following the procedure, the medical instruments are moved to a sterile processing department for cleaning, sterilization, and preparation for other procedures.SUMMARY

[0004] This specification relates to systems that detect and classify medical instruments using imaging data. The disclosed embodiments can be implemented in a medical environment to detect and classify medical tools before, during, and after cleaning and sterilization processes. The disclosed techniques can be implemented to track movements of medical instruments throughout medical facilities, and to monitor the conditions of medical instruments at various steps of sterilization processes.

[0005] This specification relates to systems that combine multiple sensors and devices, including without limitation, edge computers, three-dimensional stereographic cameras, thermal imaging cameras; all of which can be augmented by software comprising artificial intelligence algorithms (e.g., modified “you only look once” (YOLO) models using transfer learning, 1 -dimensional and 2-dimensional convolutional neural networks, human pose estimation and human hands and digits position detection via deep neural networks, natural language processing via deep neural networks, multilayer perceptrons, decision trees, and random forest search) and non-AI models (e.g., combination of machine vision image treatment methods. Fast Fourier Transform, cascade classifier, mahalanobis distance,connected components labelling algorithm) for detecting, analysing, and evaluating medical instruments captured in video images or still images.

[0006] In general, innovative aspects of the subject matter described in this specification can be embodied in methods that include the actions of receiving an image generated by an imaging device; providing the image as input to a medical instrument detection model; detecting, by the medical instrument detection model, a medical instrument in the image; in response to detecting the medical instrument in the image, providing the image as input to a medical instrument classification model; determining, by the medical instrument classification model, a classification of the medical instrument as a particular t pe of medical instrument: annotating the image with a label indicating the particular type of medical instrument to generate an annotated image; and providing the annotated image for presentation by an electronic display.

[0007] These and other implementations can each optionally include one or more of the following features. In some implementations, the actions include providing the image as input to a medical instrument evaluation model; and determining, by the medical instrument evaluation model, a condition of the medical instrument.

[0008] In some implementations, the condition of the medical instrument includes a cleanliness rating of the medical instrument.

[0009] In some implementations, the actions include: determining that the cleanliness rating of the medical instrument does not satisfy a cleanliness threshold; and in response to determining that the cleanliness rating of the medical instrument does not satisfy cleanliness threshold, generating a notification indicating that the cleanliness rating of the medical instrument does not satisfy cleanliness threshold.

[0010] In some implementations, the actions include: determining that the cleanliness of the medical instrument does not satisfy a cleanliness threshold; and in response to determining that the cleanliness rating of the medical instrument does not satisfy the cleanliness threshold, updating a status of the medical instrument in an inventory management database indicating that the cleanliness rating of the medical instrument does not satisfy the cleanliness threshold.

[0011] In some implementations, the condition of the medical instrument includes a damage rating of the medical instrument.

[0012] In some implementations, the actions include: determining that the damage rating of the medical instrument does not satisfy a damage threshold; and in response to determining that the damage rating of the medical instrument does not satisfy the damage threshold.generating a notification indicating that the damage rating of the medical instrument does not satisfy the damage threshold.

[0013] In some implementations, the actions include: determining that the damage rating of the medical instrument does not satisfy a damage threshold; and in response to determining that the damage rating of the medical instrument does not satisfy7the damage threshold, updating a status of the medical instrument in an inventory management database indicating that the damage rating of the medical instrument does not satisfy the damage threshold.

[0014] In some implementations, the medical instrument includes one or more sharp edges, and the condition of the medical instrument includes a sharpness rating of the one or more sharp edges.

[0015] In some implementations, the actions include: determining that the sharpness rating of the medical instrument does not satisfy7a sharpness threshold; and in response to determining that the sharpness rating of the medical instrument does not satisfy the sharpness threshold, generating a notification indicating that the sharpness rating of the medical instrument does not satisfy the sharpness threshold.

[0016] In some implementations, the actions include: determining that the sharpness rating of the medical instrument does not satisfy7a sharpness threshold; and in response to determining that the sharpness rating of the medical instrument does not satisfy the sharpness threshold, updating a status of the medical instrument in an inventory management database indicating that the sharpness rating of the medical instrument does not satisfy the sharpness threshold.

[0017] In some implementations, the actions include: in response to determining the classification of the medical instrument as the particular type of medical instrument, updating a quantity of the particular type of medical instrument in an inventory7management database.

[0018] In some implementations, the particular type of object includes a scalpel, a blade, a knife, a scissor, a forcep, a clamp, a retractor, or a needle.

[0019] In some implementations, the medical instrument detection model includes a first machine learning model, and detecting, by the medical instrument detection model, the medical instrument in the image includes applying the first machine learning model to the image.

[0020] In some implementations, the medical instrument classification model includes a second machine learning model, and determining, by the medical instrument classification model, the classification of the medical instrument as the particular type of medical instrument includes applying the second machine learning model to the image.

[0021] In some implementations, the medical instrument classification model includes a machine learning model that is trained to determine classifications of medical instruments based on shapes of the medical instruments.

[0022] In some implementations, the medical instrument classification model includes a machine learning model that is trained to determine classifications of medical instruments based on visual identifiers marked on the medical instruments.

[0023] In some implementations, the visual identifiers include one or more of quick response codes, bar codes, typographical labels, numeric identifiers, and hashmarks.

[0024] In some implementations, the actions include: receiving sensor data from a sensor that is configured to detect non-visible energy; and providing the sensor data as input to the medical instrument detection model: wherein the medical instrument classification model comprises a machine learning model that is trained to determine classifications of medical instruments based on non-visible energy' emitted by the medical instruments and shapes of the medical instruments.

[0025] In some implementations, the medical instrument classification model includes a machine learning model that is trained to determine classifications of medical instruments based on (a) shapes of the medical instruments and (b) visual identifiers marked on the medical instruments.

[0026] The disclosed embodiments can be implemented by one or more non-transitory computer readable storage media storing instructions that, when executed by at least one processor, cause the at least one processor to perform the actions of any of the preceding implementations.

[0027] The disclosed embodiments can be implemented by a medical instrument tracking system including: an imaging device positioned to capture images of a workstation within a medical instrument cleaning facility; one or more processors in electronic communication with the imaging device; and one or more tangible, non-transitory' media operably connectable to the one or more processors and storing instructions that, when executed, cause the one or more processors to perform the actions of any of the preceding implementations.

[0028] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. Implementations provide a system that is capable of quickly detecting and classifying medical instruments in a medical environment. The system can use a multi-factor recognition technique to classify' instruments into categories of instruments, improving accuracy. For example, the system can classify’ and identify an instrument based on a shape of the instrument in an image, based onvisual identifiers that are marked on the instrument, based on sensor emissions output by the instrument, or any combination thereof. The system can use the multi-factor recognition technique to accurately differentiate between multiple instruments that are near each other and / or overlapping on a work surface.

[0029] The disclosed techniques can be used to evaluate cleanliness and conditions of medical instruments. The automatic evaluation of conditions of medical instruments can reduce potential sources of contamination, thereby reducing the risk of post-surgical adverse effects. The detection of medical instruments as described herein can be used to track medical instruments, reducing the likelihood of losing medical instruments.

[0030] Implementations may improve the patient outcomes following invasive medical procedures. For example, implementations may prevent events that could cause postoperative infections in patients. Implementations may significantly improve sanitation within medical facilities where medical procedures are performed.

[0031] The disclosed techniques can improve efficiency , accuracy , and overall instrument management within Sterile Processing Department (SPD) departments. The disclosed techniques can improve inventory management by accurately tracking instruments, their locations, and usage history in real-time. This can reduce instances of lost or misplaced instruments, ensuring the availability of necessary' instruments, and optimizing inventory' levels.

[0032] The disclosed techniques can enhance sterilization processes. Sterilization cycles can be monitored and analyzed, ensuring that instruments are properly cleaned and sterilized according to standards. This can reduce the risk of infections and improve patient safety.

[0033] The disclosed techniques can reduce errors. Errors in instrument processing can be detected, such as incorrect cleaning or packaging. This can reduce the risk of using compromised instruments on patients.

[0034] The disclosed techniques can improve compliance and documentation. Accurate documentation and compliance with regulatory' requirements can be enhanced by automatically recording instrument usage, sterilization cycles, and maintenance history.

[0035] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the invention will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0036] FIG. 1 depicts an example medical instrument classification and tracking system.

[0037] FIG. 2 depicts an example computing system for medical instrument classification and tracking.

[0038] FIG. 3 depicts a screenshot of an exemplary' graphical user interface output from a medical instrument classification and tracking system showing an annotated image of a work surface.

[0039] FIGS. 4A and 4B depict screenshots of exemplary graphical user interfaces output from a medical instrument classification and tracking system showing instrument tracking tables.

[0040] FIG. 5 depicts a flowchart of an example process for classifying and tracking medical instruments.

[0041] FIG. 6 depicts a block diagram of a computer system that may be applied to any of the computer-implemented methods and other techniques described herein.

[0042] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0043] Embodiments of the present disclosure are directed at systems that detect and classify medical instruments using imaging data. The disclosed embodiments can be implemented in a medical environment to detect and classify medical instruments such as surgical tools.

[0044] FIG. 1 depicts an example medical instrument detection and classification system 100. The system 100 includes a computing device 110, a camera 102, and a data reader 132.

[0045] The camera 102 is an imaging device. The camera 102 can detect visible light, infrared light camera, ultraviolet light, or any combination of these. In some examples, the camera 102 is a thermal imager or ultrasonic imager. The camera 102 has a field of view 105 that includes a workstation 104. The camera 102 is configured to generate an image 130 of the workstation 104.

[0046] The data reader 132 is a sensor that is configured to detects emissions 117 from emitter(s) attached to medical instrument(s). The data reader 132 can be, for example, a radio frequency identification (RFID) reader, a Bluetooth sensor, a magnetic sensor, a Zigbee sensor. aZ-wave sensor, a near field communication (NFC) sensor, or any combinationthereof. The data reader 132 is configured to output sensor data 133 to the computing device 110 representing the detected emission 117.

[0047] An emitter 107 can be attached to an instrument such as a scalpel 101. The emitter 107 can be, for example, an RFID emitter, a Bluetooth emitter, a magnetic wave emitter, a Zigbee signal emitter, a Z-wave signal emitter, a near field communication (NFC) emitter, or any combination thereof.

[0048] The emitter 107 can be attached externally to an instrument or can be embedded within the instrument. The emitter 107 outputs emissions 117 that are detectable by the data reader 132. The emissions 117 can include non-visible energy' such as non-visible electromagnetic energy'. The emissions 117 can represent a category', subcategory', brand, and / or unique identifier for the instrument.

[0049] The workstation 104 includes a surface 103 on which instruments 121 can be placed. In the example of FIG. l, the instruments 121 includes a scissor 111, a scalpel 101, and a clamp 131, among other instruments. The instruments 121 can be placed on the surface 103 in order to survey the instruments. In some examples, the workstation 104 is located in a cleaning facility.

[0050] Medical instruments 121 can include, but are not limited to, scissors, scalpels, scalpel blades, scalpel handles, knives, forceps, clamps, retractors, and needles. Examples of instruments 121 that the system 100 can identify include, without limitation, scissors (e.g., Mayo scissors. Metzenbaum scissors, Potts scissors, and iris scissors), pickups / forceps (e.g.. tissue forceps, Adson forceps, Ferris-Smith forceps, Bonney forceps, DeBakey forceps, Kocher forceps, right angle forceps, and Russian forceps), clamps / hemostats (e.g., Crile hemostatic clamps. Kelly clamps, and Allis-Babcock clamps), retractors (e.g., rake retractors, Volkman sharp retractors, Richardson retractors, flat retractors, malleable retractors, pickle fork retractors. Army -Navy retractors, Deaver retractors, Z-retractors, Hohmann retractors, cobra retractors, scissor-esque retractors, Gelpi retractors, Weitlaner retractors, and Bookwaiter retractors), rongeurs (e.g., Adson rongeurs, and double action rongeurs), spreaders (e.g.. Lamina spreaders), mallets (e.g., ortho heavy mallets), saws (e.g., bone saws), power equipment (e.g., power drills, reamers, and pin-drivers), pliers, bone hooks, metal rulers, chisels, osteotomes (e.g., Lambotte osteotomes), and laparoscopic instruments.

[0051] In an example scenario, instruments 121 enter the cleaning facility' in a basket 106. The medical instruments in the basket 106 can be. for example, medical instruments that w ere used in an operating room prior to delivery to the cleaning facility.

[0052] A worker 122 moves the instruments 121 from the basket 106 to the surface 103 of the workstation 104 in order to survey the instruments. In some examples, the instruments 121 are spread out on the surface 103, such that the instruments do not overlap each other. The worker 122 can place multiple instruments on the surface 103 simultaneously, or can place a single instrument on the surface 103 at a given time.

[0053] The instruments 121 can remain on the surface 103 for at least a specified time duration during which the instruments 121 are surveyed. The specified time duration can be. for example, one second or more, three seconds or more, five seconds or more, ten seconds or more. To survey the instruments 121 on the surface of the workstation 104, the camera 102 generates one or more images of the workstation 104, the data reader 132 detects emissions from emitters attached to medical instruments, or both. In some examples, the camera 102 generates video images. In some examples, the camera 102 generates still images.

[0054] In some examples, the camera 102 generates images continuously. For example, the camera 102 can generate video images of the workstation continuously throughout a work day, throughout a shift, or indefinitely. The camera 102 can generate video images when instruments are positioned on the surface 103, and when instruments are not positioned on the surface 103.

[0055] In some examples, the camera 102 selects image frames of the video for image processing. For example, the camera 102 can select, from a set of video images, a subset image frames that depict at least some of the instruments 121. The camera 102 can perform image processing on the selected image frames and / or can send the selected image frames to the computing device 110 for image processing.

[0056] In some examples, instead of generating images continuously, the camera 102 generates images automatically in response to a trigger event. The camera 102 can be maintained in a default “sleep’’ or “off’ mode, and can automatically switch to an “on” mode in response to the trigger event. The trigger event can include an event detected by the camera 102 and / or another sensor. In an example, the camera 102 receives motion sensor data from an integrated sensor such as a motion sensor 115 (e.g., a passive infrared (PIR) sensor). When the camera 102 receives motion sensor data indicating that the motion sensor 115 detected movement in the field of view 105, the camera 102 can automatically “wake” and initiate generating image data. After a programmed time duration has elapsed without motion being detected by the motion sensor, the camera 102 can automatically return to the “sleep” or “off’ mode.

[0057] Motion detected within the field of view 105 of the camera 102 can be caused by the worker 122 placing the instruments 121 on the surface 103, repositioning the instruments 121 on the surface 103, removing the instruments 121 from the surface 103, or any combination thereof. In some examples, the camera 102 determines, based on the motion sensor data, when the instruments 121 have been placed on the surface 103 and are no longer in motion. In response to determining that the instruments 121 have been placed on the surface 103, the camera 102 can begin generating images and / or can begin sending the images to the computing device 110 for image processing.

[0058] In an example, the camera 102 receives motion sensor data indicating that motion is detected by the motion sensor 115 over a first time duration. Following the detected motion over the first time duration, the motion sensor data indicates no motion detected for a second time duration. After a threshold time duration of no motion detected has elapsed, the camera 102 can determine that the instruments 121 are not in motion w ithin the field of view 105. In response to determining that the instruments 121 are not in motion within the field of view, the camera 102 can begin generating images and / or can begin sending images to the computing device 110.

[0059] In some examples the camera 102 automatically sends all of the generated image data to the computing device 110 for image processing (e.g., by streaming video to the computing device 110). In some examples the camera 102 selects a subset of image frames to send to the computing device 110. as described above.

[0060] In some examples, the camera 102 determines that no objects are present within the field of view 105, and determines to not send the image data to the computing device 110. For example, the motion sensor 115 can detect motion due to the worker 122 removing instruments from the surface 103. After a threshold time duration has elapsed following the end of motion detection, the camera 102 begins generating images. The camera 102 can determine, based on the generated images, that no objects are on the surface 103, and in response can determine to not send the images to the computing device 110. In some examples, in response to determining that no objects are on the surface 103, the computing device 110 can cease generating images (e.g.. by switching to a "‘sleep” or “off’ mode).

[0061] The images 130 generated by the camera 102, the sensor data 133 output by the data reader 132, or both, are processed by the computing device 110 using machine learning algorithms to detect the presence (or absence) of medical instruments and devices on the workstation 104. The computing device 110 classifies detected instruments into categories according to instrument type, as described in greater detail with reference to FIG. 2.

[0062] The worker 122 can move the instruments 121 away from the workstation 104. In some examples, the worker 122 moves the instruments away from the workstation 104 after the specified time duration has elapsed. In some examples, the worker 122 moves the instruments 121 away from the workstation 104 after verifying that the instruments 121 have been processed (e.g., photographed, detected, classified, categorized, and / or counted) by the system 100.

[0063] In some examples, the worker 122 views a user interface 118 presented on a display 140 that indicates a status of processing each of the instruments 121. The worker 122 can determine, based on viewing the user interface 118, when the system 100 has completed processing the instruments 121, and in response, remove the instruments 121 from the workstation 104. In some examples, the user interface 118 displays real-time or near-realtime images generated by the camera 102. The user interface 118 can also display a table of instruments, with fields of the table indicating a recognition status of the instruments 121. The recognition status can be, for example, ‘‘recognized,’' “pending,’" or “not recognized.” The worker 122 can remove an instrument from the workstation 104 when the recognition status shown in the table is “recognized” for the instrument. When an instrument has a status of “not recognized,” the worker 122 can perform manual identification and / or labeling of the instrument, or can remove the unrecognized instrument from the set of instruments.

[0064] The worker 122 can move the instruments 121 from the surface 103 of the workstation 104 to a tray 108. The tray 108 with the medical instruments 121 can then be transported to a next station at the cleaning facility. In some examples, the tray 1 8 is transported to a washing machine or to a sterilization station.

[0065] In the scenario described above, the instruments 121 are surveyed at the workstation 104 after arriving at the cleaning facility. However, medical instruments can be surveyed at various stages of cleaning and transport of medical instrument. For example, the instruments 121 can be surveyed at the workstation 104 before washing the instruments in a washing machine, after washing the medical instruments in a washing machine, before performing sterilization procedures, after performing sterilization procedures, immediately before sending the instruments to a medical provider (e.g., to an operating room), immediately after receiving the instruments from a medical provider, or any combination thereof.

[0066] A survey of instruments is an instance of capturing images of a set of instruments, reading data emissions from the set of instruments, or a combination of both. In some examples, the set of instruments includes instruments that are proceeding through a cleaningprocess together in a batch. For example, the basket 106 may contain a set of instruments that are to be washed together in a washing machine. A first subset of the set of instruments are placed on the workstation 104, and the camera 102 captures images of the first subset. A second subset of the set of instruments are placed on the workstation, and the camera 102 captures images of the second subset. The process repeats until all instruments in the basket 106 have been captured in at least one image captured by the camera 102, at which point the survey can be considered complete.

[0067] In some examples, the system 100 can be reset between surveys. For example, after performing a survey of a set of instruments, the worker 122 can provide input to the camera 102 and / or to the computing device 110 indicating that the survey is complete. The system can be reset such that, for example, counted quantities of instruments return to zero prior to a next survey of a next set of instruments.

[0068] Images generated by the camera 102, the sensor data 133 output by the data reader 132, or both, are processed by the computing device 110 using machine learning algorithms to detect the presence (or absence) of medical instruments and devices on the workstation 104. In some examples, the system 100 automatically resets after a threshold time duration of time has elapsed without any instruments being placed at the workstation 104. The threshold time duration can be, for example, one minute or more, five minutes or more, ten minutes or more.

[0069] Surveys performed at various stages of a cleaning process can be compared to each other. For example, the computing device 1 10 can compare images and / or sensor data captured during a pre-sterilization survey to images captured during a post-sterilization survey to determine whether the quantity and / or conditions of the instruments have changed during sterilization.

[0070] In some examples, the instruments 121 are surveyed at the workstation multiple times at different stages of a cleaning process. For example, the instruments 121 can be surveyed before and after the instruments 121 are washed in a washing machine, so that the quantity and / or condition of the instruments 121 after washing can be compared to the quantity and / or condition of the instruments before washing. Similarly, the instruments 121 can be sun- eyed before the instruments 121 are sent to an operating room and after the instruments 121 are received from the operating room, so that the quantity and / or condition of the instruments 121 upon return from the operating room can be compared to the quantity and / or condition of the instruments prior to sending the instruments 121 to the operating room.

[0071] The camera 102 sends images(s) of the workstation (e.g., image 130) to the computing device 110. The computing device 110 can be a computer system or other electronic device configured to communicate with components of the system 100 to cause various functions to be performed for the monitoring system 100. The computing device 110 may include a processor, a chipset, a memory system, or other computing hardware. In some examples, the computing device 110 is a cloud computing platform. In some cases, the computing device 110 may include application-specific hardware, such as a field- programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or other embedded or dedicated hardware. The computing device 110 may include software, which configures the device to perform the functions described in this disclosure. In some implementations, a user communicates with the computing device 110 through a physical connection (e.g., touch screen, keypad, etc.) and / or network connection. In some implementations, the user communicates with the computing device 110 through a software application, such as an application installed on a mobile device or another computing system.

[0072] The computing device 110 receives the image 130 generated by the camera 102. The computing device 110 can analyze the image 130 to detect, classify, and / or evaluate medical instruments depicted in the image 130. Operations of the computing device 110 are described in greater detail with reference to FIG. 2.

[0073] In some examples, the computing device 110 analyzes a set of images generated by the camera 102 over a particular time duration and classifies medical instruments in each of the set of images. In some examples, the computing device 110 generates an annotated image 112 by applying labels to the image 130. The annotated image 112 can be provided for presentation by a display such as display 140. An example annotated image presented through a user interface on a display is shown and described with reference to FIG. 3.

[0074] In some examples, the computing device 110 evaluates a condition of the medical instruments depicted in the image 130. The condition can be a cleanliness of the medical instrument, a damage rating of the medical instrument, a sharpness of the medical instrument, or any combination of these. In response to evaluating the condition of the medical instrument, the computing device can output a notification 114 indicating the condition. In some examples, the notification 114 indicates that the condition of the medical instrument does not satisfy one or more criteria.

[0075] In some examples, the computing device 110 counts the medical instruments in the image 130 and generates an inventory update 116. The inventory update 116 can indicate a quantity of medical instruments depicted in the image 130. In some examples, the inventoryupdate 116 indicates a quantity of each of multiple types of instruments depicted in the image 130. For example, the inventory update 116 can indicate that the image 130 depicts one forcep, one scissor, one scalpel, and one knife. The inventory update 116 can be output to an inventory management database. In some examples, the inventory management database is hosted by the computing device 110 or another computing device. In some examples, the inventory management database is cloud-based.

[0076] The inventory update 116 can be used to update a table 124 of medical instruments. In some examples, the table 124 is presented through a user interface 118 on a display 140. Example tables are shown and described with reference to FIGS. 4A and 4B.

[0077] FIG. 2 depicts the example computing device 110 for medical instrument classification and tracking. The computing device 110 can include an instrument detector 210, a classifier 214, a condition evaluator 224, a counter 226, an image labeler 222, or any combination thereof.

[0078] The instrument detector 210, the classifier 214, and the condition evaluator 224 can each include one or more machine learning models. A machine learning model can be. for example, a deep learning model that employs multiple layers of models to generate an output for a received input. A deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each applies a non-linear transformation to a received input to generate an output. In some cases, the neural network may be a recurrent neural network. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recurrent neural network uses some or all of the internal state of the network after processing a previous input in the input sequence to generate an output from the current input in the input sequence. In some other implementations, the machine learning model is a convolutional neural network. In some implementations, the machine learning model is an ensemble of models that may include all or a subset of the architectures described above.

[0079] In some implementations, the machine learning model can be a feedforward autoencoder neural network. For example, the machine learning model can be a threelayer autoencoder neural network. The machine learning model may include an input layer, a hidden layer, and an output layer. In some implementations, the neural network has no recurrent connections between layers. Each layer of the neural network may be fully connected to the next, such that there may be no pruning betw een the layers. The neural network may include an ADAM optimizer, or any other multi-dimensional optimizer, for training the network and computing updated layer weights. In some implementations, theneural network may apply a mathematical transformation, such as a convolutional transformation, to input data prior to feeding the input data to the neural network.

[0080] In some implementations, the machine learning model can be a supervised model. For example, for each input provided to the model during training, the machine learning model can be instructed as to what the correct output should be. The machine learning model can use batch training (training on a subset of examples before each adjustment) instead of training based on the entire available set of examples. This may improve the efficiency of training the model and may improve the generalizability of the model. The machine learning model may use folded cross-validation. For example, some fraction ("fold") of the data available for training can be left out of training and used in a later testing phase to confirm how well the model generalizes. In some implementations, the machine learning model may be an unsupervised model. For example, the model may adjust itself based on mathematical distances between examples rather than based on feedback on its performance.

[0081] The instrument detector 210 analyzes the image 130 and determines whether or not the image 130 generated by the camera 102 depicts a medical instrument. The instrument detector 210 can include one or more machine learning models that are trained to detect a medical instrument in images. In some examples, the instrument detector 210 is trained to recognize shapes of medical instruments. The instrument detector 210 can be trained to receive image data as input, and to generate a predicted output, e.g., an estimate of whether or not a medical instrument is depicted in the image data. In some examples, the instrument detector 210 is trained to identify a location of the medical instrument in an image.

[0082] The instrument detector 210 analyzes the sensor data 133 and determines whether or not the sensor data 133 indicates the presence of a medical instrument at the workstation 104. For example, sensor data 133 may indicate that an emission 117 (e.g., an RFID emission) received by the data reader 132 represents the presence of the scalpel 101 at the workstation 101.

[0083] In response to determining that the image 130 depicts a medical instrument, the instrument detector 210 can output the indication of instrument detection 212 to the classifier 214. In some examples, the indication of the instrument detection 212 includes a bounding box around the depiction of the instrument in the image 130.

[0084] The classifier 214 classifies detected instruments into a particular category' of instruments. For example, the classifier 214 can classify each instrument detected in the image 130 as a forcep. a retractor, a clamp, a knife, a scalpel, a scissor, a stapler, or another category of instrument. In some examples, the classifier 214 further classifies an instrumentas a specific subcategory of instrument (e.g., Weitlaner retractor, Adson Forcep, linear stapler). In some examples, the classifier 214 classifies an instrument as a specific size of instrument (e.g., 4 inch retractor, 1.5mm tip Forcep). In some examples, the classifier 214 classifies an instrument as a specific brand of instrument (e.g., Thompson® retractor, Bailey® Forcep).

[0085] The classifier 214 can include one or more machine learning models that are trained to classify medical instruments depicted in image data. The classifier 214 can be trained to receive image data as input, and to generate a predicted output, e.g., an estimate of a type of instrument depicted in the image data. In some examples, the classifier 214 is trained to recognize shapes of different categories of medical instruments. In some examples, the classifier 214 is trained to receive, as input, image data and indications of instrument detection 212, and to output the estimate of the type of detected instrument.

[0086] An instrument can be marked with one or more visual identifiers. A visual identifier can be associated with a category of instruments, a subcategory of instruments, a brand of instruments, an individual instrument, or any combination thereof. The visual identifier can include, for example, a bar code, a quick response (QR) code, a typographical label, a numeric identifier, and / or a hashmark. The visual identifier can be marked on the instrument, for example, by applying a label to the instrument, printing the visual identifier onto the instrument, or engraving the visual identifier onto the instrument. For example, referring to FIG. 3. the scissor 311 is marked with a visual identifier that is a bar code 320.

[0087] The classifier 214 can include one or more machine learning models that are trained to detect visual identifiers marked on instruments, interpret the visual identifiers, and classify instruments based at least in part on visual identifiers marked on the instruments. In some examples, the classifier 214 classifies an instrument based on only a visual identifier. For example, the classifier 214 can perform image processing on the image data in order to interpret a typographical label printed on a scissor that says “scissor.” Based on interpreting the typographical label, the classifier 214 can classify the instrument into a “scissor” category.

[0088] In some examples, the classifier 214 classifies an instrument based on the appearance (e g., shape) of the instrument in the image data, and based on a visual identifier marked on the instrument. For example, the classifier 214 can recognize a shape of the scalpel 101 as belonging to the category “scalpel” and can interpret a QR code marked on the scalpel 101. The classifier 214 can determine that the QR code corresponds to a particular subcategory of scalpel. The classifier 214 therefore outputs a classification 216 of the scalpel101 as the particular subcategory (as determined based on the QR code) within the scalpel category (as determined based on the shape of the instrument). In this way. the classifier 214 can perform multi-factor recognition of instruments.

[0089] The classifier 214 can compare a classification of the instrument determined using the shape of the instrument in the image data to a classification of the instrument determined using a visual identifier. For example, the classifier 214 may classify an instrument as a clamp based on the shape of the instrument in the image data. The classifier 214 may classify the instrument as a retractor based on a numeric identifier marked on the instrument. The classifier 214 can determine that the classification of clamp, determined based on the shape of the instrument, does not match the category of retractor, determined based on the numeric identifier. In some cases, in response to determining that the classifications do not match, the classifier 214 can output a classification of ‘’Unrecognized.”

[0090] In some cases, the classifier 214 is configured to prioritize some classification techniques over other classification techniques. For example, the classifier 214 can be trained to prioritize classification using a visual identifier over classification based on a shape of the object in the image. In response to determining that the classifications do not match, the classifier 214 can output a classification determined using the higher priority classification technique. In the example above, the classifier 214 prioritizes classification using the visual identifier and outputs a classification 216 of “retractor,” instead of outputting a classification of “clamp,” which was determined using the lower priority classification of shape recognition.

[0091] The classifier 214 can compare a classification of the instrument determined using the shape of the instrument in the image data to a classification of the instrument determined using the sensor data 133. For example, the classifier 214 may classify an object at the workstation as a scissor based on the shape of the instrument in the image data. The classifier 214 may classify7the object as a scalpel based on the sensor data 133. The classifier 214 can determine that the classification of scissor, determined based on the shape of the instrument, does not match the category of scalpel, determined based on the sensor data 133. In some cases, in response to determining that the classifications do not match, the classifier 214 can output a classification of “Unrecognized.”

[0092] In some cases, the classifier 214 is configured to prioritize some classification techniques over other classification techniques. For example, the classifier 214 can be trained to prioritize classification using sensor data 133 over classification based on a shape of the object in the image 130. In response to determining that the classifications do not match, theclassifier 214 can output a classification determined using the higher priority classification technique. In the example above, the classifier 214 prioritizes classification using the sensor data 133 and outputs a classification 216 of '‘scalpel,” instead of outputting a classification of “scissor,” which was determined using the lower priority classification of shape recognition.

[0093] In an example scenario, two scalpels lay on the surface 103 and overlap each other. The overlapping scalpels appear, in the image 130, as an object that has a shape similar to a scissor. The instrument detector 210 determines that the object formed by the overlapping scalpels are likely a medical instrument. The classifier 214 determines, based on the shape, that the object is likely a scissor. The sensor data 133 indicates two RFID signals, both corresponding to scalpels. The classifier 214 prioritizes the sensor data 133 over the image analysis, and outputs a classification 216 of the object as two scalpels instead of a scissor.

[0094] The classifier 214 determines the classification 216 of the instrument and outputs the classification 216 to the image labeler 222, the condition evaluator 224, the counter 226, or any combination thereof.

[0095] The image labeler 222 labels the image 130. The image labeler 222 can label the image 130 based on the instrument detection 212, the classification 216, or both.

[0096] In some examples, the image labeler 222 receives condition information 225 from the condition evaluator 224. The condition information 225 indicates the condition of the detected instruments. In some examples, the image labeler 222 receives an inventory update 116 from the counter 226. The image labeler 222 can use the condition information 225 and the inventory update 1 16 to generate relevant labels regarding the conditions of the instruments and the instrument counts.

[0097] The image labeler 222 outputs an annotated image 112 for presentation through a user interface of the display 140. FIG. 3 depicts a screenshot of an exemplary graphical user interface output from a medical instrument classification and tracking system showing an annotated image of a work surface. Referring to FIG. 3, a user interface 300 can show labels 306a-c indicating the classifications 216 of various instruments depicted in the image 130. A scalpel 101 has a label 306a. a clamp 131 has a label 306b, and scissors 111, 311 have labels 306c- 1 and 306c-2.

[0098] The labels 306a-c are overlaid on image data generated by the camera 102. In some examples, the labels 306a-c are overlaid over video data streams generated by the camera 102 and provided to the computing device 110. The image labeler 222 can color code instruments by type. For example, a bounding box 312 around a scalpel 101 displayed through the user interface 300 can be pink, while a bounding box 314 around a clamp 131 displayed throughthe user interface 300 can be orange, and a bounding box 316 around a scissor 111 displayed through the user interface 300 can be green.

[0099] In some examples, the image labeler 222 can color code instruments by an identification or classification status. For example, when an instrument is identified as an instrument, but not yet classified as a particular instrument, the user interface 300 can show a green bounding box around the instrument. When the instrument is classified as a particular type of instrument, the user interface can change the green bounding box to a blue bounding box around the instrument. In some examples, the image labeler 222 first labels all detected instruments in the image 130, as indicated by the instrument detection 212 received from the instrument detector 210. The image labeler 222 can then update the labels of the detected instruments based on the classification 216 received from the classifier 214.

[0100] The labels 306a-c can include the classification of each medical instrument depicted in the image generated by the camera (e.g., scissor, clamp., scalpel). In some examples, the labels can indicate additional information about the medical instruments, such as a condition of the instruments and / or a quantity of the instruments. The labels 306a-c can be generated by the image labeler 222 using the instrument detection 212, the classification 216, the condition information 225, the inventory update 116, or any combination thereof. In some examples, labels are presented on the user interface 300 according to user preferences. For example, the worker 122 may provide input indicating a user preference to view labels indicating the classification and condition of instruments, and to not view labels indicating the inventory, or quantity, of the instruments.

[0101] For example, the annotated image presented on the user interface 300 includes a label 306a for the scalpel 101 that says “scalpel.” The label 306a also indicates the condition as “requires sharpening.” In some examples, the annotated image shown on the user interface 300 can include a label of a quantity of instrument. For example, the label 306c-l indicates that the scissor I l l is the first scissor detected in the set of instruments, with the label reading “Scissor: 1.” The label 306c-2 indicates that the scissor 311 is the second scissor detected in the set of instruments surveyed by the camera 102, with the label reading “Scissor: 2.”

[0102] Referring back to FIG. 2, the condition evaluator 224 evaluates one or more conditions of the instruments detected in the image 130. The condition evaluator 224 can evaluate conditions such as cleanliness of the instruments, sharpness of the instruments, and damage to the instruments. The condition evaluator 224 can include one or more machine learning models that are trained to identify evaluate conditions of instruments based on image data. The condition evaluator 224 can be trained to receive image data as input, and togenerate a predicted output, e.g., an estimated condition evaluation 225. The condition evaluation 225 can include a cleanliness rating, a sharpness rating, a damage rating, or any combination of these ratings.

[0103] The condition evaluator 224 can evaluate a cleanliness of the instrument. The condition evaluator 224 can compare the cleanliness of the instrument to cleanliness criteria. In some examples, the cleanliness criteria depends on a stage of a cleanliness process. For example, cleanliness criteria for instruments that have exited a washing machine may be more strict, requiring a higher degree of cleanliness, than cleanliness criteria for instruments that are entering a washing machine. In another example, cleanliness criteria for instruments returning from the operating room may be less strict, requiring a lesser degree of cleanliness, than cleanliness criteria for instruments that are being sent to the operating room.

[0104] In some examples, the cleanliness criteria includes a maximum size of visible residue on the medical instruments. In some examples, the cleanliness criteria includes a maximum percentage surface area of the instrument that is covered with residue.

[0105] In some examples, the condition evaluation 225 includes a cleanliness rating, or score, for the instrument. The condition evaluator 224 can compare the detected cleanliness rating to a threshold rating. In an example, the condition evaluator 224 can determine that a cleanliness rating of a scissor is 7 on a scale from 1 to 10, where 1 represents the least clean and 10 represents the most clean. The cleanliness threshold at the particular cleaning stage at which the instrument is being surveyed may be a rating of 6 or greater. Therefore, the condition evaluator 224 can determine that the cleanliness rating satisfies the cleanliness criteria due to the cleanliness rating of 7 exceeding the threshold rating of 6.

[0106] The condition evaluator 224 can evaluate a sharpness of one or more sharp edges of the instrument. The condition evaluator 224 can compare the detected sharpness of the instrument to sharpness criteria. In some examples, the condition evaluation 225 includes a sharpness rating, or score, for the instrument. The condition evaluator 224 can compare the sharpness rating to a threshold rating. In an example, the condition evaluator 224 can determine that a sharpness rating of a knife is 5 on a scale from 1 to 10. where 1 represents the least sharp and 10 represents the most sharp. The sharpness threshold may be a rating of 7 or greater. Therefore, the condition evaluator 224 can determine that the sharpness rating does not satisfy the sharpness criteria due to the sharpness rating of 5 being less than the threshold rating of 7. In some examples, sharpness criteria can be different for different types of instruments. For example, sharpness criteria for a scissor may be different than sharpness criteria for a scalpel.

[0107] The condition evaluator 224 can evaluate a damage of the instrument. Damage of the instrument can include, for example, a surface defect such as a chip or scratch. Damage can include an instrument being bent, stretched, or snapped. In some examples, damage includes a loose fastener, such as a loose screw or rivet in a pair of scissors.

[0108] The condition evaluator 224 can compare the damage of the instrument to damage criteria. The damage criteria can include, for example, a maximum size of a visible defect on the medical instrument, a maximum quantity of visible defects on the instrument, a maximum amount of bend of the instrument, or any combination thereof.

[0109] In some examples, the condition evaluation 225 includes a damage rating, or score, for the instrument. The condition evaluator 224 can compare the damage rating to a threshold rating. In an example, the condition evaluator 224 can determine that a damage rating of a retractor is 2 on a scale from 1 to 10, where 1 represents the least damage and 10 represents the most damage. The damage threshold may be a rating of 3 or less. Therefore, the condition evaluator 224 can determine that the damage rating satisfies the damage threshold due to the damage rating of 2 being less than the threshold rating of 3.

[0110] The condition evaluator 224 can generate a notification 114 for presentation on a display 140. The notification 114 can indicate a condition of the instrument. In some examples, the notification 114 indicates that the cleanliness rating does not satisfy cleanliness criteria, that the sharpness rating does not satisfy sharpness criteria, that the damage rating does not satisfy’ damage criteria, or any combination thereof.[OH l] In some examples, the notification 114 generated by the condition evaluator 224 includes a recommended action to be taken. For example, based on the cleanliness rating, the condition evaluator 224 can generate a notification 114 recommending that the instrument be sent to a washing machine or another cleaning station. In another example, based on the sharpness rating, the condition evaluator 224 can generate a notification 114 recommending that the instrument, or a particular edge of the instrument, be sharpened. In another example, based on the damage rating, the condition evaluator 224 can generate a notification 114 recommending that the instrument be repaired or replaced.

[0112] In some examples, the condition evaluator 224 compares the condition evaluation 225 to a previously detected condition of the respective instruments. For example, the condition evaluator 224 may determine a condition of each of a set of instruments prior to sending the set of instruments to an operating room, and a condition of the same set of instruments after the set of instruments returns from the operating room. The condition evaluator 224 can compare the detected conditions for the set of instruments at the differentstages to determine whether the conditions of the instruments have changed over time. In some examples, the condition evaluator 224 generates a notification 114 indicating that the condition of the instrument has changed compared to a previous survey. The notification 1 14 indicating the change in condition can assist workers with investigating the timing and / or locations of damage that occurred to instruments.

[0113] In some examples, the condition evaluator 224 can provide the condition evaluation 225 to an inventory management database. The inventory management database is hosted by the computing device 110 or another computing device. In some examples, the inventory’ management database is cloud-based.

[0114] The conditions of the instruments can be tracked over time in the inventory' management database. For example, the sharpness of an instrument can be tracked over time. In some examples, the computing device 110 can use the tracked conditions of an instrument to predict when the instrument will need to be repaired or replaced. For example, the sharpness of a scalpel may decrease from a sharpness rating of 8 to a sharpness rating of 7 after being washed in a washing machine. The sharpness threshold may be 5 or greater. Therefore, the computing device 1 10 can predicted a number of cleaning cycles after which the scalpel will need to be sharpened. The computing device 110 can output a notification to the display 140 indicating the predicted number of cleaning cycles that can be performed prior to the scalpel needing to be sharpened.

[0115] The counter 226 determines a quantity of the instruments depicted in the image 130 generated by the camera 102. In some examples, the counter 226 determines a quantity of each of multiple different ty pes of instruments.

[0116] In some examples, the counter 226 compares a detected quantity of instruments to a previously detected quantity of instruments in order to track the instruments over time. For example, the counter 226 may determine a quantity of instruments detected for a set of instruments during a pre-washing survey of the set of instruments and a quantity of instruments detected during a post-washing survey of the same set of instruments. The counter 226 can determine whether the quantity of the instrument has changed, for example, due to one or more instruments being lost during the washing process.

[0117] In some examples, the counter 226 can determine that the quantity of instruments has decreased between surveys. The counter 226 can generate an inventory' update 116 indicating the decrease in quantity of instruments. In some examples, the counter 226 generates a notification indicating that the quantity of one or more types of instruments has decreased. A decrease in the number of instruments can result when an instrument is lostduring a time between surveys. The inventory update 116 indicating the decreased quantity of instruments can assist workers with investigating the timing when instruments are lost and / or locations of lost instruments.

[0118] In some examples, the counter 226 can determine that the quantity of instruments has increased between survey s. The counter 226 can generate an inventory update 116 indicating the increase in quantity of instruments. In some examples, the counter 226 generates a notification indicating that the quantity of instruments has increased. An increase in the number of instruments can result when an instrument is gained during a time between surveys. For example, instruments from different batches or sets of instruments can get mixed together during the washing or sterilization process or when transported from the medical facility (e.g., operating room) to the sterilization facility, resulting in some batches increasing in quantity of instruments and other batches decreasing in quantity of instruments. The inventory update 116 indicating the increased quantity of instruments can assist workers with investigating the timing and / or locations of lost or misplaced instruments.

[0119] Example operations of the system 100 include performing automated medical instrument usage and count through feeding and processing of the video data streams from cameras to machine vision background subtraction, color filtering, and connected components labeling algorithms.

[0120] Example operations of the system 100 include classifying medical instruments through feeding and processing of images from camera 102 to a trained Al machine vision (e.g., via a modified YOLO model architecture using transfer learning and deep convolutional neural networks) capable of locating and identify ing instruments within the video stream.

[0121] Referring to FIG. 3. a screen of user interface 300 can display identification and classification of various instruments, such as scalpel 101, clamp 131, and scissors 111, overlaid on video data streams from camera 102. Examples of instruments that the system can identity' include, without limitation, scissors (e.g., Mayo scissors, Metzenbaum scissors, Potts scissors, and iris scissors), pickups / forceps (e.g., tissue forceps, Adson forceps. Ferris- Smith forceps, Bonney forceps, DeBakey forceps. Kocher forceps, right angle forceps, and Russian forceps), clamps / hemostats (e.g., Crile hemostatic clamps, Kelly clamps, and Allis- Babcock clamps), retractors (e.g., rake retractors, Volkman sharp retractors, Richardson retractors, flat retractors, malleable retractors, pickle fork retractors, Army -Navy retractors, Deaver retractors, Z-retractors, Hohmann retractors, cobra retractors, scissor-esque retractors, Gelpi retractors, Weitlaner retractors, and Bookwaiter retractors), rongeurs (e.g., Adsonrongeurs, and double action rongeurs), spreaders (e.g., Lamina spreaders), mallets (e.g., ortho heavy mallets), saws (e.g., bone saws), power equipment (e.g., power drills, reamers, and pindrivers), pliers, bone hooks, metal rulers, chisels, osteotomes (e.g., Lambotte osteotomes), and laparoscopic instruments. As previously described, the image labeler 222 can color code instruments by ty pe, for example, scalpels 101 with pink boxes, clamps 131 with orange boxes, and scissors 111 with green boxes, and these color-coded bounding boxes 312, 314, 316 can be displayed on the user interface 300.

[0122] FIGS. 4A and 4B depict screenshots of an exemplary graphical user interfaces output from a medical instrument classification and tracking system showing instrument tracking tables.

[0123] Referring to FIG. 4 A. a user interface 400 can display a table 402 representing an inventor}’ or part of an inventory' of medical instruments generated based on processing the image(s) 130 generated by the camera 102, as described herein. The table 402 can include a list of instruments, or tools 404. In some examples, the table 402 is generated based on data stored in an inventory management database. When inventory updates 116 are generated by the computing device 110, as described herein, the inventory updates 116 can populate corresponding fields of the table 402.

[0124] In the example table 402, the tools 404 include scissor 412, clamp 414, knife 41 , and forcep 418. In some examples, the list of tools include various different sub-categories of tools, brand names of tools, sizes of tools, or any combination thereof.

[0125] The table 402 can include a list of stages 406. In the example table 402, the stages 406 include "Pre-Wash." “Post-Wash,” “To OR [Operating Room],” and “From OR.” Other possible stages include “pre-sterilization” and “post-sterilization.”

[0126] The table 402 can indicate, for each tool at each stage, whether the tool was present at the stage. Presence of a tool, as determined by the instrument detector 210 and the classifier 214, can be represented by7a checkmark, with absence of the tool being indicated by an “X.” The table 402 can indicate, for each tool at each stage, whether the condition of the tool passed (“Pass”) or failed (“Fail”) at each stage, as determined by the condition evaluator 224.

[0127] As shown in the table 402, the scissor 412 is present at both the Pre-Wash and Post-Wash stages, but the condition of the scissor 412 fails at the Post-Wash stage. When the computing device 110 determines that the condition of the scissor 412 fails, the computing device 110 can update the table 402 to indicate the failure by inserting the word “Fail” into the corresponding table field. In some examples, the computing device 110 generates an alertor notification indicating that the condition of the scissor 412 has failed. In the example of FIG. 4. the scissor 412 was removed from the set of instruments after the Post-Wash stage and did not proceed to the following stages, as indicated by the shaded table fields at the To OR and From OR stages.

[0128] As shown in the table 402, the clamp 414 is present at the Pre-Wash, Post-Wash, and To OR stages. The claim 414 is missing at the From OR stage. Therefore, the table 402 includes an “X” in the table field corresponding to the claim 414 and the From OR stage. The table 402 thus indicates that the clamp 414 went missing from the set of instruments after the clamp 414 was sent to the OR. This information can assist workers with locating the missing clamp 414 and / or identifying a party that is responsible for losing the clamp 414.

[0129] As shown in the table 402, the knife 416 is present at all stages, and the condition of the knife 416 passes at all stages.

[0130] As shown in the table 402, the forcep 418 is present at all stages. The condition of the forcep 418 fails at the From OR stage, as indicated by the word “Fail'’ in the corresponding table field.

[0131] Referring to FIG. 4B, a user interface 450 can display a table 422 representing an inventory or part of an inventory of medical instruments generated based on processing the image(s) 130 generated by the camera 102, as described herein. Similar to the table 402, the table 422 includes a list of instruments, or tools 404. In some examples, the table 422 is generated based on data stored in an inventory management database. When inventory updates 1 16 are generated by the computing device 1 10, as described herein, the inventory updates 116 can populate corresponding fields of the table 422.

[0132] Similar to the table 402. in the example table 422, the tools 404 include scissor 412, clamp 414, knife 416. and forcep 418. The stages 406 include “Pre-Wash.” “PostWash,” “To OR,” and “From OR.”

[0133] The table 422 can indicate, for each tool at each stage, a quantify of the type of tool that is present at the stage. For example, as show n in the table 422, the quantify of scissors 412 changes from 8 to 7 between the Post-Wash stage and the To OR stage. Therefore, based on the table 422. it can be determined that one scissor 412 was lost or misplaced between the Post-Wash stage and the To OR stage. In contrast, the quantify of clamps 414 remains constant across all stages 406. Therefore, based on the table 422, no clamps 414 were lost during the stages.

[0134] In some examples, the quantities of the tools indicated in the table 422 for each stage represent quantities of tools in a single image, such as the image 130, corresponding tothe survey for the respective stage. In some examples, the quantities of the tools indicated in the table 422 represent quantities of tools in multiple images. For example, a set of instruments can be surveyed over a time duration of several minutes, during which multiple still images and / or video images can be captured. The quantities indicated in the table 422 can represent the quantities of the different ty pes of instruments that are included in the set of instruments.

[0135] FIG. 5 depicts a flowchart of an example process 500 for medical instrument classification and tracking. Process 500 can be executed by one or more computing systems including, e.g., the system 100 described above. In some examples, all steps of the process 500 are performed by a same component of a system, such as the computing device 110. In some examples, some steps of the process 500 are performed by one component, and other steps of the process 500 can be performed by another component. For example, some steps of the process 500 can be performed by the computing device 110, while other steps can be performed by the camera 102.

[0136] The process 500 includes obtaining in image (502). For example, the computing device 110 can receive one or more images (e.g., image 130) generated by an imaging device such as the camera 102. The images can depict a workstation at a medical facility', and may depict one or more medical instruments positioned on the workstation.

[0137] The process 500 includes detecting a medical instrument depicted in the image (504). For example, the system can employ object detection algorithms to locate individual medical instruments depicted within still images and / or a video feed. In some examples, the computing device 110 provides the image 130 as input to a medical instrument detection model such as the instrument detector 210. The computing device 110 obtains, as output from the medical instrument detection model, data indicating detection of a depiction of one or more medical instruments in the image 130 (e.g., instrument detection 212).

[0138] The process 500 includes classifying the medical instruments detected in the image (506). In some examples, in response to obtaining the data indicating the detection of the depiction of the medical instrument in the image, the computing device 110 provides the image 130 as input to a medical instrument classification model such as the classifier 214 of FIG. 2. The computing device 110 obtains, as output from the medical instrument classification model, a classification (e.g., classification 216) of the medical instruments as a particular ty pe of medical instrument.

[0139] In some examples, the system can identify a type of each detected instrument in the image 139. For example, the system can employ a YOLO machine learning model to analyzea region of pixels within one or more images or within one or more frames of the video feeds to identify instrument category- or type. The region of pixels can include pixels representing a workstation, such as the workstation 104.

[0140] The process 500 includes annotating the image with the classification (508). For example, the image labeler 222 of the computing device 110 can annotate the image 130 with a label indicating the particular type of medical instrument depicted in the image 130. In some examples, the image labeler 222 receives information related to a condition of the medical instrument 512 (e.g., condition information 225)., and annotates the image 130 with a label indicating the condition of the medical instrument 512. The image labeler 222 outputs the annotated image 112. The computing device 110 can output the annotated image 130 for presentation by an electronic display such as the display 140.

[0141] The process 500 includes evaluating a condition of the medical instrument (512). In some examples, the computing device 110 provides the image 130 as input to a medical instrument evaluation model such as the condition evaluator 224 of FIG. 2. The computing device 110 obtains, as output from the medical instrument evaluation model, data indicating a condition of the medical instrument. The condition of the medical instrument can include a cleanliness rating of the medical instrument, a damage rating of the medical instrument, a sharpness rating of the medical instrument, or any combination thereof.

[0142] The process 500 can include determining that the instrument condition 514 is unsatisfactory (515). The condition evaluator 224 can determine that the instrument condition 514 is unsatisfactory in response to determining that the condition does not satisfy condition criteria. For example, the condition evaluator 224 can determine that the cleanliness rating of the medical instrument does not satisfy cleanliness criteria, that the damage rating of the medical instrument does not satisfy damage criteria, that the sharpness of the medical instrument does not satisfy sharpness criteria, or any combination of these determinations.

[0143] The process 500 includes, in response to determining that the instrument condition is unsatisfactory- (515), generating a notification indicating the condition of the medical instrument (520). For example, the condition evaluator 224 can generate a notification 114 indicating that the cleanliness rating of the medical instrument does not satisfy cleanliness criteria, that the damage rating of the medical instrument does not satisfy damage criteria, that the sharpness of the medical instrument does not satisfy sharpness criteria, or any combination thereof.

[0144] The process 500 can include determining that an instrument condition 514 is satisfactory (516). The condition evaluator 224 can determine that the instrument condition issatisfactory in response to determining that the condition satisfies condition criteria. For example, the condition evaluator 224 can determine that the cleanliness rating of the medical instrument satisfies cleanliness criteria, that the damage rating of the medical instrument satisfies damage criteria, that the sharpness of the medical instrument satisfies sharpness criteria, or any combination thereof. In response to determining that the instrument condition 514 is satisfactory (516). the system does not generate a notification related to the condition of the medical instrument (518). For example, the condition evaluator can determine that the condition satisfies criteria for the instrument condition, and in response, the system determines to not generate a notification related to the condition of the instrument.

[0145] The process 500 includes updating a status of the medical instrument in an inventory (510). In some examples, the counter 226 outputs an inventory update 116 including data indicating a quantity of medical instruments counted within the image 130 by the counter 226. The inventory update 116 can be entered into an inventory’ management database.

[0146] In some examples, the inventory update 116 indicates a quantity of each of multiple types of medical instruments depicted in the image 130 and automatically detected by the counter 226. In some examples, inventory update 116 includes a condition of the medical instruments depicted in the image 130, as determined by the condition evaluator 224. For example, the inventory update 116 can indicate that two knives were counted in the image 130. and that one out of the two knives does not satisfy cleanliness criteria.

[0147] FIG. 6 is a schematic diagram of a computer system 600. The system 600 can be used to cany’ out the operations described in association with any of the computer- implemented methods described previously, according to some implementations. In some implementations, computing systems and devices and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification (e.g., system 600) and their structural equivalents, or in combinations of one or more of them. The system 600 is intended to include various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers, including vehicles installed on base units or pod units of modular vehicles. The system 600 can also include mobile devices, such as personal digital assistants, cellular telephones, smartphones, and other similar computing devices. Additionally, the system can include portable storage media, such as, Universal Serial Bus (USB) flash drives. For example, the USB flash drives may store operatingsystems and other applications. The USB flash drives can include input / output components, such as a wireless transducer or USB connector that may be inserted into a USB port of another computing device.

[0148] The system 600 includes a processor 610, a memory 620, a storage device 630, and an input / output device 640. Each of the components 610, 620, 630, and 640 are interconnected using a system bus 650. The processor 610 is capable of processing instructions for execution within the system 600. The processor may be designed using any of a number of architectures. For example, the processor 610 may be a CISC (ComplexInstruction Set Computers) processor, a RISC (Reduced Instruction Set Computer) processor, or a MISC (Minimal Instruction Set Computer) processor.

[0149] In one implementation, the processor 610 is a single-threaded processor. In another implementation, the processor 610 is a multi -threaded processor. The processor 610 is capable of processing instructions stored in the memory 620 or on the storage device 630 to display graphical information for a user interface on the input / output device 640.

[0150] The memory 620 stores information within the system 600. In one implementation, the memory 620 is a computer-readable medium. In one implementation, the memory 620 is a volatile memory unit. In another implementation, the memory 620 is a non-volatile memory unit.

[0151] The storage device 630 is capable of providing mass storage for the system 600. In one implementation, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device.

[0152] The input / output device 640 provides input / output operations for the system 600. In one implementation, the input / output device 640 includes a keyboard and / or pointing device. In another implementation, the input / output device 640 includes a display unit for displaying graphical user interfaces.

[0153] The features described can be implemented in digital electronic circuitry , or in computer hardware, firmware, software, or in combinations of them. The apparatus can be implemented in a computer program product tangibly embodied in an information carrier, in a machine-readable storage device for execution by a programmable processor; and method steps can be performed by a programmable processor executing a program of instructions to perform functions of the described implementations by operating on input data and generating output. The described features can be implemented advantageously in one or more computer programs that are executable on a programmable system including at least oneprogrammable processor coupled to receive data and instructions from, and to transmit data and instructions to. a data storage system, at least one input device, and at least one output device. A computer program is a set of instructions that can be used, directly or indirectly, in a computer to perform a certain activity or bring about a certain result. A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0154] Suitable processors for the execution of a program of instructions include, by way of example, both general and special purpose microprocessors, and the sole processor or one of multiple processors of any kind of computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Generally, a computer will also include, or be operatively coupled to communicate with, one or more mass storage devices for storing data files; such devices include magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magnetooptical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, ASICs (application-specific integrated circuits).

[0155] To provide for interaction with a user, the features can be implemented on a computer having a display device such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user and a keyboard and a pointing device such as a mouse or a trackball by which the user can provide input to the computer. Additionally, such activities can be implemented via touchscreen flat-panel displays and other appropriate mechanisms.

[0156] The features can be implemented in a computer system that includes a back-end component, such as a data server, or that includes a middleware component, such as an application server or an Internet server, or that includes a front-end component, such as a client computer having a graphical user interface or an Internet browser, or any combination of them. The components of the system can be connected by any form or medium of digital data communication such as a communication network. Examples of communicationnetworks include a local area network (“LAN”), a wide area network (“WAN”), peer-to-peer networks (having ad-hoc or static members), grid computing infrastructures, and the Internet.

[0157] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a network, such as the described one. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0158] Embodiments of the present disclosure are directed at capturing effects having potential causal links to adverse patient outcomes through the use of a combination of many sensory device signals augmented by artificial intelligence algorithms (e.g., modified YOLO models using transfer learning, 1 -dimensional and 2-dimensional convolutional neural networks, multilayer perceptrons, decision trees, and random forest search) and non-AI models (e.g., combination of machine vision image treatment methods, Fast Fourier Transform, cascade classifier, mahalanobis distance, connected components labeling algorithm) to detect, analyze and recommend corrective actions as events take place in realtime or near real-time.

[0159] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular implementations of particular inventions. Certain features that are described in this specification in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0160] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components andsystems can generally be integrated together in a single software product or packaged into multiple software products.

[0161] Thus, particular implementations of the subject matter have been described. Other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

[0162] While the present disclosure is described in the context of a psychological diagnostic system, it is understood that the techniques and processes described herein are applicable outside of this context. For example, the techniques and processes described herein may be applicable to other types of diagnostic machine learning systems including, but not limited to, medical diagnostic systems, computer software diagnostic (debugging) systems, computer hardware diagnostic systems, or quality assurance (e.g., in manufacturing) diagnostic systems.

Claims

CLAIMSWhat is claimed is:

1. A computer-implemented method comprising: receiving an image generated by an imaging device; providing the image as input to a medical instrument detection model; detecting, by the medical instrument detection model, a medical instrument in the image; in response to detecting the medical instrument in the image, providing the image as input to a medical instrument classification model; determining, by the medical instrument classification model, a classification of the medical instrument as a particular type of medical instrument; annotating the image with a label indicating the particular type of medical instrument to generate an annotated image; and providing the annotated image for presentation by an electronic display.

2. The method of claim 1, comprising: providing the image as input to a medical instrument evaluation model; and determining, by the medical instrument evaluation model, a condition of the medical instrument.

3. The method of claim 2, wherein the condition of the medical instrument comprises a cleanliness rating of the medical instrument.

4. The method of claim 3, comprising: determining that the cleanliness rating of the medical instrument does not satisfy a cleanliness threshold; and in response to determining that the cleanliness rating of the medical instrument does not satisfy cleanliness threshold, generating a notification indicating that the cleanliness rating of the medical instrument does not satisfy' cleanliness threshold.

5. The method of claim 3, comprising: determining that the cleanliness of the medical instrument does not satisfy a cleanliness threshold; andin response to determining that the cleanliness rating of the medical instrument does not satisfy the cleanliness threshold, updating a status of the medical instrument in an inventor}’ management database indicating that the cleanliness rating of the medical instrument does not satisfy the cleanliness threshold.

6. The method of claim 2, wherein the condition of the medical instrument comprises a damage rating of the medical instrument.

7. The method of claim 6, comprising: determining that the damage rating of the medical instrument does not satisfy a damage threshold; and in response to determining that the damage rating of the medical instrument does not satisfy the damage threshold, generating a notification indicating that the damage rating of the medical instrument does not satisfy the damage threshold.

8. The method of claim 6, comprising: determining that the damage rating of the medical instrument does not satisfy a damage threshold; and in response to determining that the damage rating of the medical instrument does not satisfy the damage threshold, updating a status of the medical instrument in an inventory management database indicating that the damage rating of the medical instrument does not satisfy the damage threshold.

9. The method of claim 2, wherein the medical instrument includes one or more sharp edges, and the condition of the medical instrument comprises a sharpness rating of the one or more sharp edges.

10. The method of claim 9, comprising: determining that the sharpness rating of the medical instrument does not satisfy a sharpness threshold; and in response to determining that the sharpness rating of the medical instrument does not satisfy the sharpness threshold, generating a notification indicating that the sharpness rating of the medical instrument does not satisfy the sharpness threshold.

11. The method of claim 9, comprising: determining that the sharpness rating of the medical instrument does not satisfy a sharpness threshold; and in response to determining that the sharpness rating of the medical instrument does not satisfy7the sharpness threshold, updating a status of the medical instrument in an inventory management database indicating that the sharpness rating of the medical instrument does not satisfy7the sharpness threshold.

12. The method of claim 1, comprising: in response to determining the classification of the medical instrument as the particular type of medical instrument, updating a quantity of the particular type of medical instrument in an inventory management database.

13. The method of claim 1, wherein the particular type of object comprises a scalpel, a blade, a knife, a scissor, a forcep, a clamp, a retractor, or a needle.

14. The method of claim 1, wherein: the medical instrument detection model comprises a first machine learning model, and detecting, by the medical instrument detection model, the medical instrument in the image comprises applying the first machine learning model to the image.

15. The method of claim 14, wherein: the medical instrument classification model comprises a second machine learning model, and determining, by the medical instrument classification model, the classification of the medical instrument as the particular type of medical instrument comprises applying the second machine learning model to the image.

16. The method of claim 1, wherein the medical instrument classification model comprises a machine learning model that is trained to determine classifications of medical instruments based on shapes of the medical instruments.

17. The method of claim 1, wherein the medical instrument classification model comprises a machine learning model that is trained to determine classifications of medicalinstruments based on visual identifiers marked on the medical instruments.

18. The method of claim 17, wherein the visual identifiers include one or more of quick response codes, bar codes, typographical labels, numeric identifiers, and hashmarks.

19. The method of claim 1, comprising: receiving sensor data from a sensor that is configured to detect non-visible energy; and providing the sensor data as input to the medical instrument detection model; wherein the medical instrument classification model comprises a machine learning model that is trained to determine classifications of medical instruments based on non-visible energy emitted by the medical instruments and shapes of the medical instruments.

20. The method of claim 1, wherein the medical instrument classification model comprises a machine learning model that is trained to determine classifications of medical instruments based on (a) shapes of the medical instruments and (b) visual identifiers marked on the medical instruments.

21. One or more non-transitory computer readable storage media storing instructions that, when executed by at least one processor, cause the at least one processor to perform the method of any one of the preceding claims.

22. A medical instrument tracking system comprising: an imaging device positioned to capture images of a workstation within a medical instrument cleaning facility; one or more processors in electronic communication with the imaging device; and one or more tangible, non-transitory media operably connectable to the one or more processors and storing instructions that, when executed, cause the one or more processors to perform the method of any one of claims 1-20.

Images