Equipment tracking machine

Abstract

The machine accesses a first image captured before a procedure is initiated, wherein the first image depicts a set of instruments, and a second image captured after the procedure is initiated, wherein the second image depicts a proper subset of the set of instruments depicted in the first image. Based on the first image and the second image, the machine can determine that an instrument from the set of instruments depicted in the first image is not depicted in the proper subset of the set of instruments in the second image, and then cause presentation of a notification indicating that the instrument not depicted in the second image is missing. Alternatively or additionally, the machine can determine whether an instrument from the set of instruments is used in the procedure, and then cause presentation of a notification indicating whether the instrument is used in the procedure.

Description

[0001] Related application data

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 968,538, filed January 31, 2020, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The subject matter disclosed herein relates generally to the technical field of special-purpose machines that facilitate monitoring instruments (e.g., surgical instruments or other tools), including computerized variations of the software configurations of such special-purpose machines and improvements to such variations, as well as to techniques for improving such special-purpose machines as compared to other special-purpose machines that facilitate monitoring instruments. Background Art

[0004] A collection of instruments (e.g., a collection of surgical tools) can be arranged on a transport (e.g., a tray or cart) and taken to a performer (e.g., a surgeon) of a procedure (e.g., a medical procedure, such as a surgical procedure) to be performed (e.g., on a patient). Not all instruments are used during a procedure (e.g., 30%-80% of surgical instruments are not used), and it is helpful to track which instruments were used to ensure the time, effort, and cost of sterilization and which instruments were not used. It would be beneficial for all instruments to be present and accounted for after the procedure, regardless of whether they were used or not used during the procedure. For example, tracking surgical instruments during a medical procedure can limit or reduce the risk of such instruments being inadvertently left in a patient's body. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Some embodiments are shown by way of example and not limitation in the accompanying drawings.

[0006] Figure 1 is a network diagram illustrating a network environment suitable for operating an implement tracking machine according to some example embodiments.

[0007] Figure 2 is a block diagram illustrating components of a device suitable for use as an instrument tracking machine, according to some example embodiments.

[0008] Figure 3 and 4 is a flowchart illustrating the operation of a device in performing a method of tracking an instrument according to some example embodiments.

[0009] Figure 5 and 6 is a flowchart illustrating the operation of a device in performing another method of tracking an instrument, according to some example embodiments.

[0010] Figure 7is a screenshot illustrating an image depicting an instrument, wherein a device configured by an app has added a bounding box indicating the instrument, in accordance with some example embodiments.

[0011] Figure 8-10 is a screenshot illustrating images depicting instruments and in which the device configured by the app has added a count of instruments for each image, both individually and by instrument type, in accordance with some example embodiments.

[0012] Figure 11 is a block diagram illustrating components of a machine capable of reading instructions from a machine-readable medium and performing any one or more of the methodologies discussed herein, according to some example embodiments. DETAILED DESCRIPTION

[0013] Example methods (e.g., algorithms) facilitate detection, classification, identification, and tracking of instruments or other monitoring devices, and example systems (e.g., dedicated machines configured with specialized software) are configured to facilitate detection, classification, identification, and tracking of instruments or other monitoring devices. The examples represent only possible variations. Unless otherwise expressly stated, structures (e.g., structural components, such as modules) are optional and may be combined or subdivided, and operations (e.g., in processes, algorithms, or other functions) may be sequentially varied or combined or subdivided. In the following description, for purposes of explanation, many specific details are set forth to provide a thorough understanding of various example embodiments. However, it will be apparent to those skilled in the art that this subject matter may be practiced without these specific details.

[0014] Precise and accurate detection, classification, and identification of instruments can be valuable goals in providing cost-effective management of instrument inventory, improving health and safety (e.g., for patients undergoing medical procedures), or both. Instrument usage information can help hospitals manage updates, for example, that instrument trays contain only surgical instruments that are likely to be used (e.g., for a specific procedure, by a specific surgeon, or both). To this end, a machine (e.g., a device configured with appropriate software (such as a suitable application)) is configured to function as an instrument tracking machine and perform instrument detection, instrument classification, instrument identification, installation tracking, or any suitable combination, for one or more instruments based on images captured before and after a procedure (e.g., a medical procedure) is initiated. As used herein, "instrument detection" means detecting an instrument of unspecified type and unspecified identity depicted at a location within an image; "instrument classification" means identifying, distinguishing, or otherwise obtaining the type of a detected instrument; and "instrument identification" means identifying, distinguishing, or otherwise obtaining the identity of a particular individual instrument, particularly in contrast to the identities of other similar instruments.

[0015] Configured in accordance with one or more of the example systems and methods discussed herein, a machine can function as an instrument classifier configured to determine the type of each instrument depicted in an image (e.g., scissors or forceps) (e.g., for counting instances of each type of instrument), configured to identify an object identifier for a particular individual object (e.g., in conjunction with detection, classification, or both) (such as a particular instrument (e.g., the same scissors previously depicted in a previous image or the same forceps previously depicted in a previous image), or both. For surgical instruments, examples of instrument types include graspers (e.g., forceps), clamps (e.g., occluders), needle drivers (e.g., needle holders), retractors, retractors, cutters, scopes, suction tips, sealing devices, endoscopes, probes, and calipers.

[0016] Whether implemented as a portable (e.g., mobile) handheld device (e.g., a smartphone configured by an app), a portable cart-mounted or backpack-mounted device, a fixed machine (e.g., built into a hospital operating room, such as built into a wall or ceiling), or any suitable combination thereof, the machine can thus distinguish between different types of instruments, different individual instances of an instrument, or both. In an example scenario involving a large number of surgical instruments, the machine (e.g., used as an instrument classifier) ​​can act as a recognition tool to quickly find the corresponding type of several instruments by scanning them in real time.

[0017] In an example scenario where inventory management is important, machines can provide instrument tracking capabilities. For example, many hospital operating rooms often face the challenge of preventing any surgical instruments from remaining in the patient's body after a surgical procedure, which is unfortunately a common problem for hospitals. To address this challenge and avoid such incidents, machines (e.g., acting as object identifiers) can be deployed to identify and count surgical instruments individually or by type before and after a procedure is initiated (e.g., before initiating a procedure and after completing a procedure) to determine whether all instruments present at the beginning of the procedure have been accounted for before closing the patient.

[0018] According to some example embodiments of the systems and methods discussed herein, a suitably configured machine accesses a first image captured before a procedure is initiated, wherein the first image depicts a set of instruments that may be used in the procedure. The machine also accesses a second image captured after the procedure is initiated (e.g., midway through the procedure, just before the procedure is completed, or after the procedure is completed), wherein the second image depicts a proper subset of the set of instruments depicted in the first image. From these images, the machine determines that an instrument in the set of instruments depicted in the first image is not depicted in a proper subset of the set of instruments in the second image. The machine then causes presentation of a notification indicating that the instrument depicted in the first image but not in the second image is missing from the set of instruments.

[0019] According to certain example embodiments of the systems and methods discussed herein, a suitably configured machine accesses a first image captured before a procedure is initiated, and the first image depicts a set of instruments that may be used in the procedure. The machine also accesses a second image captured after the procedure is initiated (e.g., midway through the procedure, just before the procedure is completed, or after the procedure is completed), and the second image depicts a proper subset of the set of instruments depicted in the first image. From these images, the machine determines, based on the first and second images, which instruments in the set of instruments depicted in the first image were used or not used during the procedure (e.g., as part of performing the procedure). The machine then causes the presentation of a notification indicating that the instruments were used or not used during the procedure.

[0020] Figure 1 is a network diagram illustrating a network environment 100 suitable for operating an instrument tracking machine in accordance with some example embodiments. The network environment 100 includes a database 115 and devices 130 and 150 (e.g., as examples of instrument tracking machines), all of which are communicatively coupled to each other via a network 190. The database 115 may form all or a portion of a cloud 118 (e.g., a geographically distributed collection of multiple machines configured to function as a single server), which may form all or a portion of a network-based system 105 (e.g., a cloud-based server system configured to provide one or more network-based services to the devices 130 and 150). The database 115 and the devices 130 and 150 may be implemented in whole or in part in a dedicated (e.g., specialized) computer system, as described below with respect to FIG. Figure 11 described.

[0021] Figure 110. Also shown are users 132 and 152. One or both of users 132 and 152 may be a human user (e.g., a person, such as a nurse or surgeon), a machine user (e.g., a computer configured by a software program to interact with device 130 or 150), or any suitable combination thereof (e.g., a person assisted by a machine or a machine supervised by a person). User 132 is associated with device 130 and may be a user of device 130. For example, device 130 may be a desktop computer, a car computer, a home media system (e.g., a home theater system or other home entertainment system), a tablet computer, a navigation device, a portable media device, a smartphone, or a wearable device (e.g., a smartwatch, smart glasses, smart clothing, or smart jewelry) belonging to user 132. Similarly, user 152 is associated with device 10 and may be a user of device 150. As examples, device 150 can be a desktop computer, a car computer, a home media system (e.g., a home theater system or other home entertainment system), a tablet computer, a navigation device, a portable media device, a smart phone, or a wearable device (e.g., a smart watch, smart glasses, smart clothing, or smart jewelry) belonging to user 152.

[0022] Figure 1 Any system or machine (e.g., databases and devices) shown in the text may be, include, or otherwise be implemented as a special purpose (e.g., specialized or other non-conventional and non-general purpose) computer that has been modified to perform one or more of the functions described herein for the system or machine (e.g., configured or programmed by special-purpose software (e.g., one or more software modules of a special-purpose application, an operating system, firmware, middleware, or other software program)). For example, the following description of Figure 11 Special-purpose computer systems capable of implementing any one or more of the methods described herein are discussed, and thus such special-purpose computers can be means for performing any one or more of the methods discussed herein. Within the technical field of such special-purpose computers, a special-purpose computer that has been specifically modified (e.g., configured by specialized software) with the structure discussed herein to perform the functions discussed herein is a technical improvement over other special-purpose computers that lack the structure discussed herein or are unable to perform the functions discussed herein. Thus, a special-purpose machine configured in accordance with the systems and methods discussed herein provides a technical improvement over similar special-purpose machines.

[0023] As used herein, a "database" is a data storage resource and may store data structured in any of a variety of ways, for example, as a text file, a table, a spreadsheet, a relational database (e.g., an object-relational database), a triple store, a hierarchical data store, a document database, a graph database, key-value pairs, or any suitable combination thereof. Furthermore, Figure 1Any two or more of the systems or machines shown in the accompanying drawings may be combined into a single system or machine, and the functionality described herein for any single system or machine may be divided among the multiple systems or machines.

[0024] Network 190 can be any network that enables communication between systems, machines, databases, and devices (e.g., between machine 110 and device 130). Thus, network 190 can be a wired network, a wireless network (e.g., a mobile or cellular network), or any suitable combination thereof. Network 190 can include one or more components that constitute a private network, a public network (e.g., the Internet), or any suitable combination thereof. Thus, network 190 can include one or more components that incorporate a local area network (LAN), a wide area network (WAN), the Internet, a mobile phone network (e.g., a cellular network), a wired telephone network (e.g., a plain old telephone service (POTS) network), a wireless data network (e.g., a WiFi network or a WiMax network), or any suitable combination thereof. Any one or more components of network 190 can transmit information via a transmission medium. As used herein, "transmission medium" refers to any intangible (e.g., transient) medium capable of transmitting (e.g., transporting) instructions for execution by a machine (e.g., by one or more processors of such a machine), and includes digital or analog communication signals or other intangible media to facilitate communication of such software.

[0025] Figure 2 is a block diagram illustrating components of device 130 configured to function as an instrument tracking machine in accordance with some example embodiments. Device 130 is shown to include an image accessor 210, an instrument identifier 220, a notifier 230, a camera 240, and a depth sensor 250, all of which are configured to communicate with each other (e.g., via a bus, shared memory, or switch). Image accessor 210 may be or include an access module or similar suitable software code for accessing one or more images. Instrument identifier 220 may be or include a recognition module or similar suitable software code for recognizing instruments (e.g., by type or as specific individual instances). Notifier 230 may be or include a notification module or similar suitable software code for generating a notification and causing it to be presented (e.g., on a display screen of device 130, via an audio speaker of device 130, or both).

[0026] Camera 240 may be or include an image capture component configured to capture one or more images (e.g., digital photographs), and the captured images may include or may visualize optical data (e.g., RGB data or optical data in another color space), infrared data, ultraviolet data, ultrasound data, radar data, or any suitable combination thereof. According to various example embodiments, camera 240 may be on the back of a handset, on the front of a mounted device including a display screen, a collection of one or more cameras mounted and aimed at a surgical tray, at a scrub technician's table (e.g., in an operating room), or at an assembly workstation (e.g., in an instrument vendor's assembly room), or any suitable combination thereof. The collection of cameras or devices 130 may be configured by app 200 to fuse data from multiple cameras that image the tray, surface, or / and table (e.g., stereoscopically). In some cases, device 130 is configured by app 200 to access multiple images captured during motion before processing (e.g., via image stitching), apply structure-from-motion algorithms, or both to support or enhance instrument detection, instrument classification, instrument identification, or any suitable combination thereof, as discussed elsewhere herein.

[0027] The depth sensor 250 may be or include an infrared sensor, a radar sensor, an ultrasonic sensor, an optical sensor, a time-of-flight camera, a structured light scanner, or any suitable combination thereof. Thus, the depth sensor 250 may be configured to generate depth data (e.g., representing a distance to) one or more objects (e.g., instruments) within range of the depth sensor 250 (e.g., within a field of view or within a detection field).

[0028] like Figure 2 As shown in FIG, the image accessor 210, the machine identifier 220, the notifier 230, or any suitable combination thereof may form all or part of an app 200 (e.g., a mobile app) stored (e.g., installed) and executable on the device 130 (e.g., in response to or otherwise as a result of data received from the device 130 via the network 190). Additionally, one or more processors 299 (e.g., a hardware processor, a digital processor, or any suitable combination thereof) may be included (e.g., temporarily or permanently) in the app 200, the tire image accessor 210, the machine identifier 220, the notifier 230, or any suitable combination thereof.

[0029] Any one or more of the components (e.g., modules) described herein can be implemented using hardware (e.g., one or more of the processors 299) or a combination of hardware and software. For example, any component described herein can physically include an arrangement of one or more of the processors 299 (e.g., a subset of the processors 299 or a subset of the processors 299) that is configured to perform the operations described herein for that component. As another example, any component described herein can include software, hardware, or both that configures the arrangement of one or more of the processors 299 to perform the operations described herein for that component. Thus, the different components described herein can include and configure different arrangements of the processors 299 at different points in time or a single arrangement of the processors 299 at different points in time. Each component (e.g., module) described herein is an example of a means for performing the operations described herein for that component. Moreover, any two or more components described herein can be combined into a single component, and the functions described herein for a single component can be subdivided among multiple components. In addition, according to various example embodiments, components described herein as being implemented within a single system or machine (e.g., a single device) can be distributed across multiple systems or machines (e.g., multiple devices).

[0030] Figure 3 and 4 is a flowchart illustrating the operation of the device 130 in performing the method 300 of tracking an instrument according to some example embodiments. The operations in the method 300 may be performed by the device 130 using the method described above with respect to Figure 2 The components (e.g., modules) described herein may utilize one or more processors (e.g., microprocessors or other hardware processors), or any suitable combination thereof. Figure 3 As shown in , method 300 includes operations 310 , 320 , 330 , and 340 .

[0031] In operation 310, image accessor 210 accesses (e.g., receives, retrieves, reads, or otherwise obtains) a first image captured prior to the initiation of a procedure. The first image depicts a collection of instruments that may be used when performing the procedure (e.g., a reference collection of instruments). For example, the first image may be captured by camera 240 of device 130 (e.g., by taking a digital photograph of a surgical tray in which a collection of surgical instruments has been arranged in preparation for a surgical procedure to be performed by a surgeon). In some example embodiments, the first image is accessed by image accessor 210 from database 115 via network 190. One or more fiducial markers may also be depicted in the first image, and such fiducial markers may be a basis for improving the effectiveness of instrument classification, instrument identification, or both, to be performed in operation 330.

[0032] In operation 320, image accessor 210 accesses a second image captured after the procedure was initiated (e.g., midway through the procedure, just before the procedure was completed, or after the procedure was completed). The second image depicts a proper subset (e.g., a portion) of the set of instruments depicted in the first image. For example, the second image may be captured by camera 240 of device 130 (e.g., by taking a digital photograph of a surgical tray on which a portion of the set of instruments depicted in the first image has been arranged after the surgical procedure was initiated and before the surgeon closed the patient on whom the surgical procedure was performed). In some example embodiments, the second image is accessed by image accessor 210 from database 115 via network 190. One or more fiducial markers may also be depicted in the second image, and such fiducial markers may be the basis for improving the effectiveness of instrument classification, instrument identification, or both to be performed in operation 330.

[0033] In operation 330, the instrument identifier 220 determines that an instrument from the set of instruments depicted in the first image is not depicted in a proper subset of the set of instruments in the second image. According to some example embodiments, the instrument identifier 220 performs instrument classification to determine that a non-specific instance of a particular type of instrument is missing from the second image (e.g., one of seven tweezers is missing because seven tweezers are depicted in the first image and only six tweezers are depicted in the second image). According to certain example embodiments, the instrument identifier 220 performs instrument recognition to determine that a particular individual instrument is missing from the second image (e.g., a particular pair of scissors is depicted in the first image but not in the second image). In hybrid example embodiments, the instrument identifier 220 performs both instrument recognition and instrument classification. In further example embodiments, such as when counting of discrete instruments cannot be performed with a minimum threshold confidence value, the instrument identifier 220 performs aggregated instrument detection and aggregated instrument classification to determine that an aggregate of instruments of a shared type (e.g., a stack of clips) has changed (e.g., decreased) in volume, area, height, or other dimensional metric from the first image to the second image.

[0034] To perform instrument classification, the instrument discriminator 220 can be or include an artificial intelligence module (e.g., an artificial intelligence machine learning module trained to implement one or more computer vision algorithms) in the form of an instrument classifier trained to detect and classify instruments depicted in an image (e.g., an image depicting surgical instruments arranged in an instrument tray), for example, using real-time computer vision techniques. The instrument classifier can be or include a deep convolutional neural network, such as a deep convolutional neural network having several convolutional layers. The top of the deep convolutional neural network can have an activation layer (e.g., a Softmax activation layer) and can have N outputs to predict the probabilities of N different types (e.g., classes) of instruments. Thus, the instrument classifier can select the instrument type corresponding to the highest probability as the predicted type of the depicted instrument.

[0035] In some example embodiments, the instrument classifier in the instrument discriminator 220 is trained (e.g., by a trainer machine) based on a classification training model having multiple convolutional layers with increasing filter sizes. For example, there may be four convolutional layers with filter sizes increasing from 8 to 32, with calibrated learning units as their activation function, followed by a batch norm layer, a pooling layer, or both. Thus, a fully connected layer may include 14 nodes representing the 14 types (e.g., categories) of instruments in the training set, and a Softmax activation. The classification training model may use an adaptive learning rate optimization algorithm (e.g., Adam) and may use categorical cross entropy as the loss function.

[0036] According to certain example embodiments, the training set may include (e.g., exclusively or non-exclusively) reference images of a reference set depicting instruments, and such a reference set may be customized for a particular procedure, surgeon, hospital, instrument supplier, geographic region, or any suitable combination thereof. Furthermore, the training set may include reference images captured under various lighting conditions, reference images with corresponding three-dimensional data (e.g., depth data or a model of the depicted instrument), reference images of reference transports (e.g., a reference tray, which may be empty or loaded with instruments), reference images of background items (e.g., a towel, a curtain, a floor surface, or a table surface), or any suitable combination thereof.

[0037] To perform instrument recognition, the instrument discriminator 220 can be or include an artificial intelligence module in the form of an object recognizer that is trained to locate and identify objects of interest within a given image (e.g., by drawing a bounding box around the located instrument and analyzing the contents within the bounding box). For example, the object recognizer can be or include a single shot detector (SSD) with inception-V2 as a feature extractor. However, other variants of neural network architectures and other types of neural networks suitable for detecting, classifying, or identifying objects may be suitable to balance the trade-off between accuracy and inference time. An example training dataset can include N (e.g., N=10) different types (e.g., categories) of instruments and hundreds to thousands of images (e.g., 227 images or 5000 images). The object recognizer can use the PascalVOC metric to evaluate bounding box metrics.

[0038] In certain example embodiments, the object recognizer in the instrument discriminator 220 is trained using a large synthetic dataset (e.g., to avoid problems caused by using a dataset that is too small). The example training process begins with a training machine (e.g., controlling a renderer machine or acting as a renderer machine) physically simulating a three-dimensional (3D) scene and simulating known camera parameters. The trainer machine then randomly places 3D objects in the scene (e.g., randomly placing 3D instruments on a 3D tray) and renders the scene based on various factors such as object occlusion, lighting, and shadows. The trainer robot then manually captures images of the rendered 3D objects in the scene. The system randomly changes (e.g., via domain randomization) the position, orientation or pose of the 3D objects, the lighting, the camera position, and the number of 3D objects in this simulation to automatically generate a large and diverse synthetic dataset of images. The trainer machine can automatically generate appropriate corresponding data labels (e.g., bounding boxes and segmentation masks) during simulation, thereby reducing annotation costs.

[0039] According to some example embodiments, the trainer machine trains the object recognizer in instrument discriminator 220 as follows. First, the trainer machine pre-trains the object recognizer (e.g., trains an object recognition model implemented by the object recognizer) using a synthetically generated dataset of images depicting surgical instruments. After pre-training using this synthetic dataset, the trainer machine modifies (e.g., by further training) the object recognizer based on a small dataset of real (e.g., non-synthetic) images depicting a specific type of surgical instrument. The small dataset of real images may be manually curated. In some example embodiments, a suitable surrogate for the trainer machine performs the training of the object recognizer.

[0040] Step 1: To generate a synthetic surgical instrument dataset, the trainer machine may launch or otherwise invoke one or more rendering applications (e.g., or Unreal Gaming ). To render the synthetic images, the trainer machine has access to the following example inputs:

[0041] 1. 3D models of different surgical instruments (e.g., computer-aided design (CAD)

[0042] Model),

[0043] 2. Surface texture information of surgical instruments,

[0044] 3. The range of parameters used to define the lighting (e.g., to simulate a hospital operating room), such as the range of brightness or the range of spectral composition variation,

[0045] 4. The range of possible camera positions (e.g., azimuth, elevation, pan, tilt, etc.) relative to the surgical tray where the surgical instruments are to be placed to capture images from specific angles, and

[0046] 5. The number and type of instruments to be rendered.

[0047] In certain example embodiments, a trainer machine manually places a random number of virtual surgical instruments on a virtual surgical tray with arbitrary orientations and positions. The trainer machine then generates synthetic images exhibiting varying amounts of occlusion, ranging from no occlusion to severe occlusion. During this simulation, the amount of occlusion in the synthetic dataset can be a custom parameter.

[0048] Step 2: The trainer machine uses the synthetic surgical instrument dataset generated in step 1 as a training dataset to train the object recognizer.

[0049] Step 3: The trainer machine accesses (e.g., from database 115) a small dataset of realistic images depicting real surgical instruments naturally placed on a real surgical tray. The dataset of realistic images helps bridge the gap between using synthetic images and using real images (e.g., the difference between training an object recognizer using only a large number of synthetic images and training an object recognizer using only a small number of real images).

[0050] In some embodiments, one or more fiducial markers on a transport for an instrument (e.g., on a surgical tray) or on the instrument itself can be used to facilitate instrument detection, instrument classification, instrument identification, or any suitable combination thereof. For example, where the transport is a specialized orthopedic tray, the instrument identifier 220 can access a template image (e.g., a mask image) depicting an empty orthopedic tray without any instruments, and then subtract the template image from the first image to obtain a first difference image (e.g., a first segmented image) that more clearly depicts the individual instruments before the initiation of the procedure. Similarly, the instrument identifier 220 can subtract the template image from the second image to obtain a second difference image (e.g., a second segmented image) that more clearly depicts the individual instruments after the initiation of the procedure (e.g., at or near the end of the procedure). The first and second difference images can be prepared by the instrument identifier 220 in preparation for operation 340 or an alternative embodiment of instrument detection, instrument classification, instrument identification, or any suitable combination thereof, in which case the orthopedic tray acts as a fiducial marker in the first image, the second image, or both.

[0051] In certain example embodiments, the outputs of multiple independent classifiers (e.g., deep learning classifiers, difference image classifiers, or any suitable combination thereof) are combined to improve the accuracy, precision, or both with respect to a given conveyance (e.g., a pallet) of instruments when performing instrument classification, instrument identification, or both. Specifically, the independent algorithms can determine and output corresponding probabilities indicating (1) whether the pallet is complete or incomplete, (2) whether each of a plurality of predetermined template regions of the pallet is filled or unfilled, and (3) what the classification of each object detected on the pallet is (e.g., whether it is an instrument, and if so, what type of instrument). The union of these three independent algorithms can better indicate whether the pallet is indeed complete, and if so, which instrument may be missing.

[0052] In operation 340, the notifier 230 causes a notification to be presented (e.g., visually, audibly, or both) indicating that an instrument not depicted in the second image is missing from the set of instruments. Presentation of the notification may take the form of displaying a pop-up window, playing an alarm sound, sending a message to another device (e.g., a nurse's or surgeon's smartphone), triggering a predetermined process corresponding to the instrument deemed missing (e.g., an instrument search process or a patient check process), or any suitable combination thereof.

[0053] According to various example embodiments, the presented notification indicates whether a particular instrument is missing. Alternatively or additionally, the presented notification may indicate whether a transport (e.g., a tray or cart) for the collection of instruments depicted in the first image is complete or incomplete (e.g., compared to a reference collection of instruments, such as a standard surgical tray for instruments, a closed tray for instruments, or an orthopedic tray for instruments). Alternatively or additionally, the presented notification may include a standardized report that lists each instrument in the collection of instruments depicted in the first image, along with a corresponding indicator (e.g., a tag or sign) of whether the instrument was used, when the instrument was picked up (e.g., as a timestamp), when the instrument was returned (e.g., to a scrub technician or transport), the length of time the instrument was in the patient, whether the instrument was present or is still retained, or any suitable combination thereof. In some example embodiments, the presented notification includes a total count of missing instruments, a list of missing instruments (e.g., by type, by specific individual instruments, or both), or any suitable combination thereof.

[0054] Additionally, a user feedback feature may be implemented by the app 200 such that the user 132 is prompted to confirm or correct a presented total count of the presence or absence of tools, and the user's 132 response is used as a label to further train and improve one or more artificial intelligence modules in the instrument identifier 220. In some example embodiments, the app 200 may operate with further user interaction and prompt the user 132 to confirm (e.g., manually, visually, or both) some or all of the information contained in the presented notification.

[0055] like Figure 4 As shown in , in addition to any one or more of the operations previously described for method 300 , method 300 may further include one or more of operations 410 , 412 , 422 , 430 , 431 , 434 , 435 , 436 , and 437 .

[0056] Operation 410 may be performed as part of (e.g., a preceding task, a subroutine, or a portion of) operation 310, in which the image accessor 210 accesses a first image. In operation 410, the first image is a reference image that depicts a set of instruments (e.g., a standardized set of instruments) that corresponds to a protocol (e.g., by being specified for the protocol), corresponds to a performer of the protocol (e.g., a surgeon) (e.g., by being specified by the performer), or corresponds to both, and the reference image is accessed based on (e.g., in response to) its correspondence with the protocol, the performer of the protocol, or both.

[0057] In an alternative example embodiment, operation 412 can be performed as part of operation 310. In operation 412, a first image is accessed by capturing the first image as part of a sequence of captured frames (e.g., a first sequence of first frames of a video). For example, image accessor 210 can access video data from camera 240 while device 130 moves over a collection of instruments (e.g., past a surgical tray holding the collection of instruments) and records a sequence of video frames, where this is the first image. In such an example embodiment, app 200 can include and execute a stereo algorithm (e.g., a structure from motion algorithm) configured to infer depth data from the sequence of video frames including the first image, and this depth data can be the basis for determining instrument loss or other factors in operation 330.

[0058] Similarly, in some example embodiments, operation 422 may be performed as part of operation 320, where the image accessor 210 accesses the second image. In operation 422, the second image is accessed by capturing the second image as part of a sequence of captured frames (e.g., a second sequence of second frames of a video). For example, the image accessor 210 may access video data from the camera 240 while the device 130 moves over a portion of a collection of instruments (e.g., past a surgical tray holding a portion of the collection of instruments) and records a sequence of video frames, of which it is the second image. In such example embodiments, the app 200 may include and execute a stereo algorithm configured to infer depth data from the sequence of video frames including the second image, and this depth data may be the basis for determining instrument loss or other factors in operation 330.

[0059] like Figure 4 As shown in , operations 430 and 431 may be performed as part of operation 330 , where instrument identifier 220 determines that an instrument from the set of instruments depicted in the first image is not depicted in a proper subset of the set of instruments in the second image.

[0060] In operation 430, the instrument identifier 220 identifies the shape of the instrument in the first image (e.g., optically, with or without supplemental support from depth data). For example, as described above, the instrument identifier 220 can be or include an instrument classifier, an object recognizer, or a combination of both, and the instrument identifier 220 can be trained accordingly to detect (e.g., recognize) and classify an instrument by its shape, as depicted in the first image.

[0061] In operation 431, the instrument identifier 220 attempts but fails (e.g., optically, with or without supplemental support from depth data) to discern the shape of the instrument in the second image. For example, as described above, the instrument identifier 220 can be or include an instrument classifier, an object recognizer, or a combination of both, and the instrument identifier 220 can be trained accordingly to detect (e.g., recognize) and classify the shape of the instrument passing through it, as depicted in the second image. However, because the instrument is not depicted in the second image, the instrument identifier 220 fails to detect or classify the instrument.

[0062] like Figure 4 As shown in , operations 434 and 435 may be performed as part of operation 330 , where instrument identifier 220 determines that an instrument in the set of instruments depicted in the first image is not depicted in a proper subset of the set of instruments in the second image.

[0063] In operation 434, the instrument identifier 220 accesses a reference model of the instrument. The reference model may be three-dimensional and may be accessed from the database 115. For example, if operation 430 has been performed, the instrument identifier 220 may access the reference model of the instrument based on (e.g., in response to) identifying the instrument by its shape in operation 430.

[0064] In operation 435, the instrument identifier 220 attempts but fails to identify (e.g., optically, with or without supplemental support from depth data) each of a plurality of contours of the reference model of the instrument (e.g., as accessed in operation 434) in the second image. For example, the instrument identifier 220 may generate a set of contours from the reference model and compare each contour in the set of contours to the shape of an instrument in a proper subset of the set of instruments, as depicted in the second image. However, because the instrument is not depicted in the second image, the instrument identifier 220 fails to identify any contours of the reference model of the instrument in the second image.

[0065] like Figure 4 As shown in , operations 436 and 437 may be performed as part of operation 330 , where instrument identifier 220 determines that an instrument in the set of instruments depicted in the first image is not depicted in a proper subset of the set of instruments in the second image.

[0066] In operation 436, the instrument identifier 220 accesses depth data representing the current shape of the proper subset of the set of instruments depicted in the second image. For example, the depth data may be captured by the depth sensor 250 of the device 130, and the instrument identifier 220 may access the depth data from the depth sensor 250.

[0067] In operation 437, the instrument identifier 220 compares the reference shape of the instrument to each current shape of the proper subset of the set of instruments. As described above, the current shape may be represented by the depth data accessed in operation 436. If operation 434 was previously performed to access a reference model representing the reference shape of the instrument, the same reference shape may be used in the comparison performed in operation 437. In other example embodiments, operation 437 includes accessing or otherwise obtaining the reference shape of the instrument (e.g., in a manner similar to that described above for operation 434).

[0068] Figure 5 and 6 is a flowchart illustrating the operation of the device 130 in performing the method 500 of tracking an instrument according to some example embodiments. The operations in the method 500 may be performed by the device 130 using the above description of Figure 2 The components (e.g., modules) described herein may utilize one or more processors (e.g., microprocessors or other hardware processors), or any suitable combination thereof. Figure 5 As shown in , method 500 includes operations 510 , 520 , 530 , and 540 .

[0069] In operation 510, image accessor 210 accesses (e.g., receives, retrieves, reads, or otherwise obtains) a first image captured prior to initiation of a procedure. The first image depicts a collection of instruments that may be used in performing the procedure. For example, the first image may be captured by camera 240 of device 130 (e.g., by taking a digital photograph of a surgical tray in which the collection of surgical instruments has been arranged in preparation for a surgical procedure to be performed by a surgeon). In some example embodiments, the first image is accessed by image accessor 210 from database 115 via network 190. One or more fiducial markers may also be depicted in the first image, and such fiducial markers may be a basis for improving the effectiveness of instrument classification, instrument identification, or both, to be performed in operation 530. In various example embodiments, operation 510 is performed similarly to operation 310 in method 300, as described above.

[0070] In operation 520, image accessor 210 accesses a second image captured after the procedure was initiated. The second image depicts a subset of the set of instruments depicted in the first image. The subset may be a true subset (e.g., a portion) of the set of instruments or a subset that is consistent with the entire set of instruments. That is, there may be no missing instruments in the second image. For example, the second image may be captured by camera 240 of device 130 (e.g., by taking a digital photograph of a surgical tray on which a portion of the set of instruments depicted in the first image has been arranged after the surgical procedure was initiated and before the surgeon closed the patient on whom the surgical procedure was performed). In some example embodiments, the second image is accessed by image accessor 210 from database 115 via network 190. One or more fiducial markers may also be depicted in the second image, and such fiducial markers may be the basis for improving the effectiveness of the instrument identification to be performed in operation 530.

[0071] In operation 530, the instrument identifier 220 determines, based on the first and second images, whether an instrument in the set of instruments depicted in the first image is used or not used in the procedure. According to some example embodiments, the instrument identifier 220 performs instrument recognition to determine whether a particular individual instrument is present in both images but exhibits one or more optically detectable indications of use. Such indications include, for example, movement from a first position within a transport (e.g., a surgical tray) in the first image to a second position within the transport in the second image, a change in appearance from the absence of bioburden (e.g., one or more spots of patient fluid (such as blood)) in the first image to the presence of bioburden in the second image, or any suitable combination thereof. Example details of algorithms used by the instrument identifier 220 in performing instrument recognition (e.g., via the object identifier) ​​and example details of training the instrument identifier 220 (e.g., the object identifier) ​​are discussed above (e.g., with respect to operation 330 in method 300). For example, one or more fiducial markers may be used in the first image, the second image, or both in a manner similar to that described above.

[0072] In operation 540, the notifier 230 causes a notification to be presented (e.g., visually, audibly, or both) indicating whether the instrument is used or not used in the procedure. The presentation of the notification may take the following example forms: displaying a pop-up window, playing a warning sound, sending a message to another device (e.g., a smartphone of a nurse, surgeon, caregiver, or inventory manager), triggering a predetermined process corresponding to the use or non-use status determined for the instrument (e.g., a used instrument counting process, an unused instrument counting process, or an instrument sterilization process), or any suitable combination thereof.

[0073] Additionally, in certain exemplary embodiments, operations 510 and 530 can be performed without one or both of operations 520 and 540, such that the instruments depicted in the first image are directly classified, identified, or both, with a result presentation of a notification indicating a count of instruments classified, identified, or both; the type of instrument, the name of the instrument, a reference image of the instrument, or any suitable combination thereof. Such exemplary embodiments can be helpful in situations where a new scrub tech or new surgeon cannot recall or does not know what the instrument is called. To quickly classify or identify an instrument by machine, the new scrub tech or new surgeon can hold the instrument in front of the camera 240 of the device 130, and the app 200 can use computer vision and deep learning (e.g., by performing instrument classification, instrument identification, or both, as described above with respect to operation 330) to obtain an answer, which can be provided with a likelihood score indicating a level of confidence in the answer. Some of these example embodiments are located in an installed supply chain environment where a camera (e.g., camera 240) is positioned to image a table, an instrument tray assembler places instruments on the table, and a device (e.g., device 130 as described herein) scans the instruments and performs automatic classification, identification, or both on the scanned instruments.

[0074] Furthermore, in various example embodiments, operations 310 and 530 can be performed without one or both of operations 520 and 540, thereby enabling the systems and methods discussed herein to replace one or more highly manual processes (e.g., manually marking a standardized electronic checklist when an instrument is manually added to a new tray) with an automated checklist. In such example embodiments, a tray assembler can grab an instrument, image the instrument (e.g., using one or more of a variety of modalities), and the device (e.g., device 130) automatically checks the instrument on the assembly sheet (e.g., lists the instrument to be added to the new tray).

[0075] Furthermore, in some example embodiments, operations 510 and 530 are performed repeatedly such that a visual record is generated to track, for example, whether or when each instrument was removed from the tray, whether or when each removed instrument was returned to the tray, and whether each removed instrument appears to have been used. All or part of this visual record can be provided (e.g., by device 130 to database 115) for inclusion in an electronic medical record (e.g., corresponding to the patient undergoing the procedure).

[0076] Moreover, the automatic classification or identification of instruments discussed herein can be extended beyond instruments on trays to provide similar benefits for any other consumables found in hospital operating rooms, and also forward to other settings. For example, medications in a medication cart can be tracked in a manner similar to that described herein for instruments on trays (e.g., to ensure that controlled substances, such as opioids, are not abused or lost during surgical procedures). Thus, in some example embodiments, medications can be scanned by a device (e.g., device 130) configured by an app (e.g., app 200), and when the anesthesiologist uses each medication, the device can cause a notification to be presented indicating all or part of a visual record of the medication cart. The visual record can indicate how each medication (e.g., each controlled substance) was administered or otherwise used, as well as a corresponding timestamp of the administration or other use.

[0077] like Figure 6 , in addition to any one or more of the operations previously described with respect to method 500, method 500 may also include one or more of operations 630, 632, and 634. One or more of operations 630, 632, and 634 may be performed as part of operation 630, where the instrument identifier 220 determines, based on the first and second images, whether the instrument depicted in the first image is used or not used in the procedure.

[0078] In operation 630 , as part of determining whether the implement is in use, the implement identifier 220 determines whether the implement is moved from a first location within the vehicle depicted in the first image to a second location within the vehicle depicted in the second image.

[0079] In operation 632 , as part of determining whether the instrument is used, the instrument identifier 220 identifies (eg, optically) the absence of bioburden (eg, spots of blood or other bodily fluids from the patient) on the instrument depicted in the first image.

[0080] In operation 634 , as part of determining whether the instrument is used, the instrument identifier 220 identifies (e.g., optically) the presence of bioburden (e.g., one or more spots of blood or another bodily fluid from the patient) on the same instrument, as depicted in the second image.

[0081] According to various example embodiments, one or more of the methods described herein can facilitate tracking of instruments (e.g., surgical instruments). Furthermore, one or more of the methods described herein can facilitate detecting and quantifying instruments by type, detecting and tracking individual instruments, or both. Thus, one or more of the methods described herein can facilitate more precise and accurate management of instrument inventory and associated maintenance costs (e.g., sterilization procedures), as well as reduced health and safety risks (e.g., of patients undergoing medical procedures), compared to the capabilities of existing systems and methods.

[0082] Figure 7 is a screenshot illustrating an image depicting an instrument, and in which the device 130 configured by the app 200 has added a bounding box indicating the instrument, according to some example embodiments.

[0083] Figure 8-10 is a screenshot illustrating images depicting instruments and in which the device 130 configured by the app 200 has added a count of the number of instruments for each image, both individually and by type of instrument, in accordance with some example embodiments.

[0084] like Figure 7-10 As shown in , app 200 can configure device 130 to scan its environment in real time using camera 240 while continuously running instrument identifier 220 (e.g., running object recognizer) so that, when camera 240 captures any instruments, app 200 displays bounding boxes around the instruments, along with their types (e.g., class names) and a count of the total instruments identified in the image (e.g., in the currently displayed video frame).

[0085] According to the techniques discussed herein, app 200 can configure any suitable device (e.g., device 130) to use an instrument identification algorithm (e.g., implemented in instrument identifier 220, as described above). App 200 can provide user 132 with the ability to invoke any of a variety of operating modes for app 200, for device 130, or for both. By way of example, such operating modes can include an operating room full-feature mode (e.g., a full mode in which all features are enabled), an operating room partial-feature mode (e.g., a light mode in which computationally intensive features are disabled or offloaded), a supply chain mode, a post-operative quality control mode, an orthopedic sales mode, or any suitable combination thereof.

[0086] Any one or more of the above algorithms for instrument classification or image recognition can be independently applied to different use cases in different contexts. In addition, any one or more of these algorithms can be applied to a portion of an instrument (e.g., the tip of a pair of scissors, the handle of an instrument, or the fulcrum of an instrument) and accordingly perform partial classification, partial recognition, or both, in a manner similar to that described herein for instrument classification, instrument recognition, or both. Thus, various examples of the instrument discriminator 220 can include additional artificial intelligence modules trained on instrument portions (e.g., using one or more deep learning networks), and the additional artificial intelligence modules can be used to support (e.g., confirm, verify, or modify) the classification, recognition, or both performed by the primary artificial intelligence module on the entire instrument, the entire tray, or both.

[0087] When these effects are taken together, one or more of the methods described herein can eliminate the need for certain efforts or resources that would otherwise be involved in instrument tracking. The effort expended by a user in tracking an instrument can be reduced by using (e.g., relying on) a dedicated machine that implements one or more of the methods described herein. The computing resources used by one or more systems or machines (e.g., within the network environment 100) can be similarly reduced (e.g., compared to a system or machine that lacks the structures discussed herein or is unable to perform the functions discussed herein). Examples of such computing resources include processor cycles, network traffic, computing capacity, main memory usage, graphics rendering capacity, graphics memory usage, data storage capacity, power consumption, and cooling capacity.

[0088] Figure 11 is a block diagram illustrating components of a machine 1100 according to some example embodiments, which is capable of reading instructions 1124 from a machine-readable medium 1122 (e.g., a non-transitory machine-readable medium, a machine-readable storage medium, a computer-readable storage medium, or any suitable combination thereof) and performing any one or more of the methodologies discussed herein, in whole or in part. Specifically, Figure 11 The machine 1100 is shown in the example form of a computer system (e.g., a computer) in which instructions 1124 (e.g., software, a program, an application, an applet, an app, or other executable code) for causing the machine 1100 to perform, in whole or in part, any one or more of the methodologies discussed herein may be executed.

[0089] In some embodiments, the machine 1100 may be a standalone device or may be communicatively coupled (e.g., networked) to other machines. In a networked deployment, the machine 1100 may operate in the capacity of a server or a client machine in a server-client network environment, or as a peer machine in a distributed (e.g., peer-to-peer) network environment. The machine 1100 may be a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook computer, a cellular phone, a smartphone, a set-top box (STB), a personal digital assistant (PDA), a web appliance, a network router, a network switch, a network bridge, or any other machine capable of executing instructions 1124, sequentially or otherwise, to specify actions for that machine to take. While a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute instructions 1124 to perform all or a portion of any one or more of the methodologies discussed herein.

[0090] The machine 1100 includes a processor 1102 (e.g., one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more digital signal processors (DSPs), one or more application-specific integrated circuits (ASICs), one or more radio-frequency integrated circuits (RFICs), or any suitable combination thereof), a main memory 1104, and a static memory 1106, which are configured to communicate with each other via a bus 1108. The processor 1102 contains solid-state digital microcircuits (e.g., electronic, optical, or both) that are temporarily or permanently configurable by some or all of the instructions 1124 so that the processor 1102 can be configured to perform, in whole or in part, any one or more of the methodologies described herein. For example, the collection of one or more microcircuits of the processor 1102 can be configured to execute one or more modules (e.g., software modules) described herein. In some example embodiments, the processor 1102 is a multi-core CPU (e.g., a dual-core CPU, a quad-core CPU, an 8-core CPU, or a 128-core CPU), where each of the multiple cores behaves as a separate processor capable of performing, in whole or in part, any one or more of the methodologies discussed herein. Although the benefits described herein may be provided by a machine 1100 having at least a processor 1102, these same benefits may be provided by a different kind of machine (e.g., a purely mechanical system, a purely hydraulic system, or a hybrid mechanical-hydraulic system) that does not include a processor if such a processor-less machine is configured to perform one or more of the methods described herein.

[0091] The machine 1100 may also include a graphics display 1110 (e.g., a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, a cathode ray tube (CRT), or any other display capable of displaying graphics or video). The machine 1100 may also include an alphanumeric input device 1112 (e.g., a keyboard or keypad), a pointer input device 1114 (e.g., a mouse, touchpad, touch screen, trackball, joystick, stylus, motion sensor, eye tracking device, data glove, or other pointing device), a data storage device 1116, an audio generating device 1118 (e.g., a sound card, amplifier, speakers, headphone jack, or any suitable combination thereof), and a network interface device 1120.

[0092] The data storage 1116 (e.g., a data storage device) includes a machine-readable medium 1122 (e.g., a tangible and non-transitory machine-readable storage medium) having stored thereon instructions 1124 for implementing any one or more of the methodologies or functions described herein. The instructions 1124 may also reside, completely or at least partially, within the main memory 1104, within the static memory 1106, within the processor 1102 (e.g., within a cache memory of the processor), or any suitable combination thereof, before or during execution by the machine 1100. Thus, the main memory 1104, the static memory 1106, and the processor 1102 may be considered machine-readable media (e.g., a tangible and non-transitory machine-readable medium). The instructions 1124 may be transmitted or received over the network 190 via the network interface device 1120. For example, the network interface device 1120 may use any one or more transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)).

[0093] In some example embodiments, the machine 1100 may be a portable computing device (e.g., a smartphone, tablet, or wearable device) and may have one or more additional input components 1130 (e.g., sensors or gauges). Examples of such input components 1130 include an image input component (e.g., one or more cameras), an audio input component (e.g., one or more microphones), a directional input component (e.g., a compass), a position input component (e.g., a global positioning system (GPS) receiver), an orientation component (e.g., a gyroscope), a motion detection component (e.g., one or more accelerometers), an altitude detection component (e.g., an altimeter), a temperature input component (e.g., a thermometer), and a gas detection component (e.g., a gas sensor). Input data collected by any one or more of these input components 1130 may be accessed and used by any of the modules described herein (e.g., with appropriate privacy notices and protections, such as opt-in or opt-out consent, implemented according to user preferences, applicable regulations, or any suitable combination thereof).

[0094] As used herein, the term "memory" refers to a machine-readable medium capable of storing data temporarily or permanently, and may be considered to include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, and cache memory. Although the machine-readable medium 1122 is shown as a single medium in the example embodiment, the term "machine-readable medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) that can store instructions. The term "machine-readable medium" should also be considered to include any medium or combination of multiple media that can carry (e.g., store or transmit) instructions 1124 for execution by the machine 1100, such that the instructions 1124, when executed by one or more processors (e.g., processor 1102) of the machine 1100, causes the machine 1100 to perform, in whole or in part, any one or more of the methodologies described herein. Thus, "machine-readable medium" refers to a single storage device or apparatus, as well as a cloud-based storage system or storage network comprising multiple storage devices or apparatuses. Thus, the term "machine-readable medium" shall be taken to include, but is not limited to, one or more tangible and non-transitory data repositories (e.g., data volumes) in the example form of solid-state memory chips, optical disks, magnetic disks, or any suitable combination thereof.

[0095] As used herein, "non-transitory" machine-readable media specifically excludes propagating signals per se. According to various example embodiments, instructions 1124 for execution by the machine 1100 may be transmitted via a carrier medium (e.g., a machine-readable carrier medium). Examples of such carrier media include non-transitory carrier media (e.g., a non-transitory machine-readable storage medium, such as a solid-state memory that can be physically moved from one place to another) and transitory carrier media (e.g., a carrier wave or other propagating signal that transmits the instructions 1124).

[0096] Certain example embodiments are described herein as including modules. A module may constitute a software module (e.g., code stored or otherwise implemented in a machine-readable medium or transmission medium), a hardware module, or any suitable combination thereof. A "hardware module" is a tangible (e.g., non-transient) physical component (e.g., a collection of one or more processors) that is capable of performing certain operations and that can be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems or one or more hardware modules thereof may be configured by software (e.g., an application or portion thereof) as a hardware module that operates to perform the operations described herein for that module.

[0097] In some example embodiments, the hardware modules may be implemented mechanically, electronically, hydraulically, or any suitable combination thereof. For example, the hardware modules may include dedicated circuitry or logic that is permanently configured to perform certain operations. The hardware modules may be or include dedicated processors, such as field programmable gate arrays (FPGAs) or ASICs. The hardware modules may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. As an example, the hardware modules may include software encompassed within a CPU or other programmable processor. It will be appreciated that the decision to implement a hardware module mechanically, hydraulically, in dedicated and permanently configured circuitry, or in a temporarily configured circuitry (e.g., configured by software) may be driven by cost and time considerations.

[0098] Thus, the phrase "hardware module" should be understood to encompass a tangible entity that can be physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or perform certain operations described herein. In addition, as used herein, the phrase "hardware-implemented module" refers to a hardware module. Considering example embodiments in which hardware modules are temporarily configured (e.g., programmed), each hardware module need not be configured or instantiated at any one instance in time. For example, where a hardware module includes a CPU that is configured as a special-purpose processor by software, the CPU can be configured as different special-purpose processors at different times (e.g., each included in a different hardware module). Software (e.g., software modules) can configure one or more processors accordingly, for example, to become or otherwise constitute a particular hardware module at one instance in time, and to become or otherwise constitute a different hardware module at a different instance in time.

[0099] A hardware module can provide information to other hardware modules and receive information from other hardware modules. Thus, the described hardware modules can be considered to be communicatively coupled. In the case where multiple hardware modules exist simultaneously, communication can be achieved by signal transmission (e.g., through circuits and buses) between two or more of the hardware modules. In an embodiment in which multiple hardware modules are configured or instantiated at different times, the communication between these hardware modules can be achieved, for example, by storing and retrieving information in a memory structure accessible to multiple hardware modules. For example, a hardware module can perform an operation and store the output of the operation in a memory (e.g., a memory device) to which it is communicatively coupled. Then, another hardware module can access the memory at a later time to retrieve and process the stored output. The hardware module can also initiate communication with an input or output device and can operate on a resource (e.g., a collection of information from a computing resource).

[0100] The various operations of the example methods described herein may be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions described herein. As used herein, a "processor-implemented module" refers to a hardware module in which the hardware includes one or more processors. Thus, the operations described herein may be at least partially processor-implemented, hardware-implemented, or both, as processors are examples of hardware, and at least some of the operations in any one or more of the methods discussed herein may be performed by one or more processor-implemented modules, hardware-implemented modules, or any suitable combination thereof.

[0101] Moreover, such one or more processors can perform operations in a "cloud computing" environment or as a service (e.g., in a "software as a service" (SaaS) implementation). For example, at least some of the operations within any one or more of the methods discussed herein can be performed by a group of computers (e.g., as an example of a machine including a processor), which can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application program interfaces (APIs)). The execution of certain operations can be distributed among one or more processors, whether residing only within a single machine or deployed across multiple machines. In some example embodiments, one or more processors or hardware modules (e.g., processor-implemented modules) can be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other example embodiments, one or more processors or hardware modules can be distributed across multiple geographic locations.

[0102] Throughout this specification, multiple instances can be implemented as components, operations or structures described as single instances. Although the individual operations of one or more methods are illustrated and described as separated operations, one or more of the individual operations can be performed concurrently, and there is no requirement to perform these operations in the order shown. The structure and function thereof presented as separate components and functions in the example configuration can be implemented as a combined structure or component with a combined function. Similarly, the structure and functionality presented as a single component can be implemented as an independent component and function. These and other variations, modifications, additions and improvements fall within the scope of this paper theme.

[0103] Some parts of the subject matter discussed herein can be presented according to an algorithm or a symbolic representation of the operation of the data of the bit or binary digital signal stored as a memory (for example, a computer memory or other machine memory). Such an algorithm or symbolic representation is an example of the technology that a person of ordinary skill in the field of data processing uses to convey the essence of their work to other persons of skill in the art. As used herein, an "algorithm" is a self-consistent sequence of operations or similar processing that results in a desired result. In this context, algorithms and operations relate to the physical manipulation of physical quantities. Typically, but not necessarily, such quantities can take the form of electrical, magnetic or optical signals that can be stored, accessed, transmitted, combined, compared or otherwise manipulated by a machine. Sometimes, primarily for commonly used reasons, it is convenient to use words such as "data", "content", "bit", "value", "element", "symbol", "character", "item", "digital", "numerical value" to refer to such signals. However, these words are merely convenient labels and should be associated with appropriate physical quantities.

[0104] Unless expressly stated otherwise, discussions herein using words such as "access," "process," "select," "calculate," "compute," "determine," "generate," "present," "display," and the like refer to actions or processes that can be performed by a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electrical, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or any suitable combination thereof), registers, or other machine components that receive, store, transmit, or display information. Furthermore, unless expressly stated otherwise, the terms "a" or "an" are used herein, as is common in patent literature, to include one or more than one instance. Finally, as used herein, the conjunction "or" refers to a non-exclusive "or" unless expressly stated otherwise.

[0105] The description set forth below describes various examples of the methods, machine-readable media, and systems (eg, machines, devices, or other apparatus) discussed herein.

[0106] A first example provides a method comprising:

[0107] accessing, by one or more processors of the machine, a first image capturing a reference set of instruments on the transport prior to initiation of a procedure;

[0108] identifying, from the first image and by one or more processors of the machine, first instrument data corresponding to a reference set of instruments;

[0109] accessing, by one or more processors of the machine, a second image capturing the instrument on the transport after initiation of the procedure;

[0110] identifying, from the second image and by one or more processors of the machine, second instrument data corresponding to an instrument on the transport vehicle after initiation of the procedure;

[0111] comparing, by one or more processors of the machine, the first instrument data to the second instrument data; and

[0112] Based on the comparison and causing, by one or more processors of the machine, presentation of a notification indicating that the instrument that was on the transport prior to initiation of the procedure is not present on the transport after initiation of the procedure.

[0113] A second example provides the method according to the first example, further comprising:

[0114] access reference images of instruments;

[0115] identifying an instrument in the first image based on the reference image, the first instrument data indicating the instrument identified in the first image; and

[0116] An instrument in the second image is identified based on the reference image, and the second instrument data indicates the identified instrument in the second image.

[0117] A third example provides the method according to the first example or the second example, further comprising:

[0118] optically discerning a shape of the instrument in the first image to obtain first instrument data; and

[0119] A shape of the instrument in the second image is optically discerned to obtain second instrument data.

[0120] A fourth example provides the method according to any one of the first to third examples, wherein the first and second images correspond to at least one of a type of procedure or a performer of the procedure.

[0121] A fifth example provides the method according to any one of the first to fourth examples, wherein the first instrument data includes a first instrument count, and the second instrument data includes a second instrument count.

[0122] A sixth example provides a method according to the fifth example, wherein comparing the first instrument data to the second instrument data includes comparing the first instrument count to the second instrument count; and wherein the notification indicates at least one of a total count of lost instruments or a total count of lost instruments of a shared type.

[0123] The seventh example provides a method according to any one of the first to sixth examples, wherein the procedure includes a surgical procedure performed by a physician on a patient; the first image captures a reference set of instruments on a transport vehicle before the physician begins performing the surgical procedure on the patient; and the second image captures the instruments on the transport vehicle after the physician completes the surgical procedure on the patient.

[0124] An eighth example provides a system (e.g., a computer system), comprising:

[0125] one or more processors; and

[0126] The memory stores instructions that, when executed by at least one of the one or more processors, cause the system to perform operations including:

[0127] accessing a first image that captures a reference set of instruments on a transport prior to initiation of a procedure;

[0128] identifying, from the first image, first instrument data corresponding to a reference set of instruments;

[0129] accessing a second image capturing the instrument on the transport after the procedure is initiated;

[0130] identifying, from the second image, second device data corresponding to the device on the transport vehicle after initiation of the procedure;

[0131] comparing the first instrument data to the second instrument data; and

[0132] Based on the comparison, presentation of a notification is caused indicating that the equipment that was on the vehicle before the procedure was initiated is not present on the vehicle after the procedure was initiated.

[0133] A ninth example provides the system according to the eighth example, wherein the operations further comprise:

[0134] optically discerning a shape of the instrument in the first image to obtain first instrument data; and

[0135] A shape of the instrument in the second image is optically discerned to obtain second instrument data.

[0136] A tenth example provides a machine-readable medium (e.g., a non-transitory machine-readable storage medium) comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

[0137] accessing a first image that captures a reference set of instruments on a transport prior to initiation of a procedure;

[0138] identifying, from the first image, first instrument data corresponding to a reference set of instruments;

[0139] accessing a second image capturing the instrument on the transport after the procedure is initiated;

[0140] identifying, from the second image, second device data corresponding to the device on the transport vehicle after initiation of the procedure;

[0141] comparing the first instrument data to the second instrument data; and

[0142] Based on the comparison, presentation of a notification is caused indicating that the equipment that was on the vehicle before the procedure was initiated is not present on the vehicle after the procedure was initiated.

[0143] The eleventh example provides a method, including:

[0144] accessing, by one or more processors of the machine, a first image captured prior to initiation of a procedure and depicting a set of instruments available for use in the procedure;

[0145] accessing, by one or more processors of the machine, a second image captured after initiation of the procedure and depicting a proper subset of the set of instruments depicted in the first image;

[0146] determining, by one or more processors of the machine, that an instrument in the set of instruments depicted in the first image is not depicted in a proper subset of the set of instruments in the second image; and

[0147] Presentation of a notification is caused by one or more processors of the machine, the notification indicating that an instrument not depicted in the second image is missing from the set of instruments.

[0148] A twelfth example provides the method according to the eleventh example, wherein:

[0149] Accessing a first image of the set depicting instruments includes accessing a reference image of a reference set depicting instruments.

[0150] A thirteenth example provides the method according to the twelfth example, wherein:

[0151] The reference image corresponds to at least one of a procedure or a performer of the procedure; and

[0152] Access to the reference image is based on at least one of a procedure or an implementer of the procedure.

[0153] A fourteenth example provides the method according to the twelfth example or the thirteenth example, wherein:

[0154] A reference set of instruments corresponds to at least one of a procedure or a performer of the procedure; and

[0155] Access to the reference images of the reference set depicting the instrument is based on at least one of a procedure or a performer of the procedure.

[0156] A fifteenth example provides the method according to any one of the eleventh to fourteenth examples, wherein:

[0157] Determining that an instrument from the set of instruments depicted in the first image is not depicted in the second image includes:

[0158] optically discerning a shape of the instrument in the first image; and

[0159] The shape of the instrument cannot be discerned optically in the second image.

[0160] A sixteenth example provides a method according to any one of the eleventh to fifteenth examples, wherein:

[0161] Determining that an instrument from the set of instruments depicted in the first image is not depicted in the second image includes:

[0162] Access reference models of devices; and

[0163] Each of the plurality of contours of the reference model of the instrument cannot be optically discerned in the second image.

[0164] A seventeenth example provides the method according to any one of the eleventh to sixteenth examples, wherein:

[0165] Determining that an instrument from the set of instruments depicted in the first image is not depicted in the second image includes:

[0166] accessing a reference model representing a reference shape of the instrument depicted in the first image;

[0167] accessing depth data representing a current shape of a proper subset of the set of instruments depicted in the second image; and

[0168] The reference shape of the instrument is compared to each current shape of a proper subset of the set of instruments.

[0169] An eighteenth example provides the method according to any one of the eleventh to seventeenth examples, wherein:

[0170] Accessing the first image is performed by capturing a first sequence of first frames prior to a procedure and selecting at least a first image from the captured first sequence; and

[0171] Accessing the second image is performed by capturing a second sequence of second frames following the procedure and selecting at least a second image from the captured second sequence.

[0172] The nineteenth example provides a method, including:

[0173] accessing, by one or more processors of the machine, a first image captured prior to initiation of a procedure and depicting a set of instruments available for use in the procedure;

[0174] accessing, by one or more processors of the machine, a second image captured after initiation of the procedure and depicting a subset of the set of instruments depicted in the first image;

[0175] determining, by one or more processors of the machine based on the first and second images, whether an instrument in the set of instruments depicted in the first image is used or unused in the procedure; and

[0176] The presentation of a notification is caused by one or more processors of the machine, the notification indicating that the instrument was or was not used in the procedure.

[0177] A twentieth example provides the method according to the nineteenth example, wherein:

[0178] A subset of the set of instruments is a proper subset of the set of instruments.

[0179] A twenty-first example provides the method according to the nineteenth example or the twentieth example, wherein:

[0180] Determining whether the instrument was used or not used in the procedure includes determining whether the instrument moved from a first location within the transport depicted in the first image to a second location within the transport depicted in the second image.

[0181] The twenty-second example provides the method according to any one of the nineteenth to twenty-first examples, wherein:

[0182] Determining whether a device was used or not used during a procedure includes:

[0183] optically discerning the absence of blood on the instrument depicted in the first image; and

[0184] The presence of blood on the instrument depicted in the second image is optically discerned.

[0185] A twenty-third example provides a machine-readable medium (e.g., a non-transitory machine-readable storage medium) including instructions that, when executed by one or more processors of a machine, cause the machine to perform operations including:

[0186] accessing a first image captured prior to initiation of a procedure and depicting a set of instruments available for use in the procedure;

[0187] accessing a second image captured after initiation of the procedure and depicting a subset of the set of instruments depicted in the first image;

[0188] determining that an instrument in the set of instruments depicted in the first image is not depicted in a proper subset of the set of instruments in the second image; and

[0189] Presentation of a notification is caused indicating that an instrument not depicted in the second image is missing from the set of instruments.

[0190] A twenty-fourth example provides the machine-readable medium according to the twenty-third example, wherein:

[0191] Determining that an instrument in the set of instruments depicted in the first image is not depicted in the second image includes:

[0192] optically discerning a shape of the instrument in the first image; and

[0193] The shape of the instrument cannot be visually recognized in the second image.

[0194] The twenty-fifth example provides a machine-readable medium (e.g., a non-transitory machine-readable storage medium) including instructions that, when executed by one or more processors of a machine, cause the machine to perform operations including:

[0195] accessing a first image captured prior to initiation of a procedure and depicting a set of instruments available for use in the procedure;

[0196] accessing a second image captured after initiation of the procedure and depicting a subset of the set of instruments depicted in the first image;

[0197] determining, based on the first and second images, whether an instrument from the set of instruments depicted in the first image is used or unused in the procedure; and

[0198] Causes presentation of a notification indicating whether the instrument was used or not used in the procedure.

[0199] A twenty-sixth example provides the machine-readable medium according to the twenty-fifth example, wherein:

[0200] Determining whether the instrument was used or not used in the procedure includes determining whether the instrument moved from a first location within the transport depicted in the first image to a second location within the transport depicted in the second image.

[0201] A twenty-seventh example provides a system comprising:

[0202] one or more processors; and

[0203] The memory stores instructions that, when executed by at least one of the one or more processors, cause the system to perform operations including:

[0204] accessing a first image captured prior to initiation of a procedure and depicted for use in the procedure;

[0205] accessing a second image captured after initiation of the procedure and depicting a proper subset of the set of instruments depicted in the first image;

[0206] determining that an instrument in the set of instruments depicted in the first image is not depicted in a proper subset of the set of instruments in the second image; and

[0207] Presentation of a notification is caused indicating that an instrument not depicted in the second image is missing from the set of instruments.

[0208] A twenty-eighth example provides the system according to the twenty-seventh example, wherein:

[0209] Determining that an instrument from the set of instruments depicted in the first image is not depicted in the second image includes:

[0210] accessing a reference model representing a reference shape of the instrument depicted in the first image;

[0211] accessing depth data representing a current shape of a proper subset of the set of instruments depicted in the second image; and

[0212] The reference shape of the instrument is compared to each current shape of a proper subset of the set of instruments.

[0213] A twenty-ninth example provides a system comprising:

[0214] one or more processors; and

[0215] A memory storing instructions that, when executed by at least one of the one or more processors, causes the system to perform operations including:

[0216] accessing a first image captured prior to initiation of a procedure and depicting a set of instruments available for use in the procedure;

[0217] accessing a second image captured after initiation of the procedure and depicting a subset of the instruments depicted in the first image;

[0218] determining, based on the first and second images, whether an instrument among the instruments depicted in the first image is used or unused in the procedure; and

[0219] Causes presentation of a notification indicating whether the instrument was used or not used in the procedure.

[0220] A 30th example provides the system according to the 29th example, wherein:

[0221] Determining whether a device was used or not used during a procedure includes:

[0222] optically discerning the absence of blood on the instrument depicted in the first image; and

[0223] The presence of blood on the instrument depicted in the second image is optically discerned.

[0224] The thirty-first example provides a carrier medium carrying machine-readable instructions, the instructions being used to control the machine to perform the operations (eg, method operations) performed in any one of the preceding examples.

Claims

1. A method comprising: accessing, by one or more processors of the machine, a first image captured prior to initiation of a procedure, the first image depicting a set of instruments capable of use during the procedure; accessing, by the one or more processors of the machine, a second image captured after initiation of the procedure, the second image depicting a subset of the set of instruments depicted in the first image; determining, by the one or more processors of the machine, whether an instrument from the set of instruments depicted in both the first image and the second image exhibits one or more optically detectable indications of use in the second image, whether the instrument is used in the procedure; and The one or more processors of the machine cause presentation of a notification indicating whether the instrument in the set of instruments is used in a procedure.

2. The method of claim 1, wherein the subset of the set of instruments is a proper subset of the set of instruments.

3. The method of claim 1 , wherein determining whether the device exhibits one or more optically detectable indications of use in the second image comprises determining whether the device moves from a first position within the vehicle depicted in the first image to a second position within the vehicle depicted in the second image.

4. The method of claim 1 , wherein determining whether the device exhibits one or more optically detectable indications of use in the second image comprises: optically discerning the absence of blood on the device depicted in the first image; as well as The presence of blood on the instrument depicted in the second image is optically discerned.

5. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising: accessing a first image captured prior to initiation of a procedure, the first image depicting a set of instruments capable of use during the procedure; accessing a second image captured after initiation of the procedure, the second image depicting a subset of the set of instruments depicted in the first image; determining whether an instrument in the set of instruments depicted in both the first image and the second image is used in the procedure based on whether the instrument exhibits one or more optically detectable indications of use in the second image; and Causing presentation of a notification indicating whether the instrument in the set of instruments is used in a procedure.

6. The non-transitory machine-readable storage medium of claim 5, wherein determining whether the instrument exhibits one or more optically detectable indications of use in the second image comprises determining whether the instrument moves from a first position within the vehicle depicted in the first image to a second position within the vehicle depicted in the second image.

7. The non-transitory machine-readable storage medium of claim 5, wherein determining whether the instrument exhibits one or more optically detectable indications of use in the second image comprises: optically discerning the absence of blood on the device depicted in the first image; as well as The presence of blood on the instrument depicted in the second image is optically discerned.

8. A system comprising: one or more processors; as well as A memory storing instructions that, when executed by at least one of the one or more processors, cause the system to perform operations including: accessing a first image captured prior to initiation of a procedure, the first image depicting a set of instruments capable of use during the procedure; accessing a second image captured after initiation of the procedure, the second image depicting a subset of the set of instruments depicted in the first image; determining whether an instrument in the set of instruments depicted in both the first image and the second image is used in the procedure based on whether the instrument exhibits one or more optically detectable indications of use in the second image; and Causing presentation of a notification indicating whether the instrument in the set of instruments is used in a procedure.

9. The system of claim 8, wherein determining whether the instrument exhibits one or more optically detectable indications of use in the second image comprises: optically discerning the absence of blood on the device depicted in the first image; as well as The presence of blood on the instrument depicted in the second image is optically discerned.

10. The system of claim 8, wherein determining whether the instrument exhibits one or more optically detectable indications of use in the second image comprises determining whether the instrument moves from a first position within the vehicle depicted in the first image to a second position within the vehicle depicted in the second image.

Images