How to track equipment

Abstract

The present invention provides a specialized machine that facilitates the monitoring of instruments (e.g., surgical instruments or other tools). [Solution] The machine accesses a first image captured before the start of the procedure, showing a set of instruments, and a second image captured after the start of the procedure, showing a proper subset of the set of instruments shown in the first image. From the first and second images, the machine can determine that an instrument in the set of instruments shown in the first image is not shown in the proper subset of the set of instruments in the second image, and can provide a notification indicating that an instrument not shown in the second image is missing. Alternatively or in addition, the machine can determine whether an instrument in the set of instruments was used in the procedure, and can provide a notification indicating that the instrument was used in the procedure.

Description

Technical Field

[0001] The subject matter disclosed herein generally relates to the technical field of dedicated machines that facilitate the monitoring of instruments (e.g., surgical instruments or other tools), and to techniques where such dedicated machines are improved compared to other dedicated machines that facilitate the monitoring of instruments. The dedicated machines include computerized variants configured by software of such dedicated machines, and improved forms of such variants. uments) or other tools), and to techniques where such dedicated machines are improved compared to other dedicated machines that facilitate the monitoring of instruments. The dedicated machines include computerized variants configured by software of such dedicated machines, and improved forms of such variants. are improved compared to other dedicated machines that facilitate the monitoring of instruments. The dedicated machines include computerized variants configured by software of such dedicated machines, and improved forms of such variants. variants configured by software of such dedicated machines, and improved forms of such variants.

[0002] [Related Application Data] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 968,538, filed on January 31, 2020, the content of which is incorporated herein by reference in its entirety.

Background Art

[0003] A collection of instruments (e.g., a collection of surgical tools) can be placed on a conveyance device (e.g., a trolley or cart) and transported to a performer (e.g., a surgeon) of a procedure (e.g., a medical procedure such as a surgical procedure) to be performed (e.g., on a patient). Not all instruments are used during the procedure (e.g., 30% - 80% of surgical instruments are not used), and it is useful to track which instruments are used and thus worth the time, effort, and cost of sterilization, and which instruments were not used. It is important that all instruments are present and accounted for after the procedure, whether used or not. For example, tracking surgical instruments during a medical procedure is important to ensure that such instruments do not remain on a conveyance device (e.g., a trolley or cart) and transported to a performer (e.g., a surgeon) of a procedure (e.g., a medical procedure such as a surgical procedure) to be performed (e.g., on a patient). Not all instruments are used during the procedure (e.g., 30% - 80% of surgical instruments are not used), and it is useful to track which instruments are used and thus worth the time, effort, and cost of sterilization, and which instruments were not used. It is important that all instruments are present and accounted for after the procedure, whether used or not. For example, tracking surgical instruments during a medical procedure is important to ensure that such instruments do not remain in the patient's body after the procedure. in the patient's body after the procedure. in the patient's body after the procedure. Whether used or not during the procedure, it is important that all instruments are present and accounted for after the procedure. For example, tracking surgical instruments during a medical procedure is important to ensure that such instruments do not remain ​​​​It is possible to limit or reduce the risks left behind due to inattention.

[0004] Several embodiments are shown in the accompanying drawings, not as an limitation but as an example. [Brief explanation of the drawing]

[0005] [Figure 1] This is a network diagram showing a network environment suitable for operating an instrument tracking machine, according to several exemplary embodiments. [Figure 2] This block diagram shows the components of a device suitable for use as an instrument tracking machine, according to several exemplary embodiments. [Figure 3] A flowchart illustrating the operation of a device when performing a method for tracking an instrument, according to several exemplary embodiments. [Figure 4] A flowchart illustrating the operation of a device when performing a method for tracking an instrument, according to several exemplary embodiments. [Figure 5] This flowchart shows the operation of the device when performing an alternative method of tracking the instrument, according to several exemplary embodiments. [Figure 6] This flowchart shows the operation of the device when performing an alternative method of tracking the instrument, according to several exemplary embodiments. [Figure 7] This is a screenshot showing an image of an instrument according to several exemplary embodiments, where a device configured by an app has added a bounding box indicating the instrument. [Figure 8] Images showing devices according to several exemplary embodiments, screenshots showing images in which a device configured by an app has added device counts for each image, both individually and by device type. [Figure 9]Images showing devices according to several exemplary embodiments, screenshots showing images in which a device configured by an app has added device counts for each image, both individually and by device type. [Figure 10] Images showing devices according to several exemplary embodiments, screenshots showing images in which a device configured by an app has added device counts for each image, both individually and by device type. [Figure 11] This is a block diagram of components of a machine, according to some exemplary embodiments, that can read instructions from a machine-readable medium and perform any one or more of the methodologies discussed herein. [Modes for carrying out the invention]

[0006] The exemplary method (e.g., algorithm) is for the detection, classification, identification, and tracking of an instrument or instrument. To facilitate other monitoring, the example system (e.g., configured with dedicated software) A specialized machine facilitates the detection, classification, identification, and tracking of instruments or other monitoring of instruments. It is composed of. The examples are merely representative of possible variations. Unless otherwise specified. The structure (for example, structural components such as modules) is optional, and the combinations are... It can be combined, or subdivided, and the operation (for example, procedure, algorithm, or The behavior of other functions may change within a sequence, or be combined or detailed. They may be divided into separate categories. The following explanation contains a great deal of specific details for illustrative purposes. This is described to provide a full understanding of various exemplary embodiments. However, this main It will be obvious to those skilled in the art that the problem can be implemented even without these specific details. ru.

[0007] Precise and accurate detection, classification, and identification of instruments can provide cost-effective management of instrument inventory , provide for health and safety (e.g., the health and safety of a patient undergoing a medical procedure), or both can be valuable goals. Instrument usage information can assist hospital management in further refining instrument trays to include only those instruments (e.g., surgical instruments that are likely to be used for a particular procedure, by a particular surgeon, or both). For such purposes, a machine (e.g., a device configured using suitable software such as a suitable app) functions as an instrument tracking machine and is configured to perform instrument detection, instrument classification, instrument identification, instrument tracking, or any suitable combination thereof on one or more instruments based on images captured before and after the start of a procedure (e.g., a medical procedure). As used herein, "instrument detection" refers to detecting that an instrument having an unspecified type and an unspecified identity is shown at a location within an image, "instrument classification" refers to identifying, recognizing, or otherwise obtaining the type of a detected instrument, and "instrument identification" refers to identifying, recognizing, or otherwise obtaining the identity of a particular individual instrument as compared to the identity of other individual instruments of the same type.

[0008] When configured in accordance with one or more of the exemplary systems and methods discussed in the exemplary present specification, the machine can also function as an instrument classifier configured to determine the type (e.g., classification or category) of each instrument (e.g., scissors or forceps) shown in an image (e.g., to count instances of each type of instrument), or a particular instrument (e.g.,​​​​​​​​​​​​​​​​​​​​ For example, the same scissors previously shown in a previous image or the same forceps previously shown in a previous image Identify certain individual objects (e.g., detect, classify, or both) such as It can also function as an object identifier configured to do so, or function as both of them In the case of surgical instruments, examples of instrument types include graspers (e.g., forceps), Clamps (e.g., occluders), needle holders (e.g., needle graspers), retractors, extensors, Cutters, endoscopes, suction tips, sealing devices, scopes, probes, and calipers exist.

[0009] Portable (e.g., mobile) handheld devices (e.g., smartphones configured by an app), portable cart-mounted devices or backpack-mounted devices, stationary machines (e.g., machines incorporated in a hospital operating room such as in a wall or ceiling), or regardless of being implemented as any suitable combination of them, this machine Therefore, it can distinguish different types of instruments, instances of different individual instruments, or both of them. In an exemplary situation involving a large collection of surgical instruments, this machine (e.g., a machine functioning as an instrument classifier) can act as an identifier to quickly find the corresponding types of those instruments by scanning several instruments in real time and operate. In an exemplary situation where inventory management is important, this machine can provide an instrument tracking function For example, many operating rooms in hospitals often face the problem of preventing any surgical instruments from being left inside a patient after the patient's surgical procedure. This is unfortunately

[0010] ​​​ In particular, this is a common problem in hospitals. To address this challenge and avoid such accidents... , deploy a machine (for example, a machine that functions as an object identifier) ​​before and after the commencement of the procedure. Identify surgical instruments (for example, before the start of the procedure and after the completion of the procedure), individually or by type. The count is taken, and all instruments that were present before the start of the procedure are gathered before closing the patient's treatment area. It is possible to determine whether or not.

[0011] According to certain exemplary embodiments of the systems and methods discussed herein, appropriately The configured machine accesses the first image captured before the start of the procedure. The image shows a set of instruments that can be used for the procedure. This machine is used after the procedure has started (for example, The second image, captured during the procedure, immediately before completion, or after completion, further... Access. The second image is a proper subset of the set of instruments shown in the first image. This shows (ubset). From these images, the machine is the set of instruments shown in the first image. It is determined that the instrument inside is not shown in the proper subset of the set of instruments in the second image. The machine is a collection of instruments, with the instruments shown in the first image but not in the second image. A notice will be presented indicating that the item is missing from the set.

[0012] According to some exemplary embodiments of the systems and methods discussed herein, The configured machine accesses the first image captured before the start of the procedure, and the first image The image shows a set of instruments that can be used for the procedure. This machine is used after the procedure has started (for example, The second image, captured during the procedure, immediately before completion, or after completion, is further enhanced. Accessing the second image, we see a subset of the set of instruments shown in the first image. From these images, the machine can determine that the instruments in the set of instruments shown in the first image are being used during the procedure (for example) If so, whether it was used or not (as part of the procedure) is shown in the first picture. The determination is made based on the first and second images. The machine then determines whether the instrument was used during the procedure. Alternatively, provide a notice indicating whether it was not used.

[0013] Figure 1 shows a suitable net for operating an instrument tracking machine according to several exemplary embodiments. This is a network diagram showing network environment 100. Network environment 100 is a database -115 and devices 130 and 150 (for example, as an instrument tracking machine) These are all connected to each other via network 190 so that they can communicate with one another. Base 115 is configured to function as a single server (e.g., Cloud 118). This can form all or part of a geographically dispersed collection of multiple machines, Udo is a network-based system 105 (for example, one or more network-based A cloud-based service configured to provide the service to devices 130 and 150. It can form all or part of a (device system). Database 115 and device S130 and S150 are, respectively, the whole or a part of, as described below with respect to Figure 11. This can be implemented in a dedicated (e.g., specialized) computer system.

[0014] Figure 1 also shows users 132 and 152. Both sides are human users (e.g., nurses or surgeons) and machine users (e.g., via a software program to interact with device 130 or 150. A configured computer), or any suitable combination thereof (for example, by machine) This can be a human being who is supported or a machine managed by a human being. User 132 This can be associated with device 130 and can be the user of device 130. For example Device 130 is a desktop computer belonging to user 132, a vehicle computer Data, home media systems (e.g., home theater systems or other home entertainment systems) (Information systems), tablet computers, navigation devices, portable devices Devices, smartphones, or wearable devices (e.g., smartwatches) This can be called smart glasses, smart clothing, or smart jewelry. Similarly, user 152 is associated with device 150, and the user of device 150 This can be done. For example, device 150 is a desktop belonging to user 152. Automotive computers, vehicle computers, home media systems (e.g., home theater systems) (System or other home entertainment system), tablet computer, navigation system Communication devices, portable media devices, smartphones, or wearable devices Chairs (for example, smartwatches, smart glasses, smart clothing, or smart It can be called (Tjewelry).

[0015] Any of the systems or machines shown in Figure 1 (e.g., databases and devices) are... The system or machine (e.g., a dedicated application, operating system, file One or more software programs of the form, middleware, or other software programs. (Composed or programmed by dedicated software such as modules) A dedicated (for example, special) that has been modified to perform one or more of the functions described in the details. It can also be a specialized or other non-conventional and non-general-purpose computer. , may include such computers, or, by law, such computers It can also be implemented in any of the methodologies described herein. A dedicated computer system capable of performing more than one of these actions will be discussed below with respect to Figure 11. And such a dedicated computer is therefore one of the methodologies discussed herein. It can be a means of performing any one or more of the following. Within the field of art, the structure described herein performs the function described herein. Specially modified by the creation (for example, configured by dedicated software) The computer is another dedicated computer or so which lacks the structure described herein. Compared to other dedicated computers that cannot perform the functions discussed herein in any other way. It has been technically improved. Therefore, according to the systems and methods described herein A specialized machine configured in this way provides an improved form of technology for similar specialized machines.

[0016] As used herein, "database" refers to a data storage resource, and various Any method, such as text files, tables, spreadsheets, relays. Object relational databases (e.g., object relational databases), triple Stores, hierarchical data stores, document databases, graph databases, key-value pairs A. It is possible to store structured data as any suitable combination thereof. Furthermore, any two or more of the systems or machines shown in Figure 1 are a single system. Alternatively, it can be combined with any single system or machine as described herein. The functions can be subdivided among multiple systems or machines.

[0017] Network 190 connects systems, machines, databases, and devices (for example, To enable any network that allows communication between machine 110 and device 130. This is possible. Therefore, network 190 is a wired network, wireless network ( For example, a mobile network or a cellular network, or any suitable combination thereof It can be combined. Network 190 is a private network, public A LIC network (e.g., the Internet), or any suitable combination thereof. It may include one or more parts that make up the network. Local area network (LAN), wide area network (WAN: Wide Area Network), Internet, Mobile Phone Network (for example) , cellular network), wired telephone network (for example, basic telephone service (POTS) :plain old telephone service) network), wireless data network (for example, WiFi network or WiMAX network), or any suitable combination thereof It can include one or more parts that incorporate the Wase. Any one of Network 190 The above parts can communicate information via a transmission medium. The "transmission medium" is a machine (for example, one or more processors of such a machine) Any intangible (e.g., temporary) capable of communicating (e.g., sending) instructions for a line (na) refers to a medium, and digital communication signals or analog communication signals or such software This includes other intangible media that facilitate communication.

[0018] Figure 2 shows several exemplary embodiments configured to function as an instrument tracking machine. This is a block diagram showing the components of device 130 when it is displayed. Access device 210, device recognition device 220, notification device 230, camera 240, depth sensor It is shown as including SA250, and all of these (for example, bus, shared memory, or They are configured to communicate with each other (via a switch). The image accessor 210 is Access module or similarly suitable software code for accessing one or more images. It can be done that way, or it can be made to include such things. Device Recognition Device 22 0 recognizes the device (for example, by type or as a specific individual instance). It can also be a recognition module or similarly suitable software code, and such It can also include things. The notifier 230 generates notifications and those notifications ( For example, on the display screen of device 130, the audio of device 130 is displayed. A notification module or similarly suitable software that presents (via or through a speaker) It can be toware code, or it can include such code.

[0019] Camera 240 is configured to capture one or more images (for example, digital photographs). It can be an image capture component, or it can include such a component. It is also possible to capture images using light data (e.g., RGB data or other color space data). Optical data in between, infrared data, ultraviolet data, ultrasonic data, radar data, and It can include any suitable combination of them, and it can also visualize them. It is possible. According to various exemplary embodiments, the camera 240 is located on the back of the mobile phone, This includes the front of the installed device, including the display screen, surgical tray, and scraper. A technician's table (for example, one located in the operating room), or an assembly workstation They are installed in (for example, in the assembly room of the equipment supplier) and directed toward them. This can be a set of one or more cameras, or any suitable combination thereof. The camera array or device 130 can be used to view a tray, surface, or table (for example, standing upright). App 200 merges data from multiple cameras that are capturing images (visually). It can be configured in this way. In some situations, device 130 is used in other parts of this specification. As discussed by the way, multiple images captured while moving are processed before (for example) Access via image stitching or structure from motion (stru Apply the motion-from-motion algorithm, or both, to perform instrument detection. , to support or enhance instrument classification, instrument identification, or any suitable combination thereof It is composed of Uni App 200.

[0020] The depth sensor 250 includes an infrared sensor, radar sensor, ultrasonic sensor, optical sensor, and time sensor. • Time-of-flight cameras, structured optical scanners, or similar devices It can be any suitable combination of these, or it may include such combinations. It is also possible. Therefore, the depth sensor 250 is within the range of the depth sensor 250 (for example, the field of view). Or one or more objects (e.g., instruments) within the detection area (e.g., one or more) It can be configured to generate depth data (representing the distance to the object above).

[0021] As shown in Figure 2, the image access device 210, the device recognition device 220, the notification device 230, or Any suitable combination of these may be all or It can form a part. App 200 (for example, data from device 130) Depending on whether it was received via network 190 or otherwise as a result of such reception (e.g., installed) on device 130 and executable on device 130 Furthermore, one or more processors 299 (e.g., hardware processor, digital processor) A tar processor, or any suitable combination thereof, is used for app 200, image access. The device 210, the device recognition device 220, the notification device 230, or any suitable combination thereof (For example, it may include temporary or permanent elements.)

[0022] Any one or more of the components (e.g., modules) described herein are H Perform this using only hardware (for example, one or more of the 299 processors). It can also be done using a combination of hardware and software. For example, any component described herein is a component of its constituent elements described herein. One or more of the 299 processors configured to perform basic operations (for example, Pro The apparatus may physically include a subset of sessa 299. As another example, see the Spec Any component described herein will perform the operation of that component as described herein. Software, hardware, and other components comprising one or more of the 299 sea urchin processors. Or it may include both. Therefore, different configurations described herein The element includes different devices of processor 299 at different points in time, and constitutes different devices. It is also possible to include a single device with 299 processors at different points in time, and a single device It can also be composed of: Each component (e.g., module) described herein is This is an example of means for performing the operation of its components as described herein. Any two or more components described in this book can be combined into a single component. Functions described herein as a single component may be subdivided among multiple components. This is possible. Furthermore, according to various exemplary embodiments, a single system or machine (for example, The components described herein as being implemented within a single device include multiple components. It can be distributed across a system or machine (e.g., multiple devices).

[0023] Figures 3 and 4 show the implementation of method 300 for tracking an instrument according to several exemplary embodiments. This is a flowchart showing the operation of device 130. The operation of method 300 is shown in the figure. Regarding point 2, the above-mentioned components (e.g., modules), one or more processors (e.g., A microprocessor or other hardware processor, or any suitable combination thereof This can be performed by device 130 using the combination. As shown in Figure 3, Method 300 includes operations 310, 320, 330, and 340.

[0024] In operation 310, the image accessor 210 accesses the first image captured before the start of the procedure. Accessing the image (for example, receiving, retrieving, reading the first image, or by other means) (To be acquired). The first image shows a set of instruments that can be used when performing the procedure (e.g., instruments). This shows the reference set of the components. For example, the first image is taken by the camera 240 of device 130. (For example, a set of surgical instruments arranged in preparation for a surgical procedure to be performed by a surgeon) It can be captured by taking a digital photograph of Ray. In the exemplary embodiment, the first image is accessed by the image accessor 210 and stored in database 1 Accessed from 15 via network 190. One or more reference markers are also in the first image. Such reference markers can be shown in the image, and such reference markers are used in the instrument classification performed in operation 330. This can serve as a basis for improving the effectiveness of instrument identification, or both.

[0025] In operation 320, the image access device 210 after the start of the procedure (for example, during the procedure, Access the second image captured immediately before or after the completion of the procedure. Image 2 shows a proper subset (e.g., a part) of the set of instruments shown in Image 1. For example, the second image is taken by the camera 240 of device 130 (for example, during a surgical procedure). After the procedure, before the surgeon closes the treatment site of the patient undergoing the surgical procedure, the first image To take a digital photograph of a surgical tray in which a portion of the set of instruments shown is arranged. Therefore, it can be captured. In some exemplary embodiments, the second image The image access device 210 accesses the database 115 via the network 190. Accessed. One or more reference markers may also be shown in the second image, and such references The marker is effective for instrument classification, instrument identification, or both, performed in operation 330. It can serve as a foundation for enhancing sexuality.

[0026] In operation 330, the instrument recognizer 220 identifies the instrument among the collection of instruments shown in the first image. It is determined that the object is not shown in the proper subset of the set of objects in the second image. According to one exemplary embodiment, the device recognition device 220 performs device classification and identifies a specific type It is determined that an unspecified instance of the ipu instrument is missing from the second image (for example, the Image 1 shows seven forceps, while Image 2 shows only six forceps. (Since it is not present, we conclude that one of the seven forceps is missing.) Implementation of a specific example According to the system, the device recognition device 220 performs device identification, and if a specific individual device matches the second image... It is determined that something is missing (for example, the first image shows a specific pair of scissors, but the second...) (It is determined that this is not shown in the image.) In a hybrid illustrative embodiment, the device The recognition unit 220 performs both instrument identification and instrument classification. It counts individual instruments with minimum confidence. In a further exemplary embodiment, such as when it is not possible to do so using a threshold, the device recognition device 220 , by performing centralized equipment detection and centralized equipment classification, a collection of equipment with shared types ( For example, a stack of clamps is the first in terms of volume, area, height, or other indicators of size. We determine that the image has changed from image 1 to image 2 (for example, it has decreased).

[0027] To perform equipment classification, the equipment recognition device 220 uses, for example, real-time computer visualization. Using the compositing technique, images (for example, images showing surgical instruments arranged in an instrument tray) can be processed. Artificial intelligence in the form of an instrument classifier trained to detect and classify the indicated instruments. A module (for example, to implement one or more computer vision algorithms) It can also be a trained artificial intelligence machine learning module, or such artificial It may also include an intelligent module. The instrument classifier has several convolutional layers. To use a deep convolutional neural network such as a neural network. It is also possible to include such deep convolutional neural networks. It can also do this. Deep convolutional neural networks have activation layers (for example, Softm It can have an ax activation layer on top and N different types (e.g., categories) There can be N possible outputs that predict the probability of the instrument. Therefore, the predicted type of the instrument shown The instrument classifier can then select the instrument type corresponding to the highest probability.

[0028] In some exemplary embodiments, the instrument classifier within the instrument recognition device 220 increases Based on a classification training model with multiple convolutional layers corresponding to the filter size (e.g.) For example, it is trained by a trainer machine. For example, it increases from 8 to 32. Depending on the filter size, the four convolutional layers are modified as their activation function. They may be present with the ning unit. After these convolutional layers, batch normal A bonding layer, a pooling layer, or both may follow. Therefore, the fully bonded layer is Along with Softmax activation, 14 types of equipment in training sets (examples) For example, it can contain 14 nodes representing categories. The classification training model Adaptive learning rate optimization algorithms (e.g., Adam) can be used, and the loss function Multiclass cross-entropy can be used as a method.

[0029] According to a particular exemplary embodiment, the training set is a base set representing a standard set of equipment. Quasi-images may include (for example, exclusively or non-exclusively), and such a standard set is: Specific procedures, surgeons, hospitals, instrument suppliers, geographical areas, or any suitable combination thereof It can be customized for specific combinations. Furthermore, the training sets are diverse. Reference image captured under lighting conditions, corresponding 3D data (e.g., the shown fixture) Reference image having depth data or model, reference carrier equipment (for example, even if empty) The reference image of the reference tray (which may also be filled with equipment), background items (for example) Reference images of towels, drapes, floor surfaces, or table surfaces, or any suitable combination thereof. It may include combinations.

[0030] To perform equipment identification, the equipment recognizer 220 positions the target object in a given image. In addition to determining the location, (for example, drawing a bounding box around the located device) Trainers can identify (by analyzing the contents inside the bounding box) It can also be an artificial intelligence module in the form of a sorted object classifier, and such a person It may also include an intelligent module. For example, this object classifier may include feature extraction. A single-shot detector (SSD) with Inception-V2 as the container. It can also be a single-shot detector, and such a single-shot detector It can also be included. However, the trade-off between accuracy and inference time must be balanced. To achieve this, other variations of the neural network architecture are used for object detection. Not only may it be suitable for classification or identification, but it may also be suitable for other classes of neural networks. In some cases, a different approach may be more suitable. One example training dataset is N( For example, N=10) different types (e.g., categories) and hundreds to thousands of images ( For example, it may include 227 images or 5000 images. The object classifier is Pasc Using al VOC metrics to evaluate bounding box metrics is possible. can.

[0031] In a particular exemplary embodiment, the object identifier within the device recognition device 220 is (for example, (To avoid problems associated with using excessively small datasets) Large synthetic datasets Training is performed using a taset. One example of a training procedure is using a trainer machine ( For example, a renderer machine that controls or functions as a renderer machine in 3D The scene is physically simulated, and known camera parameters are also simulated. It starts by doing a simulation. Next, the trainer machine randomizes 3D objects into the scene. Place objects on a surface (for example, randomly place 3D objects on a 3D tray), and provide object occlusion, lighting, and shadows. The scene is rendered based on various factors such as shadows. Then, the trainer machine, The system artificially captures images of rendered 3D objects within the scene. In this simulation, the location, orientation or orientation of 3D objects, lighting, and camera location are all considered. The number of applications and 3D objects is randomly changed (for example, via domain randomization). Furthermore, it automatically generates large and diverse composite image datasets. Simulate the data labels (e.g., bounding boxes and segmentation masks) During the process, it can be automatically generated by the trainer machine, thereby allowing the label to be generated. The cost of production will be reduced.

[0032] According to some exemplary embodiments, the trainer machine performs object identification within the instrument recognition device 220. The instrument is trained as follows: First, the trainer machine generates composite images of surgical instruments. Pre-train the object classifier using the prepared dataset (e.g., object recognition) (Train an object recognition model performed by the device.) This synthetic dataset is used After pre-training, the trainer machine displays small images showing specific types of surgical instruments. Object classifiers based on real-world (e.g., non-synthetic) datasets (e.g., further tray (By editing) it can be changed. This small real-world dataset was selected by humans (hu It can be man-cured. In a particular exemplary embodiment, the trainer machine A suitable alternative machine will train the object classifier.

[0033] Step 1: To generate a synthetic surgical instrument dataset, one or more trainer machines are required. rendering applications (e.g., Blender® or Unreal) The Gaming Engine (trademark) can be started or otherwise activated. To render the final image, the trainer machine accesses the following example inputs. It is possible. 1. 3D models of various surgical instruments (e.g., computer-aided design (CAD)) -aided design) model), 2. Surface texture information of surgical instruments, 3. Brightness range or spectral composition variation range, etc. (For example, simulating a hospital operating room) (The range of parameters that define lighting, 4. A surgical tray on which surgical instruments are placed to capture images from a specific angle. The range and arrangement of possible camera locations (e.g., azimuth, elevation, pan, tilt, etc.) to 5. The number and type of fixtures to be rendered.

[0034] In a particular exemplary embodiment, the trainer machine generates a random number of virtual surgical instruments. The trainer machine then artificially places the patient in any orientation and position on the virtual surgical tray. Generates a composite image showing various amounts of occlusion ranging from occlusion to significant occlusion. The amount of shielding in this simulation is a customized parameter. It can be done this way.

[0035] Step 2: The trainer machine is generated as a training dataset in Step 1. The object classifier is trained using the synthesized surgical instrument dataset. ru.

[0036] Step 3: The trainer machine displays realistic surgical instruments naturally placed on a real surgical tray. Access a small dataset of real images that demonstrate this (for example, from database 115). The dataset of real images is used between the use of synthetic images and the use of real images (for example). Training an object classifier using only a large number of composite images versus using only a small number of real images This helps to bridge the gap between the training of the object classifier and the actual training.

[0037] In some embodiments, one of the instruments is either a transport device for the instruments (e.g., a surgical tray) or one of the instruments themselves. Using the above reference markers, you can perform instrument detection, instrument classification, instrument identification, or any other suitable method. It can assist in combinations such as specialized orthopedic trays. In this case, the instrument recognition device 220 will indicate an empty orthopedic tray in which no instruments are placed. You can access a sample image (e.g., a mask image), and then the first image By subtracting the template image, the first image more clearly shows the individual instruments before the start of the procedure. A difference image (for example, the first segmentation image) can be obtained. Similarly, the instrument The recognition unit 220 subtracts the template image from the second image, after the start of the procedure (for example) A second difference image (for example, one that more clearly shows the individual instruments at the end or near the end of the procedure) A second segmentation image can be obtained. The first difference image and the second difference image The image is used for operation 340 or instrument detection, instrument classification, instrument identification, or any suitable combination thereof. The device recognition device 220 can be used to prepare for the implementation of alternatives to the mismatch. In this sense, the orthopedic tray is a reference in the first image, the second image, or both. It fulfills its function as a cog.

[0038] In a particular exemplary embodiment, multiple independent classifiers (e.g., deep learning) The outputs of a 3D classifier, a difference image classifier, or any suitable combination thereof are combined. And, with respect to a given transport device (e.g., a tray) for the equipment, equipment classification, equipment identification, or so The accuracy, precision, or both of these will be improved when performing both. Specifically, independent The algorithm determines (1) whether the tray is complete or incomplete, and (2) whether the tray - Each of the predetermined template areas within the multiple predetermined template areas is filled. (3) Whether or not the requirements are met, and what the classification of each object detected on the tray is. (For example, whether the classification is an instrument, and if so, what type of instrument) It is possible to find and output the corresponding probability that indicates (or). These three independent elements The Gorism assembly checks whether the tray is indeed complete, and if not, which instrument This can better indicate whether something might be missing.

[0039] In operation 340, the notifier 230 indicates that the equipment not shown in the second image is part of the equipment collection. Provide a notification (e.g., visual, auditory, or both) indicating that something is missing. This notification may be presented in the following forms: a pop-up window, a warning sound, etc. Playback, messages to another device (e.g., a nurse's or surgeon's smartphone) Transmission, triggering a predetermined action in response to the device being deemed missing (e.g., device Procedures to identify or check patients, or any suitable combination thereof It can take the form of...

[0040] According to various exemplary embodiments, the presented notice indicates whether a particular instrument is missing. Alternatively, or in addition, the presented notice is a carrier for the assembly of equipment shown in the first image. The container (e.g., tray or cart) (e.g., standard surgical tray for instruments, instrument claw) It is complete (compared to a standard set of instruments such as orthopedic trays or other instruments). It may indicate that the notice is either complete or incomplete. Alternatively, or in addition, the presented notice may be A standardized report listing each instrument in the set of instruments shown in the first image, that instrument Whether or not it was used, and when the device was picked up (for example, timestamp and (and) when the instrument was returned (for example, to the scrub technician or transport equipment), the patient The duration of the device's stay in the body, whether the device has been removed or is still remaining in the body. Dolphins, or any suitable combination thereof, corresponding indicators (e.g., markers) It may be included with (or flag). In some exemplary embodiments, presented The notification included the total number of missing devices and a list of the missing devices (for example, by type). (represented as specific individual instruments, or both thereof), or any suitable combination thereof Includes combinations.

[0041] Furthermore, verify or correct the total number of devices presented, whether they exist or not. The user 132 is prompted, and the user 132's response triggers one or more artificial intelligences in the device recognition device 220. Userfiles are used as labels to further train and improve the module. The feedback function can be implemented by app 200. Several example implementations In this state, app 200 operates using more user interaction. This allows you to view some or all of the information contained in the presented notice (for example, manually or visually). User 132 may be prompted to confirm (or both).

[0042] As shown in Figure 4, for method 300, in addition to any one or more of the operations described above Method 300 is for operations 410, 412, 422, 430, 431, 434, 435, 43 It may include one or more of 6 and 437.

[0043] Operation 410 is part of operation 310 in which the image accessor 210 accesses the first image. For example, it can be executed as a preceding task, subroutine, or part of something. In 410, the first image is the treatment (for example, by specifying the treatment). To correspond to, or (for example, by being designated by the person performing the action) the person performing the action (for example) A standard set of instruments (e.g., standard) that corresponds to a surgeon, or to both. This is a reference image showing a collection of standardized instruments, and this reference image is used to indicate the response to the procedure, the procedure Based on the response to the person who carries out the action, or to both of them (for example, depending on the response) Accessed

[0044] In an alternative exemplary embodiment, operation 412 is performed as part of operation 310. This is possible. In operation 412, the first image is a sequence of frames (for example, video). As part of capturing the first image (first sequence of the first frame), It is accessed by capturing. For example, the image access device 210 is accessed by the device S130 is moving over an instrument assembly (for example, on a surgical tray that houses an instrument assembly) While passing through, access video data from camera 240, and the first image is video A sequence of deoframes can be recorded. In such an exemplary embodiment, App 200 infers depth data from a sequence of video frames containing the first image. A stereoscopic viewing algorithm configured to achieve this (for example, structure from motion algorithm) This depth data includes (Gorhythm) and can be executed, and the instrument is missing and the operation This can be the basis or other factor in the judgment under paragraph 330.

[0045] Similarly, in a particular exemplary embodiment, operation 422 is performed by the image accessor 210 This can be performed as part of operation 320, which accesses the second image. In this case, the second image is a sequence of frames (for example, the second of the second frame of the video). By capturing a second image as part of capturing the sequence of images It is accessed in this way. For example, the image access device 210 accesses the collection of devices 130 Moving over a portion (for example, passing over a surgical tray that contains a portion of an instrument assembly) While (this is happening), access video data from camera 240 and the second image, video file The frame sequence can be recorded. In such an exemplary embodiment, the app The R200 infers depth data from a sequence of video frames that include a second image. It includes a stereoscopic viewing algorithm configured in such a way that it can be executed, and this depth data is used in the This can be used as the basis or other factor in determining whether an ingredient is missing in operation 330. Cut.

[0046] As shown in Figure 4, operations 430 and 431 are performed in the set of instruments shown in the first image. If the instrument is not shown in the proper subset of the set of instruments in the second image, the instrument recognizer 220 This can be performed as part of the decision-making action 330.

[0047] In operation 430, the instrument recognizer 220 determines the shape of the instrument in the first image (for example, deep Recognizes (optically) with or without supplementary support from degree data. For example, as described above. As described above, the instrument recognition device 220 is an instrument classifier, an object identifier, or a combination of both. It can be a combination of both, or it can include them, and the device recognition device 220 is Therefore, the device is detected (e.g., identified) by its shape as shown in the first image. They can also be trained to classify things.

[0048] In operation 431, the instrument recognizer 220 determines the shape of the instrument in the second image (for example, depth It attempts to recognize (optically) with or without supplementary support from degree data, It cannot be recognized. For example, as mentioned above, the device recognition device 220 is a device classifier, It can be an object classifier, or a combination of both, or something that includes them. It is also possible for the instrument recognizer 220 to recognize the instrument as shown in the second image. It can be trained to detect (e.g., identify) and classify based on its shape. Yes, it is possible. However, since the device is not shown in the second image, the device recognition device 220 The system fails to detect or classify this device.

[0049] As shown in Figure 4, operations 434 and 435 are performed in the set of instruments shown in the first image. If the instrument is not shown in a proper subset of the set of instruments in the second image, the instrument recognizer 220 It can be executed as part of the action 330 that is determined by [the system].

[0050] In operation 434, the device recognition unit 220 accesses the reference model of the device. The quasi-model can be 3D and can be accessed from database 115. For example, when operation 430 is performed, the device recognition device 220 will perform operation 430. Based on the identification of the fixture by its shape (for example, according to the identification), an action is taken to the reference model of the fixture. It can be done.

[0051] In operation 435, the device recognition device 220 uses a reference model of the device in the second image. For example, each of the multiple silhouettes (accessed in operation 434) (example For example, attempting to recognize (optically) with or without supplementary support from depth data It can see, but it cannot recognize. For example, the device recognition device 220 can see the silhouette from the reference model. A set of silhouettes can be generated, and each silhouette in this set of silhouettes can be used in a second drawing. The shapes of the instruments can be compared to those within a proper subset of the set of instruments shown in the image. However, since the device is not shown in the second image, the device recognition unit 220 will use the second image. The system fails to recognize any of the reference model silhouettes of the equipment in the image.

[0052] As shown in Figure 4, operations 436 and 437 are performed in the set of instruments shown in the first image. If the instrument is not shown in a proper subset of the set of instruments in the second image, the instrument recognizer 220 It can be executed as part of the action 330 that is determined by [the system].

[0053] In operation 436, the instrument recognizer 220 recognizes the true portion of the set of instruments shown in the second image. Access depth data that represents the current shape of the set. For example, the depth data is for device 1 30 depth sensors 250 can capture, and the equipment recognition device 220 can capture depth Depth data from sensor 250 can be accessed.

[0054] In operation 437, the device recognition device 220 determines the reference shape of the device as a true subset of the set of devices. Compare each of the current shapes of the combination. As mentioned above, the current shape is in operation 436. It can be represented by the depth data accessed. Operation 434 is performed in advance. If a reference model representing the reference shape of the device has been accessed, in operation 437 The same reference shape can be used for the comparisons made here. In this case, operation 437 is performed by the instrument (for example, in the same manner as described above for operation 434). This includes accessing or obtaining the standard shape of the component by other means.

[0055] Figures 5 and 6 show the implementation of method 500 for tracking an instrument according to several exemplary embodiments. This is a flowchart showing the operation of device 130 during the process. Operation in method 500 This refers to the components described above in Figure 2 (e.g., modules), and one or more processors (e.g., For example, a microprocessor or other hardware processor, or any suitable one thereof This combination can be executed by device 130, as shown in Figure 5. Method 500 includes operations 510, 520, 530, and 540.

[0056] In operation 510, the image accessor 210 accesses the first image captured before the start of the procedure. Accessing the image (for example, receiving, retrieving, reading the first image, or by other means) (To be acquired). The first image shows a set of instruments that can be used when performing the procedure. For example For example, the first image is captured by the camera 240 of device 130 (for example, a collection of surgical instruments). Digital photographs are taken of surgical trays arranged in preparation for surgical procedures performed by surgeons. (by doing so) can be captured. In some exemplary embodiments, the Image 1 is accessed from database 115 to network 190 by image access device 210. It is accessed via. One or more reference markers can also be shown in the first image, and so The reference marker serves as the basis for improving the effectiveness of instrument identification performed in operation 530. This is possible. In various exemplary embodiments, operation 510 is performed in the manner described above. It is executed in the same way as operation 310 in 0.

[0057] In operation 520, the image accessor 210 captures the second image after the start of the procedure. Access the image. The second image is a subset of the set of instruments shown in the first image (e.g., For example, it represents a part of the set of instruments. This can be a subset that coincides with the entire set of instruments. That is, there is a missing part in the second image. There may be cases where the necessary equipment is not available. For example, the second image is taken by the camera 240 of device 130. Therefore (for example, after the start of a surgical procedure, and the treatment area of ​​the patient being treated is removed) Before the doctor closes, the surgical tray on which a portion of the set of instruments shown in the first image is placed. This can be captured (by taking a digital photograph). Some examples include In one embodiment, the second image is accessed by the image accessor 210 from the database 115. It is accessed via network 190. One or more reference markers are also shown in the second image. Such reference markers can be used to enable instrument identification performed in operation 530. It can serve as a foundation for enhancing sexuality.

[0058] In operation 530, the instrument recognizer 220 identifies the instrument among the collection of instruments shown in the first image. The first and second images show whether the instrument was used or not in the procedure. The determination is made based on the following. According to some exemplary embodiments, the instrument recognition device 220 recognizes the instrument By making a distinction, it is determined that specific individual devices are present in both images, but one or more lights This indicates a scientifically detectable indication of use. Such indication is, for example, the movement in the first image. Transport equipment in a transport device (e.g., surgical tray) from a first location to a second image of the transport device. Move to the second location within the first image, bioburden (e.g., patient blood, etc.) in the first image. From a state where one or more spots of fluid are absent, to a state where a bioburden is present in the second image. This includes changes in appearance to a certain state, or any suitable combination thereof. Apparatus identification (for example) The algorithm used by the instrument recognition device 220 when performing the action (via the object recognition device) Details of the example and details of the training example for the device recognition device 220 (e.g., object classifier) This is discussed above (for example, with respect to the operation 330 in method 300). , one or more reference markers in the first image, the second image, or both thereof, as described above. It can be used in the same way.

[0059] In operation 540, the notifier 230 indicates whether the instrument was used in the procedure or not. Provide a notification (e.g., visually, audibly, or both) indicating why it could not be received. The notification may take the form of displaying a pop-up window, playing a warning sound, etc. To another device (for example, a nurse's, surgeon's, janitor's, or inventory manager's smartphone) Sending a message, and responding to the determined used or unused state of the equipment. Triggers for specific actions (e.g., used instrument counting action, unused instrument counting action, or It can take the form of (or instrument sterilization), or any suitable combination thereof. .

[0060] Furthermore, in a particular exemplary embodiment, operations 510 and 530 are operations 520 and It can be performed without one or both of the above, and thereby as shown in the first image. The instruments are directly classified, identified, or both, and as a result, the classification, identification, and Alternatively, the count of instruments, the type of instrument, the name of the instrument, and the standards for the instrument, all of which have been performed. A notice will be presented showing an image, or any suitable combination thereof. The embodiment allows a novice scrub technician or novice surgeon to recall the name of the instrument. It can be useful in situations where it is impossible or unknown. The machine allows for quick classification of instruments. To identify, a novice scrub technician or novice surgeon can use the camera on device 130. The device can be held in front of 240, and app 200 is related to (for example, operation 330) (And as described above, by performing instrument classification, instrument identification, or both.) Answers can be obtained using computer vision and deep learning. For these responses, a likelihood score indicating the confidence level of the response can be assigned. In some embodiments, a camera (for example, camera 240) takes an image of the table. The instrument tray assembler positions the instrument on the table and the device (for example) Then, a device 130) as described herein scans the instrument, and the scanned Installed supply chains that perform automatic classification, automatic identification, or both of these functions on equipment. It is positioned in the setting.

[0061] Furthermore, in various exemplary embodiments, the improvement of supply chain efficiency is one or more The above is an extremely manual process (for example, when an instrument is manually added to a new tray, (Checking a standardized electronic checklist) is the system discussed herein. As can be obtained by replacing it with an automated checklist based on the method, Operations 510 and 530 shall be performed without one or both of operations 520 and 540. This is possible. In such exemplary embodiments, the tracer assembler grasps the instrument, (e.g.) For example, the instrument can be imaged using one or more of the many modalities, and the device The chair (for example, device 130) enumerates the equipment to be added to a new tray (for example, (t) Automatically place a check mark next to the device on the assembly sheet.

[0062] Furthermore, in some exemplary embodiments, for example, each instrument is removed from the tray When and how they were removed, and whether each removed instrument was returned to the tray. Was it returned? And do the removed instruments appear used or unused? To track this, operations 510 and 530 are repeated so that a visual record is generated. and are carried out. All or part of such visual records (for example, the patient receiving treatment) (corresponding to the person) to be included in the electronic medical record (for example, by device 130 in the database) It can be provided to -115.

[0063] Furthermore, the automated classification or identification of instruments discussed herein is found in hospital operating rooms. It offers similar advantages for any other consumable items and for other settings. Therefore, it can extend beyond the equipment on the tray. For example, medications in a dispensing cart (For example, to ensure that controlled substances such as opioids are not misused or lost during surgical procedures.) (In order to) track the instruments on the tray in the same manner as described herein. This is possible. Therefore, in some exemplary embodiments, an app (for example, App 2) The drug is scanned by a device (e.g., device 130) configured by (00). When each drug is administered by an anesthesiologist, the device can be used in a medical cart. A notice showing all or part of the visual record may be presented. The visual record includes each drug (For example, how each controlled substance was administered or used in another way) This can be shown along with the corresponding timestamp for administration or other use.

[0064] As shown in Figure 6, for method 500, in addition to any one or more of the operations described above Method 500 may include one or more of operations 630, 632, and 634. One or more of operations 630, 632, and 634 are performed by the apparatus shown in the first image. The device recognition device 220 determines whether the device was used or not in the installation based on the first image and This can be performed as part of an action 530 that makes a decision based on a second image.

[0065] In operation 630, as part of determining whether or not the device was used, device recognition is performed. The device 220 is moved from the first position in the transport equipment shown in the first image to the position shown in the second image. Determine if it has moved to a second position within the transport equipment.

[0066] In operation 632, as part of determining whether or not the device was used, device recognition is performed. The device 220 is used to detect bioburden (e.g., bloodstains or spots of other bodily fluids from the patient) It is recognized (for example, optically) that the instrument shown in the image is not present.

[0067] In operation 634, as part of determining whether or not the device was used, device recognition is performed. The device 220 is a bioburden (for example, one or more bloodstains or spots of other bodily fluids from the patient). (T) recognize (e.g., optically) that it is in the same instrument as shown in the second image. do.

[0068] According to various exemplary embodiments, one or more of the methodologies described herein are instruments Furthermore, as described herein, One or more of the methodologies involve the detection and quantification of each type of instrument, and the detection of individual instruments. Tracking, or both, can be facilitated. Therefore, as described herein One or more of these methodologies, compared to the capabilities of existing systems and methods, have equipment inventory and More precise and accurate management of associated costs for those maintenance procedures (e.g., sterilization procedures) and , facilitating the reduction of risks to the health and safety of (for example, patients undergoing medical procedures) It is possible.

[0069] Figure 7 is an image showing an apparatus according to several exemplary embodiments, and is used in app 200. The resulting device 130 is shown in an image with a bounding box indicating the instrument. This is a screenshot showing the result.

[0070] Figures 8 to 10 are images showing the apparatus according to several exemplary embodiments, and the app The device 130, composed of 200, counts the amount of the instrument for each image. This is a screenshot showing images added individually and by type of device.

[0071] As shown in Figures 7 to 10, when the device is captured by camera 240, Pre200 displays the type of those fixtures (e.g., class name) and images (e.g., currently displayed). Along with a count of all instruments identified within the video frame, the number of those instruments To display a bounding box around the device, app 200 uses the device recognition device 220 ( For example, while the object classifier is running continuously, the camera 240 can be used to view the environment. Device 130 can be configured to scan in real time.

[0072] According to the techniques described herein, app 200 uses an instrument recognition algorithm (for example) As described above, any suitable device (embodied within the device recognition device 220) A device (for example, device 130) can be configured. App 200 is App 2 Activate any of the multiple operating modes of device 00, device 130, or both of them. The ability to do so can be given to user 132. For example, such a mode of operation is to Operating room fully featured mode (for example, all functions are enabled) (Full mode), Operating room partially featured mode (For example, the most computationally intensive feature is disabled or not installed.) Modes), supply chain mode, post-surgery quality control mode (OL mode), orthopedic sales mode, or any of the following appropriate modes: This can include the following combinations.

[0073] Any one or more of the algorithms described above for instrument classification or image recognition may differ. It can be independently applied to different use cases in various situations. In addition, these Any one or more of the algorithms can be multiple parts of the tool (e.g., the tip of the scissors, the container). It can be applied to the handles of tools or the supports of tools, and therefore, tool classification and tool identification. Partial classification, partial identification, or both thereof, in a manner similar to the methods described herein. It is possible to do either one or both. Therefore, various examples of the device recognition device 220 , additional artificial intelligence modules trained on the device part (for example, one or more) This additional artificial intelligence module may include (having a deep learning network) The guide is trained on the entire instrument, the entire tray, or both. Support for classification, identification, or both performed by an artificial intelligence module (for example) It can be used to confirm, verify, or change something.

[0074] When these effects are considered as a whole, one of the methodologies described herein The above describes the specific effort or resources required for tracking equipment when these methodologies are not used. This eliminates the need for a source. The effort users spend tracking equipment is reduced. By using a dedicated machine to implement one or more of the methodologies described in the specification (for example) (By relying on specialized machinery) it can be reduced. One or more systems or machines Computational resources used by machines (for example, within a network environment 100) are also (for example) If a system or machine lacks the structures discussed herein, or otherwise... (Compared to systems or machines that are unable to perform the functions discussed herein) Similarly, it can be reduced. An example of such computing resources is the processor cycle. Network traffic, computing power, main memory usage, graphics rendering Ring capacity, graphics memory usage, data storage capacity, power consumption, and cooling capacity be.

[0075] Figure 11 shows a machine-readable medium 1122 (e.g., a non-temporary machine-readable medium, a machine-readable storage medium). Instruction 1124 (from a computer-readable storage medium, or any suitable combination thereof) Read the text and use any one or more of the methodologies discussed herein, either in whole or in part. Components of machine 1100 according to several exemplary embodiments can be performed in this manner. This is a block diagram. Specifically, Figure 11 shows one of the methodologies discussed herein. Instructions 1124 that cause machine 1100 to execute one or more of the following (for example, software, program (RAM, application, applet, app, or other executable code) Exemplary forms of computer systems that can be run as a whole or in part (e.g.) This shows machine 1100 in a computer.

[0076] In an alternative embodiment, machine 1100 operates as a stanza lone device. Alternatively, it may be connected to other machines in a communicative manner (for example, networked). When deployed in a networked configuration, machine 1100 acts as a server-client network. It can also operate as either a server or client machine in a network environment. And it operates as a peer machine in a distributed (e.g., peer-to-peer) network environment. It can also be done. Machine 1100 is a server computer, client computer, personal Personal computers (PCs), tablet computers, laptops Computers, netbooks, cellular phones, smartphones, set-top boxes ( STB (set-top box), Personal Digital Assistant (PDA) (assistance), web equipment, network routers, network switches, network The instructions 1124 that specify the actions to be performed by the bridge or machine are executed sequentially or by other means. It can be any machine capable of doing so. Furthermore, only a single machine is shown. However, the term “machine” means any one or more of the methodologies discussed herein or This also includes any set of machines that individually or collectively execute instruction 1124, which is a partial execution of the instruction. It shall be interpreted as follows.

[0077] Machine 1100 includes a processor 1102 (for example, one or more central processing units (CPU: ce (Intral processing unit), one or more graphics processing units (GPU: graphics proc (sending unit), one or more digital signal processors (DSP: digital signal processor) r) one or more application-specific integrated circuits (ASICs) IRCUI, one or more radio-frequency integrated circuits (RFICs) cuit), or any suitable combination thereof, main memory 1104, and stat It includes a memory 1106 and these are configured to communicate with each other via a bus 1108. Processor 1102 performs any one or more of the methodologies described herein. The processor 1102 is configured to perform the whole or in part. , solid states that can be configured temporarily or permanently by part or all of instruction 1124 Includes digital microcircuits (e.g., electronic, optical, or both). For example, processor A collection of one or more microcircuits of the 1102 is one or more modules described herein. It can be configured to run a `route` (for example, a software module). In some exemplary embodiments, the processor 1102 has multiple cores, each of them However, if any one or more of the methodologies discussed herein are implemented in whole or in part A multi-core CPU that can behave as separate processors (for example, dual This refers to a single-core CPU, quad-core CPU, 8-core CPU, or 128-core CPU. The beneficial effects described in the document apply to at least the machine 1100 having processor 1102. Therefore, these same beneficial effects can be provided, but different processors do not have the same beneficial effects. A type of machine (for example, a purely mechanical system, a purely hydraulic system, or a hybrid) If a mechanical hydraulic system of a lid is configured to perform one or more of the methodologies described herein, such a processorless machine can also provide it.

[0078] Machine 1100 can further include a graphics display 1110 (e.g., a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, a cathode ray tube (CRT), or any other display capable of displaying graphics or video). Machine 1100 can further include an alphanumeric input device 1112 (e.g., a keyboard or keypad), a pointer input device 1114 (e.g., a mouse, touchpad, touch screen, trackball, joystick, stylus, motion sensor, eye tracking device, data glove, or other pointing instrument), a data storage device 1116, an audio generation device 1118 (e.g., a sound card, amplifier, speaker, headset jack, or any suitable combination thereof), and a network interface device 1120.

[0079] The data storage device 1116 (e.g., a data storage device) includes a machine-readable medium 1122 (e.g., a tangible non-transitory machine-readable storage medium) that stores instructions 1124 embodying any one or more of the methodologies or functions described herein. The instructions 1124 cause the machine 1 Before or during the execution of those by 100, static memory in main memory 1104 Within 1106, within processor 1102 (for example, within the processor's cache memory), and They can be present, fully or at least partially, in any suitable combination of them. Therefore, the main memory 1104, static memory 1106, and processor 1102 can be considered a machine-readable medium (e.g., a tangible, non-temporary machine-readable medium). Instruction 1124 is to connect to the network via the network interface device 1120. It can be transmitted or received on the 190. For example, network interface device IS1120 supports any one or more transfer protocols (e.g., hypertext transfer protocol). Instruction 1124 is communicated using Tocol (HTTP: hypertext transfer protocol). It is possible.

[0080] In some exemplary embodiments, the machine 1100 is a portable computing device (for example) (e.g., a smartphone, tablet computer, or wearable device) It can have one or more additional input components 1130 (e.g., sensors or gauges). This is possible. Examples of such input components 1130 include image input components (for example). (, one or more cameras), audio input components (e.g., one or more microphones) , Direction input component (e.g., compass), Location input component (e.g., global) Positioning system (GPS: global positioning system) receiver), azimuth component (for example) , gyroscope), motion detection component (e.g., one or more accelerometers), altitude detection component Components (e.g., altimeter), temperature input components (e.g., thermometer), and gas detection components There is an element (for example, a gas sensor). Any one of these input components 1130 The input data collected above includes (for example, user preferences, applicable regulations, or other information). Opt-in consent or opt-out consent to be implemented in any suitable combination thereof. Among the modules described herein (using appropriate privacy notices and protections, etc.) It can be made accessible and usable by any of the following.

[0081] As used herein, the term “memory” means the temporary or permanent storage of data. This refers to a machine-readable medium that can perform random access memory (RAM). Memory, read-only memory (ROM), buffer memory, flash memory This can be interpreted as including cache memory and data memory. However, this It is not limited to these. In one exemplary embodiment, the machine-readable medium 1122 is simply Although presented as one medium, the term "machine-readable medium" refers to a medium capable of storing instructions. A single medium or multiple mediums (for example, a centralized database or a distributed database, The term "machine-enabled" should be interpreted as including (or related caches and servers). The "reading medium" is an instruction 1124 that is executed by one or more processors of machine 1100 (for example, processor When performed by (1102), any one of the methodologies described herein may be used. The above is performed by machine 1100, either entirely or partially. Any medium capable of carrying (for example, storing or communicating) the command 1124 to be executed, shall be construed to include combinations of multiple media. Thus, "machine-readable" "media" refers not only to a single storage device or storage device, but also to a cloud-based storage system or storage network including multiple storage devices or storage devices. The term "machine-readable medium" is thus construed to include, but is not limited to, exemplary forms of solid state memory chips, optical disks, magnetic disks, or any suitable combination thereof, including one or more tangible non-transitory data repositories (e.g., data volumes).

[0082] "Non-transitory" machine-readable media, as used herein, specifically excludes the propagation signal itself. According to various exemplary embodiments, instructions 1124 executed by machine 1100 can communicate via a carrier medium (e.g., a machine-readable carrier medium). Examples of such carrier media include non-transitory carrier media (e.g., non-transitory machine-readable storage media such as solid state memory physically movable from one location to another)

[0083] and transient carrier media (e.g., a carrier wave or other propagation signal communicating instructions 1124). Certain exemplary embodiments are described herein as including modules. A module can comprise a software module (e.g., code stored in or otherwise embodied on a machine-readable medium or transmission medium), a hardware A collection of one or more processors, configured or arranged in a particular physical manner. This is possible. In various exemplary embodiments, one or more computer systems or similar One or more hardware modules are described herein with respect to that module. Software as a hardware module that operates to perform an action (for example, It can consist of an application or a part thereof.

[0084] In some exemplary embodiments, the hardware module is mechanically, electronically It can be carried out by hydraulic means, or by any suitable combination thereof. For example, A software module is a dedicated circuit permanently configured to perform a specific operation. It may include similar or dedicated logic. The hardware module is a field programmer. Field programmable gate array (FPGA) or ASIC, etc. It may be a dedicated processor, or it may include such a dedicated processor. It is also possible. Hardware modules can be programmed with software to perform certain actions. Therefore, this also includes temporarily configured programmable logic or programmable circuits. This is possible. For example, a hardware module can be a CPU or other programmable programmer. The cessor can include software. The hardware module is mechanical. Whether to implement it using a hydraulic system, or in a dedicated, permanently configured circuit. or circuits configured temporarily (for example, by software) The decision of whether or not to implement it will be made by considering the cost and time involved. It will be understood.

[0085] Therefore, the term "hardware module" is used to describe a device that operates in a particular way. It is physically and permanently configured to perform certain operations as described herein. For example, hardwired, or temporarily configured (for example, programmed) It should be understood that this includes tangible entities. Furthermore, in this specification The phrase "hardware-implemented module" used in this context refers to a hardware module. Examples of hardware modules being temporarily configured (e.g., programmed). When considering the embodiment shown, each of the hardware modules is at any given time It does not need to be configured or instantiated. For example, a hardware module does not need to be dedicated This C includes a CPU that has been configured by software to be a processor. PUs are included in different hardware modules at different times (for example, in different hardware modules). Each can be configured as a different dedicated processor. Software (for example, software A wear module is therefore, for example, a specific hardware module at a certain point in time. Configure a specific hardware module in a way that is either void or otherwise, at different points in time. This will result in different hardware modules or different hardware modules in another way One or more processors can be configured to form a single unit.

[0086] Hardware modules provide information to other hardware modules, and other hardware Information can be received from the wear module. Therefore, the hardware described A module can be considered to be connected in a communicative manner. When multiple modules are present simultaneously, communication occurs among those hardware modules. This can be achieved through signal transmission between two or more entities (e.g., via circuits and buses). Embodiments in which multiple hardware modules are configured or instantiated at different times. In such a case, communication between such hardware modules is, for example, between multiple hardware modules. Through the storage and retrieval of information for memory structures accessed by the software module This can be achieved. For example, one hardware module performs the operation, and The output of the operation is sent to a memory to which the hardware module is communicatively connected (for example, It can be stored in a memory device. Another hardware module then... At that point in time, it is possible to access memory, retrieve the stored output, and process it. The hardware module also initiates communication with the input or output device. This allows for the processing of resources (for example, a collection of information from computing resources). ru.

[0087] The various operations of the exemplary methods described herein are such that they perform the related operations (e.g. For example, one or more (configured temporarily or permanently by software) It can be executed at least partially by the processor. It can be configured temporarily or Whether or not it is permanently configured, such a processor is one of the processors described herein. A processor-implemented module is configured to perform the above operations or functions. It is possible. As used herein, “processor-implemented module” is Hardware refers to a hardware module that includes one or more processors. Therefore, the operations described herein are less than the same because the processor is an example of hardware. It can be partially implemented by the processor or partially implemented by hardware. It can also be carried out in both of those ways, and any one of the methods discussed herein At least some of the operations within the above methods are performed by one or more processors in a module. By using hardware-implemented modules, or any suitable combination thereof It can be executed.

[0088] Furthermore, one or more such processors are in a "cloud computing" environment. In or as a service (for example, "Software as a Service" (SaaS) It can perform operations as a service within an embodiment of the sa service. For example, At least one specific operation within any one or more of the methods discussed herein These actions are performed on a network (e.g., the Internet) and one or more appropriate inputs. Interface (for example, Application Programming Interface (API: application) Computers accessible via a program interface (e.g., processors) It can be performed by a group of machines (including the S). The execution is not limited to a single machine or is deployed across multiple machines. It can be distributed across just one or more processors. In some exemplary embodiments... In this case, one or more of those processors or hardware modules (for example, processor The modules implemented (for example, in residential environments, office environments, or server farms) It can be located in a single geographical location (within). In other exemplary embodiments One or more of those processors or hardware modules are located in multiple geographical locations. It can be distributed across multiple points.

[0089] Throughout this specification, multiple instances are described as a single instance. It is possible to perform the components, operations, or structures of one or more methods. Although they are illustrated and described as separate actions, one or more of these individual actions These can be executed simultaneously, and the operations do not need to be performed in the order shown in the diagram. Example configuration The structures and their functionalities presented as separate components and functions are combined It can be implemented as a combined structure or component having combined functions. Similarly, structures and functionalities presented as single components are separate components and These can be implemented as functions. These and other variations, modified forms, additional forms, and And improved forms are included within the scope of the subject matter as defined herein.

[0090] Some parts of this subject discussed herein are memory (e.g., computer memory). On data stored as bits or binary digital signals in (or other machine memory), The operation can be represented by an algorithm or symbolic representation. A m or symbolic representation is used by a person skilled in the field of data processing technology to describe the content of their work to other persons skilled in the field of the same technology. This is an example of a technique used to communicate with a contractor. As used herein, "Al "Gorhythm" is a self-consistent sequence of actions or similar processes that produce a desired result. Yes. In this context, algorithms and operations involve physical manipulation of physical quantities. Not necessarily. While not always the case, such quantities are typically stored, accessed, transferred, combined, and compared by machines. It can take the form of an electrical signal, magnetic signal, or optical signal that can be operated in any other way. Sometimes, such signals are referred to as "data" or "content," primarily for common use. "bit", "value", "element", "symbol", "character", "term", "number", "digit" It is convenient to refer to these words. However, these words are merely convenient references. It is merely a bell and should be associated with an appropriate physical quantity.

[0091] Unless otherwise specified, "accessing" and "processing" are used. "Detecting", "Computing", "Calculating", "Decision (d 'etermining', 'generating', 'presenting', 'displaying' Any discussion in this specification using words such as "ng" refers to one or more memories (e.g., volatile). Memory, non-volatile memory, or any suitable combination thereof), registers, or information Other mechanical components that receive, store, transmit, or display physical (e.g., electronic, magnetic) information within them. A machine (e.g., a computer) that manipulates or converts data expressed as quantity (or light) Therefore, it refers to an actionable activity or process. Furthermore, unless otherwise specified, the term " "a" or "an" is common in patent documents, and in this specification, one or Used to include two or more instances. Finally, as used herein The conjunctions "or / or" are, unless otherwise specified, non-exclusive "or / or It refers to "ha".

[0092] The following descriptions apply to the methods, machine-readable media, and systems discussed herein (e.g., For example, it describes various examples of machines, devices, or other apparatus.

[0093] The first example is, The first image captures a reference set of equipment on the transport device before the procedure begins, and the machine's 1 Steps accessed by more than one processor, The first instrument data corresponding to the reference set of the aforementioned instruments is obtained from the first image of the machine Steps for one or more processors to identify, A second image, captured after the commencement of the aforementioned procedure, shows the equipment on the transport device, and the machine The steps of accessing one or more processors, The second instrument data corresponding to the instrument on the transport equipment after the commencement of the above procedure is the second The steps include: the process of one or more processors of the machine identifying from an image; The one or more processors of the machine process the first instrument data and the second instrument data. The step of comparing with, Based on the above comparison, the equipment on the transport device before the commencement of the procedure is the same as the procedure A notification indicating that the transport equipment is not on the machine after the start is issued by one or more processors of the machine. The steps presented and We provide a method that includes the following:

[0094] The second example is, Steps to access the reference image of the device, A step of identifying an instrument in the first image based on the reference image, The instrument data in step 1 indicates the identified instrument in the first image, and , A step of identifying an instrument in the second image based on the reference image, The second instrument data indicates the identified instrument in the second image, step and The present invention provides a method of the first example, further including the following:

[0095] The third example is, The shape of the instrument in the first image is optically recognized, and the first instrument data is acquired. Steps and The shape of the instrument in the second image is optically recognized, and the second instrument data is acquired. Steps and The present invention provides a method according to the first or second example, further including the above.

[0096] A fourth example is that the first and second images are of the type of treatment or the treatment A method according to any of the first to third examples, corresponding to at least one of the performers. To provide.

[0097] A fifth example is that the first instrument data includes the first instrument count, and the second instrument data The present invention provides a method according to any of the first to fourth examples, including a second instrument count. .

[0098] The sixth example is the step of comparing the first instrument data with the second instrument data. , including comparing the first instrument count with the second instrument count, the notification is , the total number of missing fixtures or the number of shared types, of which less The fifth example provides a method that demonstrates one of the two possibilities.

[0099] The seventh example is that the procedure includes a surgical procedure performed on a patient by a physician, Image 1 shows the transport device before the start of the surgical procedure on the patient by the physician. The reference set of the aforementioned instruments is captured, and the second image is used by the physician for the patient. Examples 1 to 6 capture the instrument on the transport device after the completion of the surgical procedure. This provides a method for one of the examples.

[0100] The eighth example is, One or more processors, When at least one of the aforementioned one or more processors is executed, a certain operation Memory that stores the instructions to execute and A system (for example, a computer system) that is equipped with, The aforementioned operation is, Access the first image, which captures a reference set of instruments on the transport equipment before commencing the procedure. to, Identifying first instrument data corresponding to the reference set of the aforementioned instruments from the first image. , Access a second image capturing the equipment on the transport device after the commencement of the aforementioned procedure. That thing, The second instrument data corresponding to the instrument on the transport equipment after the commencement of the above procedure is the second Identifying from images, Comparing the first instrument data with the second instrument data, Based on the above comparison, the equipment on the transport device before the commencement of the procedure is the same as the procedure To display a notice indicating that the transport equipment is not on the transport equipment after the start of the process. We provide a system that includes this.

[0101] The ninth example is that the operation described above is The shape of the instrument in the first image is optically recognized, and the first instrument data is acquired. That thing, The shape of the instrument in the second image is optically recognized, and the second instrument data is acquired. thing and The system described in the eighth example is provided, further including the following.

[0102] The tenth example is a machine in which, when one or more processors are executed, the machine is made to perform an operation. A machine-readable medium (e.g., a non-temporary machine-readable storage medium) containing instructions, The aforementioned operation is, Access the first image, which captures a reference set of instruments on the transport equipment before commencing the procedure. to, Identifying first instrument data corresponding to the reference set of the aforementioned instruments from the first image. , Access a second image capturing the equipment on the transport device after the commencement of the aforementioned procedure. That thing, The second instrument data corresponding to the instrument on the transport equipment after the commencement of the above procedure is the second Identifying from images, Comparing the first instrument data with the second instrument data, Based on the above comparison, the equipment on the transport device before the commencement of the procedure is the same as the procedure To display a notice indicating that the transport equipment is not on the transport equipment after the start of the process. We provide a machine-readable medium that includes [the specified element].

[0103] The eleventh example is a collection of instruments captured before the commencement of the procedure and usable in the procedure. The steps include: one or more processors of the machine accessing a first image showing the match; Captured after the commencement of the procedure, the true part of the assembly of instruments shown in the first image The step involves one or more processors of the machine accessing a second image representing a set. and, The instrument in the set of instruments shown in the first image is the same as the instrument in the second image. If not shown in the proper subset of the set, then one or more processors of the machine The steps to make a decision, This indicates that the instrument not shown in the second image is missing from the set of instruments. The step of one or more processors of the machine presenting a notification We provide a method that includes the following:

[0104] The twelfth example is, The step of accessing the first image showing the set of instruments is a reference set of instruments The 11th example provides a method that includes accessing a reference image showing [the relevant information].

[0105] The 13th example is, The aforementioned reference image corresponds to at least one of the procedure or the person performing the procedure, Accessing the aforementioned reference image is the procedure or the person performing the procedure. The method of the 12th example is provided, which is based on at least one of the following.

[0106] The 14th example is, The reference set of the aforementioned instruments corresponds to at least one of the aforementioned procedure or the person performing the procedure. death, Accessing the reference image showing the reference set of the aforementioned instruments is the procedure or the The 12th or 13th example is based on at least one of the persons who perform the procedure. The method described is provided.

[0107] The 15th example is, The instrument in the set of instruments shown in the first image is shown in the second image. The step of determining that there is none is, Optically recognizing the shape of the instrument in the first image, The shape of the instrument in the second image cannot be optically recognized. The present invention provides a method according to any of the 11th to 14th examples, which includes [the specified element].

[0108] The 16th example is, The instrument in the set of instruments shown in the first image is shown in the second image. The step of determining that there is none is, Accessing the reference model of the aforementioned device, Each of the multiple silhouettes of the reference model of the instrument in the second image is optically The inability to recognize The present invention provides a method according to any of the 11th to 15th examples, which includes [the specified element].

[0109] The 17th example is, The instrument in the set of instruments shown in the first image is shown in the second image. The step of determining that there is none is, Accessing a reference model representing the reference shape of the device shown in the first image and , Depth data representing the current shape of the true subset of the set of instruments shown in the second image Accessing the data and The reference shape of the aforementioned device is the current shape of each of the proper subsets of the set of the aforementioned devices. Comparing and The present invention provides a method according to any of the 11th to 16th examples, which includes [the specified element].

[0110] The 18th example is, The step of accessing the first image is performed before the first frame of the first image. Capture the sequence of and at least prior to the captured first sequence. This is done by selecting the first image. The step of accessing the second image is followed by the second of the second frame after the procedure. Capture the sequence and at least prior to the captured second sequence. This is done by selecting the second image, as described in one of the examples from the 11th to the 17th. Provides a method for loading.

[0111] The 19th example is, A first set of instruments captured before the start of the procedure, which shows the set of instruments that can be used in the procedure. The steps include: one or more processors of the machine accessing the image, A portion of the assembly of the instruments shown in the first image, captured after the commencement of the procedure. The steps include: one or more processors of the machine accessing a second image showing a set; , The instruments among the set of instruments shown in the first image were used in the procedure. Based on the first and second images, the machine A step in which one or more processors of the machine make a determination, A notice indicating whether the aforementioned instrument was used or not in the aforementioned procedure is provided beforehand. Steps presented by one or more processors of the machine and We provide a method that includes the following:

[0112] The 20th example is, The subset of the set of instruments is a proper subset of the set of instruments, as in the 19th example. The method described is provided.

[0113] The 21st example is, The determination of whether the instrument was used or not in the procedure is made as described above. The step is to move the device from the first position in the transport equipment shown in the first image The process includes determining whether the transport device has moved to a second position shown in image 2. The method described in Example 19 or Example 20 is provided.

[0114] The 22nd example is, The determination of whether the instrument was used or not in the procedure is made as described above. The steps are, Optically recognizing that there is no blood on the instrument shown in the first image. and, Optically recognizing the presence of blood on the instrument shown in the second image above. The present invention provides a method according to any of the 19th to 21st examples, including the method described herein.

[0115] The 23rd example is, When one or more processors of a machine are executed, they include instructions that cause the machine to perform an action. a machine-readable medium (for example, a non-temporary machine-readable storage medium), The aforementioned operation is, A first set of instruments captured before the start of the procedure, which shows the set of instruments that can be used in the procedure. Accessing images and Captured after the commencement of the procedure, the true part of the assembly of instruments shown in the first image Accessing a second image that shows the set division, The instrument in the set of instruments shown in the first image is the same as the instrument in the second image. Determining that it is not shown in the aforementioned proper subset of the set, This indicates that the instrument not shown in the second image is missing from the set of instruments. To present the notification and We provide a machine-readable medium that includes [the specified element].

[0116] The 24th example is, The instrument in the set of instruments shown in the first image is shown in the second image. The conclusion that they are not present is, Optically recognizing the shape of the instrument in the first image, The shape of the instrument in the second image cannot be optically recognized. The present invention provides a machine-readable medium, including the one described in the 23rd example.

[0117] The 25th example is, When one or more processors of a machine are executed, they include instructions that cause the machine to perform an action. a machine-readable medium (for example, a non-temporary machine-readable storage medium), The aforementioned operation is, A first set of instruments captured before the start of the procedure, which shows the set of instruments that can be used in the procedure. Accessing images and A portion of the assembly of the instruments shown in the first image, captured after the commencement of the procedure. Accessing the second image showing the set, The instruments among the set of instruments shown in the first image were used in the procedure. Whether or not it was used is determined based on the first and second images. That thing, A notice is submitted indicating whether the aforementioned instrument was used or not in the aforementioned procedure. To show and We provide a machine-readable medium that includes [the specified element].

[0118] The 26th example is, The determination of whether the instrument was used or not in the procedure is made as described above. This means that the device moves from the first position in the transport equipment shown in the first image to the second position The 25th includes determining whether the transport equipment has moved to a second position shown in the image. The machine-readable media described in the example is provided.

[0119] The 27th example is, One or more processors, When at least one of the aforementioned one or more processors is executed, a certain operation Memory that stores the instructions to execute and A system that is equipped with, The aforementioned operation is, A first set of instruments captured before the start of the procedure, which shows the set of instruments that can be used in the procedure. Accessing images and Captured after the commencement of the procedure, the true part of the assembly of instruments shown in the first image Accessing a second image that shows the set division, The instrument in the set of instruments shown in the first image is the same as the instrument in the second image. Determining that it is not shown in the aforementioned proper subset of the set, This indicates that the instrument not shown in the second image is missing from the set of instruments. To present the notification and We provide a system that includes this.

[0120] The 28th example is, The instrument in the set of instruments shown in the first image is shown in the second image. The conclusion that they are not present is, Accessing a reference model representing the reference shape of the device shown in the first image and , Depth data representing the current shape of the true subset of the set of instruments shown in the second image Accessing the data and The reference shape of the aforementioned device is the current shape of each of the proper subsets of the set of the aforementioned devices. Comparing and The system described in Example 27, including the above, is provided.

[0121] The 29th example is, One or more processors, When at least one of the aforementioned one or more processors is executed, a certain operation Memory that stores the instructions to execute and A system that is equipped with, The aforementioned operation is, A first set of instruments captured before the start of the procedure, which shows the set of instruments that can be used in the procedure. Accessing images and A portion of the assembly of the instruments shown in the first image, captured after the commencement of the procedure. Accessing the second image showing the set, The instruments among the set of instruments shown in the first image were used in the procedure. Whether or not it was used is determined based on the first and second images. That thing, A notice is submitted indicating whether the aforementioned instrument was used or not in the aforementioned procedure. To show and We provide a system that includes this.

[0122] The 30th example is, The determination of whether the instrument was used or not in the procedure is made as described above. That is, Optically recognizing that there is no blood on the instrument shown in the first image. and, Optically recognizing the presence of blood on the instrument shown in the second image above. The system described in Example 29, including the above, is provided.

[0123] The 31st example is the above action performed in any one of the aforementioned examples (for example) A carrier medium that carries machine-readable instructions that control a machine to perform a method of operation. To provide.

Claims

1. The first image captures a reference set of equipment on the transport device before the procedure begins, and the machine's 1 Steps accessed by more than one processor, The first instrument data corresponding to the reference set of the aforementioned instruments is obtained from the first image of the machine A step in which one or more processors identify, A second image, captured after the commencement of the aforementioned procedure, shows the equipment on the transport device, and the machine The steps of accessing one or more processors, The second instrument data corresponding to the instrument on the transport equipment after the commencement of the above procedure is the second The steps include: the process of one or more processors of the machine identifying from the image; The one or more processors of the machine process the first instrument data and the second instrument data. The step of comparing with, Based on the above comparison, the equipment on the transport device before the commencement of the procedure is the same as the procedure A notification indicating that the transport equipment is not on the machine after the start is issued by one or more processors of the machine. The steps presented and A method that includes this.

2. Steps to access the reference image of the device, A step of identifying an instrument in the first image based on the reference image, The instrument data in step 1 indicates the identified instrument in the first image, and 、 A step of identifying an instrument in the second image based on the reference image, The second instrument data indicates the identified instrument in the second image, step and The method according to claim 1, further comprising:

3. The shape of the instrument in the first image is optically recognized, and the first instrument data is acquired. Steps and The shape of the instrument in the second image is optically recognized, and the second instrument data is acquired. Steps and The method according to claim 1, further comprising:

4. The first image and the second image are among the types of the procedure or the performers of the procedure. The method according to claim 1, which corresponds to at least one of the above.

5. The first instrument data includes the first instrument count, and the second instrument data includes the second instrument count. The method according to claim 1, including a count of ingredients.

6. The step of comparing the first instrument data with the second instrument data is the first The notification includes comparing the instrument count with the second instrument count, and the notification is missing. Shows at least one of the total number of fixtures or the total number of missing fixtures of a shared type. The method according to claim 5.

7. The procedure includes a surgical procedure performed on a patient by a physician, and the first image is The base of the instrument on the transport device before the start of the surgical procedure on the patient by the physician A quasi-set is captured, and the second image is the surgical procedure performed on the patient by the physician. The method according to claim 1, comprising capturing the equipment on the transport equipment after the installation is complete.

8. One or more processors, When at least one of the aforementioned one or more processors is executed, a certain operation Memory that stores the instructions to execute and A system that is equipped with, The aforementioned operation is, Access the first image, which captures a reference set of instruments on the transport equipment before starting the procedure. to, Identifying first instrument data corresponding to the reference set of the aforementioned instruments from the first image. 、 Access a second image capturing the equipment on the transport device after the commencement of the aforementioned procedure. That thing, The second instrument data corresponding to the instrument on the transport equipment after the commencement of the above procedure is the second Identifying from images, Comparing the first instrument data with the second instrument data, Based on the above comparison, the equipment on the transport device before the commencement of the procedure is the same as the procedure To display a notice indicating that the transport equipment is not on the transport equipment after the start of the process. A system that includes this.

9. The aforementioned operation is, The shape of the instrument in the first image is optically recognized, and the first instrument data is acquired. That thing, The shape of the instrument in the second image is optically recognized, and the second instrument data is acquired. thing and The system according to claim 8, further comprising:

10. When one or more processors of a machine are executed, they include instructions that cause the machine to perform an action. Hmm, a tangible machine-readable storage medium, The aforementioned operation is, Access the first image, which captures a reference set of instruments on the transport equipment before starting the procedure. to, Identifying first instrument data corresponding to the reference set of the aforementioned instruments from the first image. 、 Access a second image capturing the equipment on the transport device after the commencement of the aforementioned procedure. That thing, The second instrument data corresponding to the instrument on the transport equipment after the commencement of the above procedure is the second Identifying from images, Comparing the first instrument data with the second instrument data, Based on the above comparison, the equipment on the transport device before the commencement of the procedure is the same as the procedure To display a notice indicating that the transport equipment is not on the transport equipment after the start of the process. A tangible, machine-readable storage medium that includes [the specified element].

11. A first image captured before the start of the procedure, showing the instrument that can be used in the procedure. The process involves one or more processors of the machine accessing a first image showing a collection of components, A second image captured after the commencement of the aforementioned procedure, which is shown in the first image. A second image showing a proper subset of the set of instruments shows one or more processors of the machine. Steps to access, The instrument in the set of instruments shown in the first image is the same as the instrument in the second image. If not shown in the proper subset of the set, then one or more processors of the machine The steps to make a decision, This indicates that the instrument not shown in the second image is missing from the set of instruments. The step of one or more processors of the machine presenting a notification A method that includes this.

12. The step of accessing the first image showing the set of instruments is a reference set of instruments. The method according to claim 11, comprising accessing a reference image showing a reference image.

13. The aforementioned reference image corresponds to at least one of the procedure or the person performing the procedure, Accessing the aforementioned reference image is the procedure or the person performing the procedure. The method according to claim 12, which is based on at least one of the following.

14. The reference set of the aforementioned instruments corresponds to at least one of the aforementioned procedure or the person performing the procedure. death, Accessing the reference image showing the reference set of the aforementioned instruments is the procedure or the The method according to claim 12, wherein the method is based on at least one of the persons who perform the procedure.

15. The instrument in the set of instruments shown in the first image is shown in the second image. The step of determining that there is none is, Optically recognizing the shape of the instrument in the first image, The shape of the instrument in the second image cannot be optically recognized. The method according to claim 11, including the method described in claim 11.

16. The instrument in the set of instruments shown in the first image is shown in the second image. The step of determining that there is none is, Accessing the reference model of the aforementioned device, Each of the multiple silhouettes of the reference model of the instrument in the second image is optically The inability to recognize The method according to claim 11, including the method described in claim 11.

17. The instrument in the set of instruments shown in the first image is shown in the second image. The step of determining that there is none is, Accessing a reference model representing the reference shape of the device shown in the first image and 、 Depth diagram representing the current shape of the true subset of the set of instruments shown in the second image Accessing the data and The reference shape of the aforementioned device is the current shape of each of the proper subsets of the set of the aforementioned devices. Comparing and The method according to claim 11, including the method described in claim 11.

18. The step of accessing the first image is performed before the first frame of the first image. Capture the sequence of and at least prior to the captured first sequence. This is done by selecting the first image. The step of accessing the second image is followed by the second of the second frame after the procedure. Capture the sequence and at least prior to the captured second sequence. The method according to claim 11, which is performed by selecting the second image.

19. A first image captured before the start of the procedure, showing the instrument that can be used in the procedure. The process involves one or more processors of the machine accessing a first image showing a collection of components, A second image captured after the commencement of the aforementioned procedure, which is shown in the first image. In the second image showing a subset of the aforementioned set of devices, one or more of the machine's processors are Steps to success, The instrument among the set of instruments shown in the first image was used in the procedure. Based on the first and second images, the machine The steps of one or more processors of the machine making a determination, A notice indicating whether the aforementioned instrument was used or not in the aforementioned procedure is provided beforehand. Steps presented by one or more processors of the machine and A method that includes this.

20. Claim 19 states that the subset of the set of instruments is a proper subset of the set of instruments. Method of description.

21. The determination of whether the instrument was used or not in the procedure is made as described above. The step is that the device moves from the first position in the transport equipment shown in the first image This includes determining whether the device has moved to the second position within the transport equipment shown in image 2. The method described in item 19.

22. The determination of whether the instrument was used or not in the procedure is made as described above. The steps are, Optically recognizing that there is no blood on the instrument shown in the first image. and, Optically recognizing the presence of blood on the instrument shown in the second image above. The method according to claim 19, including the method described in claim 19.

23. When one or more processors of a machine are executed, they include instructions that cause the machine to perform an action. Hmm, a tangible machine-readable storage medium, The aforementioned operation is, A first image captured before the start of the procedure, showing the instrument that can be used in the procedure. Accessing the first image showing the set of ingredients, A second image captured after the commencement of the aforementioned procedure, which is shown in the first image. Accessing a second image showing a proper subset of the aforementioned set of devices, The instrument in the set of instruments shown in the first image is the same as the instrument in the second image. Determining that it is not shown in the aforementioned proper subset of the set, This indicates that the instrument not shown in the second image is missing from the set of instruments. To present the notification and A tangible, machine-readable storage medium that includes [the specified element].

24. The instrument in the set of instruments shown in the first image is shown in the second image. The conclusion that they are not present is, Optically recognizing the shape of the instrument in the first image, The shape of the instrument in the second image cannot be optically recognized, A tangible machine-readable storage medium according to claim 23, including the following:

25. When one or more processors of a machine are executed, they include instructions that cause the machine to perform an action. Hmm, a tangible machine-readable storage medium, The aforementioned operation is, A first image captured before the start of the procedure, showing the instrument that can be used in the procedure. Accessing the first image showing the set of ingredients, A second image captured after the commencement of the aforementioned procedure, which is shown in the first image. Accessing a second image showing a subset of the aforementioned set of devices, The instrument among the set of instruments shown in the first image was used in the procedure. Whether or not it was used is determined based on the first and second images. That thing, A notification is provided indicating whether the aforementioned instrument was used or not in the aforementioned procedure. To show and A tangible, machine-readable storage medium that includes [the specified element].

26. The determination of whether the instrument was used or not in the procedure is made as described above. This means that the device moves from the first position in the transport equipment shown in the first image to the second position A claim comprising determining whether the transport device has moved to a second position shown in the image. A tangible machine-readable storage medium as described in 25.

27. One or more processors, When at least one of the aforementioned one or more processors is executed, a certain operation Memory that stores the instructions to execute and A system that is equipped with, The aforementioned operation is, A first image captured before the start of the procedure, showing the instrument that can be used in the procedure. Accessing the first image showing the set of ingredients, A second image captured after the commencement of the aforementioned procedure, which is shown in the first image. Accessing a second image showing a proper subset of the aforementioned set of devices, The instrument in the set of instruments shown in the first image is the same as the instrument in the second image. Determining that it is not shown in the aforementioned proper subset of the set, This indicates that the instrument not shown in the second image is missing from the set of instruments. To present the notification and A system that includes this.

28. The instrument in the set of instruments shown in the first image is shown in the second image. The conclusion that they are not present is, Accessing a reference model representing the reference shape of the device shown in the first image and 、 Depth diagram representing the current shape of the true subset of the set of instruments shown in the second image Accessing the data and The reference shape of the aforementioned device is the current shape of each of the proper subsets of the set of the aforementioned devices. Comparing and The system according to claim 27, including the system described in claim 27.

29. One or more processors, When at least one of the aforementioned one or more processors is executed, a certain operation Memory that stores the instructions to execute and A system that is equipped with, The aforementioned operation is, A first image captured before the start of the procedure, showing the instrument that can be used in the procedure. Accessing the first image showing the set of ingredients, A second image captured after the commencement of the aforementioned procedure, which is shown in the first image. Accessing a second image showing a subset of the aforementioned set of devices, The instrument among the set of instruments shown in the first image was used in the procedure. Whether or not it was used is determined based on the first and second images. That thing, A notification is provided indicating whether the aforementioned instrument was used or not in the aforementioned procedure. To show and A system that includes this.

30. The determination of whether the instrument was used or not in the procedure is made as described above. That is, Optically recognizing that there is no blood on the instrument shown in the first image. and, Optically recognizing the presence of blood on the instrument shown in the second image above. The system according to claim 29, including the system described in claim 29.

Images