Method for monitoring consumables during a medical procedure and medical image processing system

Abstract

A method for monitoring consumables during a medical procedure, wherein unused consumables are provided in a staging area (15), consumables are used in an operating area (7), and used consumables are deposited in a collection area (20), comprising the steps of: - observing the staging area (15) and the collection area (20) with at least one camera (21, 22), - recording, by means of a first automatic image processing method, when unused consumables are removed from the staging area (15), - recording, by means of a second automatic image processing method, when used consumables are deposited in the collection area (20), and - accounting for the consumables removed from the staging area (15) and the consumables deposited in the collection area (20), characterized in thatthat the first and / or second automatic image processing method is set up to detect when several pieces of a countable consumable are simultaneously removed from the dispensing area (15) or placed in the collection area (20).

Description

[0001] The invention relates to a method for monitoring consumables during a medical procedure and to a medical image processing system.

[0002] Many medical procedures, particularly surgical interventions, utilize consumable materials. For the purposes of this invention, consumable materials are defined as auxiliary materials that are provided for single use during the respective procedure and disposed of after use.

[0003] A typical example of consumables are surgical swabs, which are used during surgical procedures to absorb blood and other bodily fluids in the surgical field to ensure good visibility for the surgeon.

[0004] Depending on the amount of fluid during the procedure, such a swab can remain in the surgical site for a longer period of time, for example to absorb small amounts of seeping blood, or be removed again after a short time, for example in the case of heavy bleeding.

[0005] In every case, at the end of a procedure, it must be ensured that no consumables are accidentally left in the surgical site when it is closed. Consumables left in the surgical site can cause persistent discomfort, delayed wound healing, or even more serious complications such as infections.

[0006] Studies show that despite strict regulations for completeness checks during surgical procedures in body cavities, there are approximately 20 incidents per 100,000 procedures in which consumable material is unintentionally left in the surgical site ("Every swab counts", https: / / www.aps-ev.de / wp-content / uploads / 2016 / 09 / Glossar_JTZ_Internet.pdf).

[0007] One object of the invention is to provide methods and systems that significantly reduce the risk of consumables being forgotten in the surgical setting.

[0008] This problem is solved according to a first aspect of the invention by a method for monitoring consumables during a medical procedure, wherein unused consumables are provided in a staging area, consumables are used in an operating area, and used consumables are deposited in a collection area, comprising the steps of: observing the staging area and the collection area with at least one camera, registering, by means of a first automatic image processing method, when unused consumables are removed from the staging area, registering, by means of a second automatic image processing method, when used consumables are deposited in the collection area, and accounting for the consumables removed from the staging area and the consumables deposited in the collection area.

[0009] The use of automated image processing methods eliminates human error in the accounting of consumables. The end-to-end monitoring according to the invention is particularly reliable because the respective supply and collection areas are generally easily detectable by cameras. This prevents detection errors caused by obstructions of the relevant areas, such as people in the camera's field of view.

[0010] Classical image processing methods, which recognize the desired objects based on predefined rules, can be used as automatic image processing methods. Alternatively, image processing methods based on artificial intelligence can be used.

[0011] In a further development of a method according to the invention, the operating area can additionally be monitored with at least one camera, and a third automatic image processing method can be used to determine at which point in the operating area registered consumables are introduced and / or from which point in the operating area registered consumables are removed. This makes it possible, upon detection of a shortage of consumables, to automatically determine where in the operating area consumables are likely to be found. This can thus accelerate the process of locating missing consumables.

[0012] In a particular embodiment of a method according to the invention, the consumable material can be countable, and a consumable counter can be maintained which is set to 0 at the beginning of the medical procedure; is incremented when a consumable is removed from the staging area; and is decremented when a consumable is placed in the collection area. The consumable counter thus indicates at all times how many consumables are available in the operating room area.

[0013] The counter value can be displayed after the medical procedure is completed. For example, a consumables log can be generated as documentation after the procedure, explicitly stating the consumables counter value.

[0014] In a further embodiment of the method according to the invention, a still image can be captured and stored when the insertion and / or removal of a consumable into or from the operating area is registered. Storing such still images enables a manual recheck of the consumable count in case of deviations.

[0015] Similarly, a video sequence of a predefined or configurable length can be recorded and saved when the insertion and / or removal of consumables into or from the operating room area is registered. Video sequences can provide further information or enable more precise identification during a manual follow-up check of consumable counts.

[0016] In a particular embodiment of a method according to the invention, the consumable material is inserted and / or removed using one or more surgical grasping instruments, and a third automatic image processing method can detect whether a surgical grasping instrument with or without consumable material is inserted into and / or removed from the operating area. Including surgical grasping instruments in the analysis can provide the automatic image processing methods with additional information to increase the detection rate for consumables.

[0017] The first and / or second automatic image processing method may preferably be configured to detect when several pieces of a countable consumable are simultaneously removed from the staging area or placed in the collection area.

[0018] The first automated image processing method can be set up to determine the number of consumables removed based on geometric characteristics. This could be, for example, the change in the height of a stock stack, or the contour of fold lines if the consumables are held in folded form.

[0019] In the collection area, the countable consumables can be manually separated, and a second automatic image processing method can be configured to determine the number of consumables placed after separation. Since separation is often mandatory in the collection area anyway, determining the number of each consumable placed is greatly simplified.

[0020] According to a further aspect of the invention, the problem is solved by a medical image processing system comprising: one or more cameras configured to capture a staging area for unused consumables, a collection area for used consumables, and optionally an operating area; and a controller configured to receive video data from the one or more cameras; wherein the controller is configured to execute a method according to the above. Regarding the effects and advantages achievable thereby, explicit reference is made to the above.

[0021] The invention will now be explained in more detail with reference to some exemplary illustrations. These exemplary embodiments are intended solely to facilitate a better understanding of the invention, without limiting its scope.

[0022] They show: Fig. 1: The layout of an operating room in a simplified representation, Fig. 2: An accounting method, Fig. 3a, b: optical scenarios in the deployment area, Fig. 4a, b: optical scenarios in the collection area, Fig. 5: An image processing system.

[0023] In Fig. Figure 1 shows a possible layout of an operating room 1 in a highly simplified top view. Approximately in the center of the operating room 1 is an operating table 2, on which a procedure can be performed. Around the operating table 2 are movement areas 3, 4, 5, and 6 for personnel directly involved in the procedure. The operating table 2 and the movement areas 3, 4, 5, and 6 constitute an operating area 7.

[0024] In Fig. To the right of operating area 7 are staging tables 10 and 11. Staging table 10 contains various surgical instruments, while staging table 11 holds several stacks of medical supplies, in this example swabs for absorbing blood and other fluids. Between staging tables 10 and 11 and operating area 7 is a movement area 12 for an assistant, who retrieves surgical instruments and / or supplies from staging tables 10 and 11 and hands them to the personnel directly involved in the procedure. Staging tables 10 and 11 and movement area 12 together form a staging area 15.

[0025] In Fig. To the left of operating room area 7 are storage tables 16 and 17, as well as a waste container 18. Used surgical instruments can be placed on storage table 16. Used consumables can be placed on storage table 17 and separated for inventory purposes. After separation and inventory, the consumables can be placed in waste container 18. The consumables are generally placed in waste container 18 only after the procedure has been completely finished, to allow for recounting in case of discrepancies in the inventory. Between storage tables 16 and 17 and waste container 18 on one side, and operating room area 7 on the other, a movement area 19 is indicated for another assistant, who receives used surgical instruments and consumables. Storage tables 16 and 17, waste container 18, and movement area 19 form a collection area 20.

[0026] The positions of staging area 15 and collection area 20 relative to operating area 7 can vary depending on the type of procedure; the configuration shown is merely one possible example to illustrate the individual functions. In many cases, staging tables 10 and 11 will be configured as a single table. It is also not uncommon to combine staging area 15 and collection area 20 so that staging and receiving of materials can be carried out by a single assistant.

[0027] To ensure that no surgical instruments or consumables are unintentionally left inside a patient after a procedure, especially an open procedure in a body cavity, these instruments and consumables are tracked. This means that all instruments and consumables removed from staging area 15 are also deposited in collection area 20. This tracking is usually done manually and is therefore prone to errors. Particularly during emergency surgeries, unplanned changes to the surgical procedure, or staff changes during the operation, tracking errors can occur, resulting in items such as surgical instruments or consumables being left inside the patient unintentionally.While the risk of this is rather low for surgical instruments due to their good visual visibility, there is an increased risk for consumables such as swabs, as they are difficult to identify, especially after contact with blood inside a body cavity.

[0028] To make the accounting of consumables independent of human influence, cameras 21 and 22 are provided in the supply area 15 and the collection area 20, which monitor the respective areas and transmit video images of the corresponding areas to a [missing information - likely a system or system]. Fig. 1. Send control not shown.

[0029] The control system executes several automatic image processing procedures in parallel, which evaluate the video images supplied by cameras 21,22 in order to register the removal of consumables from the supply area 15 and / or the deposit of consumables in the collection area 20.

[0030] Another camera 23 can be installed in the operating room area to send video images from the operating room to the control system. There, an additional automated image processing procedure can analyze where consumables are inserted and / or removed within the operating room.

[0031] In addition, the control system performs a consumables accounting procedure, which is divided into three independent subprocesses, which are in Fig. Figure 2 shows the accounting method. This method uses a single consumption counter, which is defined in the control system, for example, as a variable and is set to 0 at the beginning of an intervention.

[0032] A first subprocess 100 processes information from the staging area 15. In the first step 101, a video image or a short video sequence is received from camera 21. In a further step 102, it is checked whether the video image or video sequence shows the removal of consumables from the staging area 15. If this is not the case, subprocess 100 starts again at step 101 to receive another image or video sequence.

[0033] If, however, the removal of consumables is detected in step 102, then in step 103 the number of countable consumables removed is determined. In step 104, the consumables counter is then incremented by the corresponding number. This indicates that the specified number of consumables has been removed from the supply area and is therefore potentially located inside the patient's body. Subprocess 100 then restarts at step 101.

[0034] A second subprocess 200 processes information from collection area 20. In a first step 201, a video image or a short video sequence is received from camera 22. In a further step 202, it is checked whether the video image or video sequence shows the storage of consumables in collection area 20. If this is not the case, subprocess 200 starts again at step 201 to receive another image or video sequence.

[0035] If, however, the placement of consumables is detected in step 202, step 203 determines how many countable consumables were placed. In step 204, the consumables counter is then deducted by the corresponding number. This indicates that the relevant number of consumables has been removed from the operating area and therefore can no longer potentially be inside the patient's body. Subsequently, subprocess 200 restarts at step 201.

[0036] A third subprocess, 300, processes information from operating room area 7. Here, in a first step (301), a video image or short video sequence is received from camera 23. In a further step (302), it is checked whether the video image or video sequence shows the insertion or removal of consumables into or from operating room area 7. If this is not the case, subprocess 300 also starts again at step 301 to receive another image or video sequence.

[0037] Unlike subprocesses 100 and 200, subprocess 300, in step 303, determines the point within the operating room area where consumables were inserted or removed. For this purpose, a virtual grid of positions is defined, covering the operating room area, and a separate consumables counter is provided for each cell of the grid.

[0038] In step 304, the number of consumables added or removed is determined again. In step 305, the consumables counter corresponding to the position determined in step 303 is increased or decreased by the number determined in step 304 to indicate how many consumables are currently located at that specific point in the operating room area.

[0039] After completion of the intervention, which can be indicated to the controller manually via an input device, the value of the consumables counter can be output via a suitable output device, such as a monitor or other display. If the consumables counter value is not 0 at the end of the intervention, an error signal and / or an alarm can also be issued.

[0040] If subprocesses 100, 200, and 300 work with video sequences instead of individual images, the length of these sequences can be chosen so that, on the one hand, the volume of data to be processed remains manageable, and on the other hand, meaningful conclusions about trajectories are still possible. Reasonable sequence lengths could, for example, be between 1 and 5 seconds. With excessively long sequences, there is a risk that several relevant events will be present in a single sequence and that not all of them will be detected.

[0041] The image processing methods required for the individual subprocesses 100, 200, and 300 vary in complexity due to the different conditions. These methods generally employ well-known image processing operations, which a person skilled in the art selects from their expertise according to the specific requirements. In addition to deterministic algorithms, non-deterministic methods, commonly referred to as "artificial intelligence" (AI), can also be used for subtasks or for the entire image processing process. For image processing tasks such as object recognition and object classification, neural networks, such as convolutional neural networks, are particularly suitable.Such networks can be trained with training data in the form of video images or video sequences showing the removal or storage of unused or used consumables, and which have been pre-evaluated by manual classification.

[0042] The check to be carried out in steps 102 and 202, to determine whether consumables are being removed or placed down, is relatively straightforward. Both unused and used swabs or similar consumables can be easily identified visually against a background such as a stainless steel table; known object recognition algorithms can be used for this purpose. When evaluating video sequences, the trajectories of detected objects can also be determined. Removal or placement can then be considered reliably detected if, for example, the trajectory of a detected object crosses a predefined demarcation line, such as the edge of the respective table.

[0043] In the Fig. 3a, Fig. Section 3b shows two possible scenarios in deployment area 15. Fig. Figure 3a shows how a stack of 400 consumables is moved on the supply table 11, for example, to allow easier access. The stack of 400 does not cross the demarcation line 401; therefore, no material removal is detected.

[0044] In Fig. In step 3b, a piece 402 is removed from the stack 400 and moved across the demarcation line 401. The control system therefore registers the removal of consumables. The number of consumables actually removed can be determined relatively easily, for example, by taking a photograph at an angle and measuring the height of the stack 400 before and after removal. Alternatively, geometric characteristics of the consumables can be evaluated, such as characteristic creases of folded materials, or similar features.

[0045] In the Fig. 4a, Fig. Section 4b shows two possible scenarios in collection area 20. Fig. Figure 4a shows how a lump 500 of consumable material is placed across a demarcation line 501 onto the storage table 17. The control system interprets this as the placement of consumable material. Fig. Figure 4b shows how the lump of 500 consumable material is subsequently separated into two pieces of 500' and 500". No consumable material is moved across demarcation line 501, so no new deposit is registered. However, the control system can recognize from the separation process that two pieces of consumable material were indeed deposited previously and account for this accordingly.

[0046] In subprocess 200, further methods can be applied to precisely determine the number of deposited consumables. For example, camera 22 can be configured as a stereo camera or a time-of-flight (TOF) camera, so that the volume of the lump 500 of consumables can also be estimated from the video data. Since the volume of a single piece of consumables is known, the number of consumables present in lump 500 can also be deduced from the actual volume.

[0047] The video data to be processed in subprocess 300 is generally more difficult to analyze than the data to be processed in subprocesses 100 and 200. This is partly because, in the operating room, it is almost impossible to position camera 23 in such a way that its field of view is not occasionally obscured by the personnel involved in the procedure. Furthermore, the background of the image, which might depict, for example, an opened body cavity, does not provide a sharp contrast to pieces of used consumables. To nevertheless extract reasonably reliable information about the distribution of consumables, additional image elements can be analyzed. For example, consumables are typically inserted and removed using grasping instruments, which can be located relatively easily using image processing methods.Once a gripping instrument has been detected, further image analysis can also reliably determine whether the instrument is currently loaded with consumables. By evaluating this information, the whereabouts of the consumables can be indirectly determined.

[0048] To determine the respective number of consumables supplied or removed, subprocess 300 can access information from subprocesses 100 and 200; this is described in Fig. 2 indicated by the dashed lines 310, 320.

[0049] The primary function of consumable material accounting is implemented in sub-processes 100 and 200. Due to the good optical conditions in these sub-processes, a high level of detection reliability is ensured, so that the accuracy of the automatic accounting slightly exceeds that of manual accounting.

[0050] For subprocess 300, the probability of false positives is higher due to the poorer optical conditions. Therefore, the results of this subprocess are not used directly for accounting purposes, but only serve to more quickly identify missing consumables in the event of discrepancies in the accounting.

[0051] For tracking purposes, video data from cameras 21, 22, and 23 can be stored in whole or in part. If doubts arise regarding the accuracy of the accounting after an intervention, this can be verified manually or automatically using the stored video data. To reduce the amount of data to be stored, for example, only those video images or video sequences can be saved in which a removal, placement, or separation is clearly visible.

[0052] In Fig.Figure 5 shows a medical image processing system 600. The image processing system 600 comprises cameras 21, 22, 23, a controller 601, and a monitor 602.

[0053] The controller 601 includes a processor 605, a program memory 606, and a working memory 607.

[0054] Program memory 606 contains program information that is read and executed by processor 605 to perform the procedures described above. Processor 605 can be a standard processor. To achieve the highest possible processing speed, processor 605 can be a multi-core processor capable of executing subprocesses 100, 200, and 300 fully or partially in parallel. Multiprocessor systems with multiple CPUs can also be used. When using AI-based methods, graphics processing units (GPUs) specifically optimized for such methods can be employed. The video data from cameras 21, 22, and 23 is stored in main memory 607 during processing.

[0055] Monitor 602 is used to display the accounting result after the intervention has been completed.

[0056] The above description represents only one possible embodiment of the invention and can be modified within the scope of the appended claims. For example, with a suitable arrangement of operating area 7, staging area 15, and collection area 20, fewer than three cameras may be required; in the most favorable case, one camera may suffice. The camera intended for the operating area can be a room camera already present in modern operating rooms.

[0057] Although the invention is described here only in connection with open surgical procedures, it can equally be used in minimally invasive procedures performed with endoscopic instruments.

Claims

[1] Method for monitoring consumables during a medical procedure, wherein unused consumables are provided in a staging area (15), consumables are used in an operating area (7), and used consumables are deposited in a collection area (20), comprising the steps: - Observing the deployment area (15) and the collection area (20) with at least one camera (21, 22), - Register, by means of an initial automatic image processing procedure, when unused consumables are removed from the staging area (15), - Register, by means of a second automatic image processing procedure, when used consumables are placed in the collection area (20), and - Balance the consumables taken from the supply area (15) and the consumables placed in the collection area (20). [2] Method according to claim 1, characterized by , that in addition the operating area (7) is monitored with at least one camera (23), and that a third automatic image processing method is used to determine at which point in the operating area (7) registered consumables are introduced, and / or from which point in the operating area (7) registered consumables are removed. [3] Method according to claim 1 or 2, characterized by that the consumable material is countable, and that a consumables counter is kept, which - At the beginning of the medical procedure, the setting is set to 0, - When registering the withdrawal of a consumable from the staging area (15), the amount is increased, and - Reduced when registering the placement of a consumable in the collection area (20). [4] Method according to claim 3, characterized by, that the value of the counter will be displayed after completion of the medical procedure. [5] Method according to any one of claims 2 to 4, characterized by , that a still image is captured and stored when the insertion and / or removal of a consumable into or from the operating area (7) is registered. [6] Method according to any one of claims 2 to 4, characterized by , that a video sequence of a predetermined or predefinable length is recorded and stored when the insertion and / or removal of a consumable into or from the operating area (7) is registered. [7] Method according to any of the preceding claims, characterized by, that the consumable material is inserted and / or removed by means of one or more surgical grasping instruments, and that the third automatic image processing method detects whether a surgical grasping instrument with or without consumable material is inserted into and / or removed from the operating area (7). [8] Method according to any of the preceding claims, characterized by , that the first and / or second automatic image processing method is set up to detect when several pieces of a countable consumable are simultaneously taken from the staging area (15) or placed in the collection area (20). [9] Method according to claim 8, characterized by , that the first automatic image processing method is set up to determine the number of consumables removed based on geometric characteristics. [10] Method according to claim 8, characterized by, that countable consumables in the collection area (20) are manually separated, and that the second automatic image processing procedure is set up to determine the number of consumables placed after separation. [11] Medical image processing system, comprising: - one or more cameras (21, 22, 23) that are set up to capture a staging area (15) for unused consumables, a collection area (20) for used consumables, and, if applicable, an operating area (7), - a controller (601) configured to receive video data from one or more cameras (21, 22, 23), wherein the controller (601) is configured to execute a method according to any one of claims 1 to 10.

Images