Monitoring of surface cleaning of medical surfaces using video streaming

Abstract

To monitor surface cleaning using a camera together with machine learning.SOLUTION: A cleaning wizard monitors and provides feedback for cleaning of medical equipment to ensure that cleaning is performed on the basis of best practices. The cleaning wizard receives a video stream including an item of medical equipment and inputs a first set of video frames from the video stream into a first machine learning model. The first machine learning model is trained to output whether the first set of video frames corresponds to activity that initiates a cleaning protocol for the item of medical equipment. Responsive to the cleaning protocol being initiated, the cleaning wizard inputs a second set of video frames into a second machine learning model trained to output whether the second set of frames meets criteria of the cleaning protocol.SELECTED DRAWING: None

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Application No. 62 / 868,243, filed June 28, 2019, which is incorporated by reference in its entirety. [Background technology]

[0002] The present disclosure relates generally to using cameras in conjunction with machine learning to monitor surface cleaning, and more particularly to using video streaming to monitor and provide feedback of compliance with protocols for cleaning medical surfaces.

[0003] In healthcare, proper cleaning of medical equipment is crucial to controlling and reducing the spread of infectious diseases between patients. For example, small medical devices that come into contact with a patient's skin, such as stethoscopes and thermometers, can contribute to the spread of infectious diseases. As a specific example, stethoscopes, most notably the diaphragm, have been shown to be nearly 100% contaminated with potential pathogens and skin microflora typically acquired from colonized or infected patients. It is widely understood and documented that the level of stethoscope contamination, even after a single physical examination, is comparable to the contamination of a portion of a physician's dominant hand. Similar types of medical equipment that come into direct contact with a patient's skin or membranes are also at risk for this level of contamination.

[0004] Despite extensive documentation of contamination within healthcare environments, compliance with environmental cleaning standards and protocols varies, leading to outbreaks. Compliance challenges include failure to follow proper preparation, timing, and application of disinfectants as prescribed by manufacturer instructions, and inconsistencies in workflow that result in lack of adherence to standardized protocols and best practices. Summary of the Invention [Means for solving the problem]

[0005] These and other challenges are addressed by a cleaning wizard that uses video streaming to monitor and provide feedback during the cleaning of medical surfaces. The video stream is captured by a camera and transmitted to the cleaning wizard. The camera may be mounted on or within an item of medical equipment, or may be positioned externally to capture a view of the item of medical equipment. Based on the video data, the cleaning wizard monitors the cleaning protocol for the item of medical equipment and notifies one or more operators when the cleaning protocol requires attention, fails, or is complete based on best practices, including CDC guidelines and manufacturer instructions.

[0006] In one embodiment, the cleaning wizard receives a video stream including frames depicting the medical equipment. The cleaning wizard determines when a cleaning protocol for the item of medical equipment is initiated based on the video stream. In one embodiment, the cleaning wizard inputs the video stream data into a machine learning model to detect when cleaning is initiated. For example, the machine learning model is trained to output that a cleaning protocol is initiated when a wiping motion with a cloth is detected in the video stream moving across the surface of the item of medical equipment.

[0007] In response to a detected cleaning attempt, the cleaning wizard may monitor the video stream to ensure that the cleaning agent is applied correctly to the surface of the item of medical equipment and that the cleaning agent is applied sufficiently to meet manufacturer instructions. For example, many cleaning agents require that the surface be kept sufficiently wet for a certain length of time to thoroughly clean the surface. To maintain surface wetness, the operator may be notified and requested to reapply the cleaning agent for that length of time. In one embodiment, the cleaning wizard inputs the video stream data into one or more machine learning models that are trained to output whether the video stream meets the criteria of the cleaning protocol.

[0008] If the cleaning wizard determines that the cleaning shown in the video stream is at risk of failing to comply with the cleaning protocol criteria, the cleaning wizard may notify the operator to take corrective action, such as applying cleaning agent to missed surface areas or reapplying cleaning agent to areas approaching insufficient wetness. If the cleaning wizard determines that the cleaning shown in the video stream has failed to comply with the cleaning protocol criteria, the cleaning wizard may notify the operator to restart part of or the entire cleaning protocol. If the cleaning wizard determines that the cleaning shown in the video stream meets the cleaning protocol criteria, the cleaning wizard may notify the operator that the cleaning protocol is complete. The present invention provides, for example, the following items. (Item 1) 1. A method comprising: receiving a video stream; inputting a first set of frames of the video stream into a first machine learning model, the first machine learning model being trained to output whether the first set of frames corresponds to an activity that initiates a cleaning protocol for an item of medical equipment; inputting a second set of frames from the video stream into a second machine learning model in response to receiving an output that the first set of frames corresponds to an activity that initiates the cleaning protocol, wherein the second machine learning model is trained to output whether the second set of frames satisfies a criterion of the cleaning protocol; In response to the second set of frames satisfying criteria of the cleaning protocol, transmitting a notification to an operator that the cleaning protocol is complete. A method comprising: (Item 2) 2. The method of claim 1, wherein the cleaning protocol criteria include a defect value for an aspect of the cleaning protocol. (Item 3) The second machine learning model generates an output for each frame of the set of frames, and the method further comprises: analyzing a given output of the second machine learning model corresponding to a given frame of the set of frames; In response to the given output being within a threshold range of the defect value, transmitting a notification to the operator to correct the defect. The method according to item 2, comprising: (Item 4) monitoring an output of the second machine learning model corresponding to a frame subsequent to the given frame; transmitting a notification to the operator to restart the cleaning protocol in response to at least one of the monitored outputs failing to satisfy the fault value; and The method according to item 2, further comprising: (Item 5) Item 10. The method of item 1, wherein the first machine learning model is a proposal network. (Item 6) Item 10. The method of item 1, wherein the first machine learning model is trained to output that the cleaning protocol is initiated in response to detecting a wiping motion performed with a cloth across a surface of the item of medical equipment in the first set of frames. (Item 7) further, in response to receiving an output that the first set of frames corresponds to an activity that initiates the cleaning protocol, determining a location of the item of medical equipment within the video stream relative to a camera capturing the video stream; In response to the item of medical equipment being positioned outside a threshold distance relative to the camera, transmitting a notification to the operator to reposition the item of medical equipment; Item 1, the method of claim 1 further comprising: (Item 8) Item 10. The method of item 1, wherein the video stream is captured by a camera mounted on or within the item of medical equipment. (Item 9) Item 10. The method of item 1, wherein the output of the second machine learning model includes a pixel map, the pixel map representing one or more aspects of the cleaning protocol based on the second set of frames. (Item 10) inputting the pixel map into one or more classifiers of the second machine learning model; receiving as output from each of the one or more classifiers whether a pixel of the pixel map conforms to a requirement of the cleaning protocol; determining that the second set of frames meets the criteria of the cleaning protocol in response to pixels of the pixel map conforming to the required values of the cleaning protocol; Item 10. The method of item 9, further comprising: (Item 11) 11. The method of claim 10, wherein the cleaning protocol includes a wettability requirement for the surface of the item of medical equipment. (Item 12) Initiating the cleaning protocol for the item of medical equipment further comprises: determining a type of cleaning agent to be used during the cleaning protocol based at least in part on the first set of frames; and adjusting the criteria of the cleaning protocol based on the determined type of cleaning agent; Item 1. The method according to item 1, comprising: (Item 13) A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium including a memory with instructions encoded thereon, the instructions executable by one or more processors to perform operations, the instructions comprising: receiving a video stream; inputting a first set of frames of the video stream into a first machine learning model, the first machine learning model being trained to output whether the first set of frames corresponds to an activity that initiates a cleaning protocol for an item of medical equipment; inputting a second set of frames from the video stream into a second machine learning model in response to receiving an output that the first set of frames corresponds to an activity that initiates the cleaning protocol, wherein the second machine learning model is trained to output whether the second set of frames satisfies a criterion of the cleaning protocol; In response to the second set of frames satisfying criteria of the cleaning protocol, transmitting a notification to an operator that the cleaning protocol is complete. A non-transitory computer-readable storage medium comprising instructions for performing (Item 14) Item 14. The computer-readable storage medium of item 13, wherein the cleaning protocol criteria include a defect value for an aspect of the cleaning protocol. (Item 15) The second machine learning model generates an output for each frame of the set of frames, and the instructions further include: analyzing a given output of the second machine learning model corresponding to a given frame of the set of frames; In response to the given output being within a threshold range of the defect value, transmitting a notification to the operator to correct the defect. Item 15. The computer-readable storage medium of item 14, comprising instructions for performing (Item 16) The instructions further include: monitoring an output of the second machine learning model corresponding to a frame subsequent to the given frame; transmitting a notification to the operator to restart the cleaning protocol in response to at least one of the monitored outputs failing to satisfy the fault value; and Item 15. The computer-readable storage medium of item 14, comprising instructions for performing (Item 17) Item 14. The computer-readable storage medium of item 13, wherein the first machine learning model is a proposal network. (Item 18) 1. A system comprising: a memory with instructions encoded thereon; one or more processors, wherein the one or more processors, upon executing the instructions, receiving a video stream; inputting a first set of frames of the video stream into a first machine learning model, the first machine learning model being trained to output whether the first set of frames corresponds to an activity that initiates a cleaning protocol for an item of medical equipment; inputting a second set of frames from the video stream into a second machine learning model in response to receiving an output that the first set of frames corresponds to an activity that initiates the cleaning protocol, wherein the second machine learning model is trained to output whether the second set of frames satisfies a criterion of the cleaning protocol; In response to the second set of frames satisfying criteria of the cleaning protocol, transmitting a notification to an operator that the cleaning protocol is complete. one or more processors configured to perform operations including A system comprising: (Item 19) Item 19. The system of item 18, wherein the cleaning protocol criteria include a defect value for an aspect of the cleaning protocol. (Item 20) The second machine learning model generates an output for each frame of the set of frames, and the one or more processors further: analyzing a given output of the second machine learning model corresponding to a given frame of the set of frames; In response to the given output being within a threshold range of the defect value, transmitting a notification to the operator to correct the defect. 20. The system of claim 19, wherein the system is configured to perform operations including: [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram of a system environment in which the cleaning wizard operates, according to one embodiment.

[0010] [Figure 2] 2A-2B are exemplary diagrams of a camera for capturing a video stream for use in monitoring and providing feedback regarding surface cleaning by a cleaning wizard, according to an embodiment.

[0011] [Figure 3] FIG. 3 is a block diagram of the architecture of a cleaning wizard, according to one embodiment.

[0012] [Figure 4] FIG. 4 is an exemplary flowchart of a method for monitoring and providing feedback regarding surface cleaning of a medical surface, according to an embodiment.

[0013] [Figure 5] FIG. 5 is an exemplary diagram of a method for monitoring and providing feedback regarding the surface cleaning of a medical surface, according to an embodiment.

[0014] The figures depict various embodiments for illustrative purposes only. Those skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein. DETAILED DESCRIPTION OF THE INVENTION

[0015] 1 is a block diagram of a system environment 100 for a cleaning wizard 150. The system environment 100 illustrated by FIG. 1 includes a camera 105, a client device 110, a network 115, and a cleaning wizard 150. In alternative configurations, different and / or additional components may be included within the system environment 100. For example, the system environment 100 may include one or more cameras 105 and one or more client devices 110.

[0016] Client device 110 is one or more computing devices capable of receiving user input, displaying information to a viewing user, and transmitting and / or receiving data over network 115. In one embodiment, client device 110 is a conventional computer system such as a desktop computer or laptop computer. Alternatively, client device 110 may be a device having computer functionality, such as a personal digital assistant (PDA), a mobile phone, a smartphone, or another suitable device. Client device 110 is configured to communicate over network 115. In one embodiment, client device 110 executes an application that allows a user of client device 110 to interact with cleaning wizard 150. For example, client device 110 executes a browser application to enable interaction between client device 110 and online system 150 over network 115. In another embodiment, client device 110 interacts with online system 150 through an application programming interface (API) running on the client device's 110's native operating system, such as IOS® or ANDROID®.

[0017] Client device 110 is configured to communicate over network 115, which may comprise any combination of local area networks and / or wide area networks using both wired and / or wireless communication systems. In one embodiment, network 115 uses standard communication technologies and / or protocols. For example, network 115 includes communication links using technologies such as Ethernet, 802.11, Worldwide Interoperable Microwave Access (WiMAX), 3G, 4G, Code Division Multiple Access (CDMA), Digital Subscriber Line (DSL), etc. Examples of networking protocols used for communication over network 115 include Multiprotocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), and File Transfer Protocol (FTP). Data exchanged over network 115 may be represented using any suitable format, such as Hypertext Markup Language (HTML) or Extensible Markup Language (XML). In some embodiments, all or some of the communication links of network 115 may be encrypted using any suitable technique or techniques.

[0018] One or more cameras 105 may also be coupled to network 115 for communication with cleaning wizard 150, described below in conjunction with FIGS. 2A-2B and 3. Camera 105 captures a video stream, including a continuous series of image frames, and transmits the captured video stream to cleaning wizard 150. In one embodiment, camera 105 is a component of client device 110, e.g., a camera on a laptop or desktop computer. In one embodiment, camera 105 is internal to, e.g., built into, or mounted on, an item of medical equipment (e.g., the lens of camera 105 needs to be cleaned and the lens is an item of medical equipment). In another embodiment, the camera may be external to the item of medical equipment and positioned to capture a video stream including the item of medical equipment.

[0019] The cleaning wizard 150 receives video data from one or more cameras 105 and monitors the video stream to determine when a cleaning protocol is initiated by an operator for an item of medical equipment. In one embodiment, the initiation of a cleaning protocol is defined by the initial application of a cleaning agent to the surface of the item of medical equipment. The cleaning agent may include one or more liquids, sprays, wet napkins, wipes, etc., that require the operator to leave the surface sufficiently wet for a certain length of time and / or perform some additional interaction with the surface. The cleaning agent may be applied via wiping, spraying, rinsing, or other methods. Once the cleaning protocol is initiated, the cleaning wizard 150 monitors the video stream of the cleaning attempt to ensure that the cleaning attempt meets all criteria of the cleaning protocol, including, for example, sufficient coverage of the surface of the item of medical equipment, sufficient wetness of the surface of the item of medical equipment, and sufficient duration of time elapsed with the cleaning agent being applied to the surface of the item of medical equipment. During the cleaning protocol, if the cleaning attempt is in danger of failing to comply with the criteria of the cleaning protocol, the cleaning wizard 150 transmits a notification to the operator's client device 110 to correct the deficiencies in the cleaning attempt. When all criteria of the cleaning protocol are met, the cleaning wizard transmits a notification to the operator's client device 110 that the cleaning protocol is complete. Further details of the activities of the cleaning wizard 150 are described below with respect to Figures 2-5.

[0020] 1, the cleaning wizard 150 is remotely hosted and operated entirely on the cloud, for example, via a service provider such as AMAZON WEB SERVICES, MICROSOFT AZURE, or GOOGLE CLOUD PLATFORM. In another embodiment, part or all of the cleaning wizard 150 is implemented locally. For example, part or all of the cleaning wizard 150 is operated on-board one or more cameras 105 and / or one or more client devices 110.

[0021] 2A is an exemplary diagram of a camera 105A positioned externally to capture a video stream relating to an item of medical equipment 210A. In one embodiment, the camera 105A is positioned to capture a video stream including at least the surface of the item of medical equipment 210A. In another embodiment, the camera 105A is positioned to capture a video stream including a surface on which the item of medical equipment is positioned for cleaning (e.g., the camera is positioned to capture a video stream of a table on which a stethoscope is placed for cleaning purposes). When the operator 215A performs a wiping motion with a cloth across the surface of the item of medical equipment 210A in the video stream captured by the camera 105A, the cleaning wizard 150 initiates a cleaning protocol for the item of medical equipment.

[0022] In one embodiment, because the external camera 105A may be moved or positioned inaccurately relative to the item of medical equipment, the cleaning wizard 150 initiates checks on the position of the camera 105A to ensure that the captured video stream accurately represents the cleaning attempt. Thus, in such an embodiment, when a cleaning protocol is initiated, the cleaning wizard 150 may determine the position of the item of medical equipment in the video stream relative to the camera 105A. In response to the item of medical equipment 210A being positioned outside a threshold distance to the camera—e.g., too far from or too close to the camera 105A for accurate analysis of the cleaning attempt to be performed—the cleaning wizard 150 may transmit a notification to the operator to reposition the item of medical equipment. While the cleaning protocol is being performed, the cleaning wizard 150 may additionally perform periodic checks on the position of the camera 105A to ensure that the video stream is not obstructed, e.g., by the operator or by other items or objects, and that the camera is not moved during the cleaning protocol. In response to detecting an occlusion or change in position for an amount of time that exceeds a threshold, the cleaning wizard 150 transmits a notification to the operator to take corrective action.

[0023] FIG. 2B is an exemplary diagram of a camera 105B mounted on or within an item of medical equipment 210B. In the example of FIG. 2B, the camera 105B is mounted inside the item of medical equipment 210B and captures a video stream of the surface of the item of medical equipment through a transparent or translucent panel, such that as an operator 215B performs a wiping motion with a cloth across the surface of the item of medical equipment, the camera 105B captures the wiping motion on the video stream. For example, the medical equipment may be a retinal camera in which a user places their forehead and chin on the medical equipment and gazes through a glass screen. The retinal camera or another internal camera may have the chin rest, forehead rest, and glass screen all partially or completely within its field of view and may monitor the cleaning protocol without the need for an external camera. In other examples, camera 105B may be mounted on an exterior surface or component of the item of medical equipment 210B and may be positioned, attached, or removed by an operator of the item of medical equipment. In another example, camera 105B may capture a video stream of a component of the camera, such as a lens cleaning attempt.

[0024] In some embodiments, multiple cameras, including one or more external cameras 105A and one or more internal cameras 105B, are used to capture video streams of the medical device to ensure that the video streams provide a comprehensive view of the medical device. For example, a retinal camera with internal camera 105B may have a partially obscured view of the chin rest and forehead rest, which must be accurately cleaned, in addition to the lens of the retinal camera itself. One or more external cameras 105A are positioned to capture the blind spot of internal camera 105B. The video streams of internal camera 105B and one or more external cameras 105A are transmitted to cleaning wizard 150. In one embodiment, the video streams are logically stitched together for processing by cleaning wizard 150.

[0025] 3 is a block diagram of the architecture of a cleaning wizard 150, according to one embodiment. The cleaning wizard 150 shown in FIG. 3 includes a video receiving module 305, a protocol store 310, a machine learning model store 315, a cleaning initiation module 320, a wipe detection module 325, a wetness prediction module 330, a residue detection module 335, a cleaning adjustment module 340, and a notification module 345. In other embodiments, the cleaning wizard 150 may include additional, fewer, or different components for various applications. Conventional components such as network interfaces, security features, load balancers, failover servers, management and network operation consoles, and the like are not shown so as not to obscure the details of the system architecture.

[0026] The video receiving module 305 receives the video stream from the camera 105. In some embodiments, the video stream is received continuously, periodically, or at some other programmed interval or regularity. Additionally or alternatively, the video stream may be initiated by an operator prior to initiating a cleaning protocol, for example, at the start of a work day. In some embodiments, the video receiving module 305 transmits a set of frames of the received video stream to the cleaning initiation module 320, the wipe detection module 325, the wetness prediction module 330, and the residue detection module 335. Alternatively or additionally, the video receiving module 305 may store the frames in a shared memory accessible by other modules of the cleaning wizard 150, from which they may then access the frames. In some embodiments, the video stream module 305 receives instructions to store and / or transmit the video stream from the cleaning adjustment module 340.

[0027] The protocol store 310 stores and maintains information describing cleaning protocols. Generally, a cleaning protocol specifies one or more required or failing values for one or more aspects of a cleaning attempt that must be met in order for a cleaning attempt with a given cleaning agent to be successful for a given item of medical equipment to meet, for example, CDC guidelines or manufacturer's mandates for cleaning agents. The required values represent minimum or maximum values required to successfully meet the criteria for the cleaning protocol. The failing values represent minimum or maximum values below which the cleaning protocol fails. In some embodiments, a cleaning protocol may specify a range of acceptable values for one or more aspects of the cleaning protocol, or may specify thresholds below which the cleaning attempt has not yet failed but requires corrective action. A cleaning protocol may specify, for example, the type, amount, or concentration of cleaning agent to be used for the item of medical equipment, the required wetness over the surface of the item of medical equipment, the duration for wiping or maintaining surface wetness, rinsing of the surface, required drying of the surface prior to contact with human skin, and the like.

[0028] The machine learning model store 315 stores one or more models that are trained to output information describing the fulfillment criteria of a cleaning protocol. In one embodiment, one or more models are trained to detect whether a cleaning attempt fulfills the criteria of a cleaning protocol using positive and negative training samples. Positive training samples may be, for example, video data or frames approved by the CDC, a cleaning material manufacturer, or medical professionals as best practices for cleaning medical equipment. Negative training samples may be, for example, video data or frames of movement near an item of medical equipment without cleaning the item, incorrect cleaning of the item of medical equipment, and the like. The positive and negative training samples are labeled to identify clips as, for example, correct cleaning attempts, incorrect cleaning attempts, or no attempts. One or more models may be trained, for example, with different cleaning materials with different manufacturer instructions, different items of medical equipment with different surfaces, and the like. Certain values of the cleaning protocol, such as thresholds for triggering operator notifications, may be adjusted by the model or may be established manually.

[0029] The cleaning initiation module 320 applies a suggestion network to a set of frames from the video stream to determine when a cleaning protocol should be initiated for an item of medical equipment. In one embodiment, the suggestion network is a machine learning model that receives a set of frames from the video stream as data and is trained to output whether the set of frames corresponds to an activity that initiates a cleaning protocol for the item of medical equipment. For example, the suggestion network is trained to output that if the set of frames includes an operator wiping the surface of the item of medical equipment with a cloth or wipe, the set of frames corresponds to a cleaning protocol being initiated. In another example, the suggestion network is trained to output that if the set of frames includes an operator spraying or applying a cleaning agent to the surface of the item of medical equipment, the set of frames corresponds to a cleaning protocol being initiated.

[0030] In one embodiment, the wash initiation module 320 applies the proposal network to a sampled frame or subset of frames of the video stream. For example, the wash initiation module 320 applies the proposal network to every tenth frame received from the camera 105 by the wash wizard 150. In response to a wash attempt being detected by the wash initiation module 320, the wash initiation module 320 transmits a set of frames to the wipe detection module 325 that correspond to a time window surrounding the sampled frame or subset of frames. In one embodiment, the proposal network is a convolutional neural network. In other embodiments, other methods of machine learning may be used.

[0031] In one embodiment, the suggestion network is a lightweight network used prior to heavy processing performed downstream. The use of the suggestion network reduces the processing power required by the cleaning wizard 150 when no cleaning attempts are being made. Because the suggestion network is invoked periodically on individual frames or short sets of frames and therefore does not require extensive video data processing, the cleaning wizard 150 can efficiently identify when a cleaning protocol is initiated by a medical device operator without the need for extensive computing resources, battery use, and the like. If the suggestion network outputs that a cleaning protocol has been initiated, the cleaning wizard 150 may apply one or more heavy networks to confirm that the cleaning protocol has been initiated. Thus, the use of the suggestion network prior to heavy processing improves the overall efficiency of the cleaning wizard 150 while still employing a robust network and eliminating the possibility of false positives that trigger cleaning protocol cycles when they are not needed.

[0032] In response to receiving a set of frames from the initiation cleaning module 320, the wipe detection module 325 applies a wipe detector model to the set of frames from the video stream to determine whether the cleaning agent is being applied correctly. While "wiping" is a common example throughout this disclosure, this is merely exemplary, and any action that cleans a surface may be used in place of "wiping." For example, the wipe detection module 325 may be used to detect, but not wipe, rinse, spray, etc. In such cases, the term "wipe detection module" is not intended to require detection of "wiping," but may be used to detect whatever corresponding cleaning mechanism is relevant.

[0033] Correct application of the cleaning agent may include spatial and temporal parameters, such as whether the cleaning agent is applied to a sufficient area of the surface of the item of medical equipment and / or whether the cleaning agent is applied within a threshold amount of time. In alternative or additional embodiments, the wipe detection module 325 is used to detect any surface cleaning activity, including alternative methods of applying the cleaning agent to a surface. For example, the wipe detection module 325 may detect spraying of the cleaning agent via a spray bottle and / or rinsing of the surface with the cleaning agent, e.g., as running water or a cleaning solution applied to the surface of the item of medical equipment.

[0034] In one embodiment, the wipe detector model is trained using positive and negative training samples. The training samples may include video clips containing various methods for cleaning a surface, including wiping the surface with a cloth, rinsing the surface, spraying a cleaner on the surface, and the like. The training samples may be video clips released, for example, by the CDC or by a cleaner manufacturer, and labeled, for example, for correct application of cleaner, incorrect application of cleaner, or no application of cleaner. The training samples may also be generated by a trained user who intentionally generates examples with various labels, or by a trained labeler who labels video clips from actual cleaning trials. Additionally, these clips can be collected while the cleaning wizard 150 is running and used for ongoing improvement. In one embodiment, the wipe detector model is a 3D convolutional neural network. In other embodiments, other methods of machine learning may be used.

[0035] For example, the wipe detector model may receive as input a set of video frames and a cleaning protocol and be trained to output a binary value representing whether the cleaning agent has been correctly applied to the item of medical equipment. In another example, the wipe detector model may be trained to output a numeric value representing a probability of whether the cleaning agent has been correctly applied. Some cleaning protocols do not require the application of a cleaning agent (e.g., a rinse protocol requiring only water), and for such cleaning protocols, the wipe detector model may be trained to detect whether the appropriate action has been taken (e.g., a frame reflecting that rinsing has begun). The wipe detection module 325 compares the wipe detector model's output probability to a threshold probability to determine whether the cleaning agent has been correctly applied. In another example, the wipe detector model may be trained to output one or more flags representing some aspect of the cleaning protocol, for example, a flag representing whether the cleaning agent has been applied to a sufficient area of the item of medical equipment, a flag indicating whether the cleaning agent has been applied within a threshold amount of time, and / or one or more flags representing the correct application of the cleaning agent to a portion of the item of medical equipment.

[0036] The wipe detection module 325 transmits a notification to the cleaning adjustment module 340 in response to an output by the wipe detector model that includes information describing whether the cleaning agent was applied correctly. In response to the wipe detector model outputting that the cleaning agent was applied incorrectly (or in response to detecting that any requested activity was performed improperly, if wiping or application of cleaning agent is not at issue), the wipe detection module 325 may restart the cleaning attempt or may receive a set of frames as an additional input to the wipe detector model. For example, if a cleaning attempt fails due to an operator failing to apply cleaning agent to a portion of the surface of an item of medical equipment, the operator may be instructed to apply cleaning agent to that portion rather than restarting the cleaning attempt. In addition to inputting the initial set of frames into the wipe detector model, the wipe detection module 325 inputs a set of frames from a video stream of the operator applying cleaning agent to that portion. In another example, if a cleaning attempt fails due to the operator failing to apply enough cleaning agent within the time frame, the operator may be instructed to restart the cleaning attempt. The wipe detection module 325 inputs into the wipe detector model a set of frames from the operator's video stream that restarts the cleaning attempt, without the initial set of frames.

[0037] The wetness prediction module 330 applies a wetness predictor model to a set of frames from the video stream to determine whether sufficient wetness is maintained across the surface of the item of medical equipment during a cleaning attempt. In order for the cleaner to sufficiently disinfect the surface to prevent the spread of infectious diseases, the cleaner must maintain a threshold wetness of the surface for a period of time, often in conjunction with manufacturer instructions. For example, the surface may need to remain wet for three minutes for the cleaner to disinfect the surface.

[0038] In one embodiment, the wetness predictor model is trained using training samples. The training samples may be, for example, frames of video data or images containing a hard surface. The training samples are associated with labels that identify the hard surface as fully wet, poorly wet, nearly dry, or dry. Once frames of video data are labeled, labels for intermediate frames may be interpolated from already labeled frames. For example, if a surface is wet in frames 1 and 10, frames 2-9 may be assumed to be wet. In other embodiments, additional, fewer, or different labels may be used. In some embodiments, portions of a surface within a training sample may be individually labeled such that a first portion may be identified as "fully wet" and a second portion may be identified as "dry." In some embodiments, during training on the training samples, the output pixel map may require smoothness adjustment across spatial and temporal dimensions. This smoothness adjustment may take the form of minimizing the derivative of the pixel map. This adjustment term will be added to the standard neural network loss along with the coefficients.

[0039] In one embodiment, the wetness predictor model is a recurrent convolutional neural network. The wetness predictor model receives as input a set of video frames and a cleaning protocol and outputs a pixel map, where a per-pixel wetness value is generated for each timestamp of the set of frames. In one example, the wetness predictor model applies a smoothing operation to the set of frames from the video stream to reduce the impact of single-pixel errors. In another example, the frames are downsized so that each pixel in the output pixel map represents a region of the set of frames; for example, an initial frame of 10,000 x 10,000 pixels is downsized to generate a pixel map at 512 x 512 pixels. In other embodiments, other methods of machine learning may be used, as may other qualitative or quantitative representations of surface wetness.

[0040] The wetness prediction module 330 determines an overall wetness value for the surface based on the pixel map. In some embodiments, the wetness prediction module 330 receives as input the pixel map and the cleaning protocol and applies one or more classifiers that are trained to output a value representing whether the pixels of the pixel map individually meet the required values (e.g., wetness values) of the cleaning protocol. In some embodiments, the value may be a binary representation of whether the pixels of the pixel map meet the required values of the cleaning protocol. In other embodiments, the value may be a numeric value representing the percentage of pixels of the pixel map that meet the required values of the cleaning protocol or the probability of a pixel of the pixel map individually meeting the required values of the cleaning protocol. In other embodiments, the value may be a numeric value representing the average wetness or minimum wetness of the surface of the medical device. One or more classifiers may be stored in the machine learning model store 315. The wetness prediction module 330 determines whether the overall wetness of the surface of the medical device meets the criteria of the cleaning protocol based on the output of the one or more classifiers.

[0041] In other embodiments, the wetness prediction module 330 determines and evaluates a minimum value of the pixel map to ensure that the minimum value of the pixel map remains above a defined threshold value for the cleaning protocol throughout the required duration. The wetness prediction module 330 evaluates the pixel map corresponding to the current frame of the video stream and one or more pixel maps corresponding to one or more previous frames of the video stream so that the wetness prediction module 330 can alert the cleaning adjustment module 340 when the wetness value decreases over time and approaches a threshold value.

[0042] The residue detection module 335 applies a residue detector model to a set of frames from the video stream to determine whether residue on the surface of an item of medical equipment increases or decreases by a sufficient amount during the cleaning protocol. In some embodiments, the residue detector model is an alternative or addition to the wetness predictor model, in that wettability or droplets of cleaning agent on the surface can be represented as residue. Residue can additionally include, for example, dust or dirt particles, other liquids, and the like.

[0043] In one embodiment, the residue detector model is a recurrent convolutional neural network that outputs a pixel map representing the amount of residue per pixel on the surface of the item of medical device. In some examples, the residue detector model outputs one or more pixel maps representing residue types, such as a pixel map representing liquid residue on the surface and a pixel map representing dust particles on the surface, where the pixel map representing liquid residue on the surface is evaluated by the residue detection module 335 against a different threshold or criterion than the pixel map representing dust particles on the same surface.

[0044] The cleaning adjustment module 340 receives information describing the outputs of the machine learning models from the wipe detection module 325, the wetness prediction module 330, and the residue detection module 335 and determines a signal based on the received information. The cleaning adjustment module 340 monitors the outputs from the machine learning models, and in response to at least one monitored output being within a threshold range of the defect value (e.g., a wetness value of 0.3 approaching a defect value of 0.2), the cleaning adjustment module signals the notification module 345 to notify an operator to correct the defect. In response to at least one monitored output failing to meet a defect value (e.g., a wetness value of 0.1 below a defect value of 0.2 due to an operator failing to apply the cleaning agent in time), the cleaning adjustment module 340 signals the notification module 345 to notify the operator to resume the cleaning protocol and signals the wipe detection module 325 to resume the cleaning protocol. In response to all monitored outputs meeting the criteria of the cleaning protocol, the cleaning adjustment module 340 signals the notification module 345 to notify the operator that the cleaning protocol is complete.

[0045] In some embodiments, the wash tuning module 340 additionally identifies a wash protocol for use in the wash trial. Because wash protocols can vary between cleaners and items of medical equipment, the wash tuning module 340 determines, in response to an operator's initiation of a wash trial, at least one of the type of cleaner being used during the wash trial and the item of medical equipment being cleaned. For example, the wash tuning module 340 may apply a model trained to identify the type of cleaner and the item of medical equipment from the visual data based, for example, on brand names or labels appearing in the video stream, the method of application of the cleaner, the spatial shape of the medical equipment, and the like. In other embodiments, the wash tuning module 340 receives, as input from the operator, information defining the type of cleaner and the item of medical equipment being cleaned. Based on the information, the wash tuning module 340 accesses the protocol store 310 and selects a wash protocol to be applied to the wash trial. In some embodiments, the cleaner may simply be water (e.g., as required in a rinse protocol).

[0046] The notification module 345 transmits notifications to one or more client devices 110 to provide feedback during cleaning attempts of an item of medical equipment. The notification module 345 communicates with the operator's client devices 110 implementing the cleaning protocol via the network 115. In response to signals from the cleaning adjustment module 340, the notification module 345 provides textual or graphical notifications to the operator to implement corrective action during the cleaning attempt, to restart the cleaning attempt, or upon successful completion of the cleaning protocol. Exemplary notifications are further discussed in conjunction with FIG. 5.

[0047] In alternative embodiments, one or more modules may be omitted to streamline the functionality of the cleaning wizard 150. For example, in one alternative embodiment, the start cleaning module 320 and the suggestion network are omitted. In this embodiment, the wipe detection module 325 may be prompted to activate in response to the cleaning adjustment module 340 signaling that the operator is being prompted to reapply cleaning agent. Alternatively, the wipe detection module 325 is prompted to activate throughout the cleaning protocol, for example, in response to the cleaning wizard 150 being started or turned on.

[0048] In another alternative embodiment, the wipe detection module 325 and wipe detector model are omitted. In this embodiment, the cleaning initiation module 320 applies a proposal network to determine when a cleaning attempt should be initiated, and the wetness predictor model 330 applies a wetness predictor model to determine when a portion of the surface will become too dry or did not have the cleaning agent properly applied in the initial application. This embodiment can also be used when a non-wiping method, such as rinsing, is used to apply the cleaning agent.

[0049] In another alternative embodiment, the wetness prediction model 330 and the wetness predictor model are omitted. In this embodiment, the cleaning adjustment module 340 signals the notification module 345 to notify the operator to reapply the cleaning agent at periodic intervals after the initial application of the cleaning agent. The periodic intervals are timed to maintain a wetness value on the surface of the item of medical equipment for sufficient application of the cleaning agent. If sufficient application is not detected by the wipe detection module 340 during each interval (e.g., an interval is missed or a portion of the surface is missed), the cleaning adjustment module 340 signals the notification module 345 to notify the operator that the cleaning protocol has failed and must be restarted.

[0050] 4 is an exemplary flowchart of a method for monitoring and providing feedback regarding surface cleaning of a medical surface, according to an embodiment. The steps of FIG. 4 may be performed by the cleaning wizard 150, as described below. Some or all of the steps may be performed by other modules in other embodiments. Additionally, other embodiments may include different and / or additional steps, and the steps may be performed in a different order.

[0051] The video receiving module 305 of the cleaning wizard 150 receives 405 the video stream. The initiating cleaning module 320 inputs 410 a first set of frames of the video stream into a first machine learning model. The first machine learning model is stored in the machine learning model store 315 and trained to output whether the first set of frames corresponds to an activity that initiates a cleaning protocol for the item of medical equipment. One or more cleaning protocols may be stored by the cleaning wizard 150 in the protocol store 310 and accessed by the initiating cleaning module 320. In one example, the first machine learning model is trained to output that a cleaning protocol is initiated in response to detecting a wiping motion performed with a cloth across the surface of the item of medical equipment in the first set of frames. In another example, the first machine learning model is trained to output that a cleaning protocol is initiated in response to detecting a cleaning agent being sprayed onto the surface of the item of medical equipment in the first set of frames.

[0052] In response to the cleaning protocol being initiated, the cleaning wizard 150 inputs (415) a second set of frames of the video stream into a second machine learning model stored in the machine learning model store 315. The second machine learning model is trained to output whether the second set of frames meets the criteria of the cleaning protocol. The cleaning protocol may specify one or more deficiency values for aspects of the cleaning attempt that will cause the cleaning attempt to fail and / or requirement values for aspects of the cleaning attempt that must be met for the cleaning attempt to be successful. The cleaning wizard's cleaning adjustment module 340 may signal one or more modules to apply one or more machine learning models based on the specified aspects of the cleaning attempt. For example, the wetness prediction module 330 inputs the second set of frames into a machine learning model trained to output a representation of the surface wetness of the item of medical equipment during the cleaning attempt. Alternatively, or in addition, the wipe detection module 325 inputs the second set of frames into a machine learning model trained to output whether a cleaning agent was accurately applied to the item of medical equipment. Alternatively, or in addition, the residue detection module 335 inputs the second set of frames into a machine learning model that is trained to output a representation of residue on one or more surfaces of the item of medical equipment during the cleaning attempt.

[0053] In some embodiments, the second machine learning model may additionally or alternatively comprise one or more classifiers trained to output whether aspects of the cleaning attempt captured in the second set of frames conform to criteria of the cleaning protocol. For example, the second machine learning model determines whether criteria such as surface wetness, surface residue, and surface coverage by spraying or wiping remain within the correct ranges over the duration of the second set of frames. For each aspect of the cleaning protocol, the one or more classifiers output whether the pixels of the pixel map conform to the required values of the cleaning protocol.

[0054] In response to the output of the second machine learning model being within a threshold range of the cleaning protocol's defect value, e.g., the surface having a surface wetness value of 0.3 for a defect value of 0.2, the cleaning adjustment module 340 of the cleaning wizard 150 signals the notification module 345 to transmit a notification to an operator to correct the defect. For example, the notification may instruct the operator to reapply a cleaning agent to at least a portion of the surface of the item of medical equipment. In response to the output of the second machine learning model failing to meet the cleaning protocol's defect value, e.g., the surface having a surface wetness value of 0.1 for a defect value of 0.2, the cleaning adjustment module 340 signals the notification module 345 to transmit a notification to an operator to restart the cleaning protocol.

[0055] In response to the output of the second machine learning model satisfying the criteria of the cleaning protocol, the cleaning adjustment module 340 signals the notification module 345 to transmit (420) a notification to the operator that the cleaning protocol has been successfully completed.

[0056] FIG. 5 is an exemplary diagram of a method for monitoring and providing feedback regarding the surface cleaning of a medical surface, according to an embodiment. A camera 105 captures a video stream of an item of medical equipment and transmits the video stream to a cleaning wizard 150. An operator of the item of medical equipment initiates a cleaning protocol 505A by wiping the surface of the item of medical equipment with a cloth. In response to determining that the cleaning protocol has been initiated, the cleaning wizard 150 determines a requirement for the cleaning protocol to be successfully completed. In the example of FIG. 5, the requirement for the cleaning protocol specifies that the surface of the item of medical equipment maintains a surface wetness for three minutes for the surface to be adequately cleaned. In other examples, other requirements and other aspects of the cleaning protocol may be specified.

[0057] In response to the surface of the item of medical equipment maintaining the required surface wetness for the three minutes (510A), the cleaning wizard completes the cleaning protocol and transmits to the operator's client device 110 a notification 515 that the cleaning protocol is complete, e.g., "Cleaning Complete" and one or more visual elements indicating successful completion of the cleaning protocol.

[0058] In response to the cleaning wizard 150 determining that the surface of the item of medical equipment fails to maintain surface wetness for three minutes (520), the cleaning wizard transmits a notification 525 to the operator's client device 110. The notification 525 provides instructions for the operator to restart the cleaning protocol, e.g., "Reapply cleaning agent to lens. Wait three minutes before use." The operator of the item of medical equipment restarts the cleaning protocol 505B by reapplying cleaning agent to the surface of the item of medical equipment. In response to the cleaning wizard 150 determining that the cleaning protocol has been restarted (505B), the cleaning wizard monitors the cleaning attempt to ensure that surface wetness is maintained for the required three minutes. In response to surface wetness being maintained for the required three minutes (510B), the cleaning wizard 150 completes the cleaning protocol and transmits a notification 515 to the operator's client device 110 that the cleaning protocol is complete.

[0059] The foregoing description of the present embodiments has been presented for purposes of illustration and is not intended to be exhaustive or to limit the patent to the precise form disclosed. Those skilled in the art will recognize that numerous modifications and variations are possible in light of the above disclosure.

[0060] Some portions of this description will describe embodiments in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. These operations, while described functionally, computationally, or theoretically, will be understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Further, it has also proven convenient at times, without loss of generality, to refer to arrangements of these operations as modules. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0061] Any of the steps, operations, or processes described herein may be performed or implemented with one or more hardware or software modules, alone or in combination with other devices. In one embodiment, the software modules are implemented using a computer program product comprising a computer-readable medium containing computer program code that can be executed by a computer processor to perform any or all of the described steps, operations, or processes.

[0062] Embodiments may also relate to apparatus for performing the operations herein. The apparatus may be specially constructed for the required purposes and / or may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored therein. Such a computer program may be stored in a non-transitory, tangible computer-readable storage medium or any type of medium suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing system referred to herein may include a single processor or may be an architecture employing a multiple processor design for increased computing power.

[0063] Embodiments may also relate to products produced by the computing processes described herein. Such products may comprise information resulting from the computing processes, which information may be stored on a non-transitory, tangible computer-readable storage medium and may include any embodiment of a computer program product or other data combination described herein.

[0064] Finally, the language used herein has been selected primarily for readability and indicative purposes, and may not have been selected to delimit or define the scope of the present patent. Accordingly, it is intended that the scope of the present patent be limited not by this detailed description, but rather by any claims issuing on an application based thereon. Accordingly, the disclosure of the embodiments is intended to illustrate, but not limit, the scope of the present patent, which is set forth in the following claims.

Claims

1. A method comprising: receiving a video stream; inputting a first set of frames of the video stream into a first machine learning model, the first machine learning model being trained to output whether the first set of frames corresponds to an activity that initiates a cleaning protocol for an item of medical equipment; inputting a second set of frames from the video stream into a second machine learning model in response to receiving an output that the first set of frames corresponds to an activity that initiates the cleaning protocol, the second machine learning model being trained to output whether the second set of frames satisfies criteria of the cleaning protocol; In response to the second set of frames not satisfying the criteria of the cleaning protocol, transmitting a notification to an operator to restart the cleaning protocol. A method comprising:

2. The method described in claim 1, wherein the criteria for the cleaning protocol include a defect value representing a minimum or maximum value at which the cleaning protocol fails.

3. The second machine learning model generates an output for each frame of the set of frames, and the method further comprises: analyzing a given output of the second machine learning model corresponding to a given frame of the set of frames; In response to the given output being within a threshold range of the defect value, transmitting a notification to the operator to correct a defect in an attempt to clean the item of medical equipment. The method of claim 2 , comprising:

4. monitoring an output of the second machine learning model corresponding to a frame subsequent to the given frame; transmitting a notification to the operator to restart the cleaning protocol in response to at least one of the monitored outputs failing to meet the fault value; and The method of claim 3 further comprising:

5. The method described in claim 1, wherein the first machine learning model is a lightweight network that is periodically launched on individual frames or short sets of frames to identify when a cleaning protocol should be initiated prior to heavy processing being performed downstream.

6. The method of claim 1, wherein the first machine learning model is trained to output that the cleaning protocol is initiated in response to detecting a wiping motion performed with a cloth across the surface of the item of medical equipment within the first set of frames.

7. Further in response to receiving an output that the first set of frames corresponds to an activity that initiates the cleaning protocol, determining a position of the item of medical equipment within the video stream relative to a camera capturing the video stream; in response to the item of medical equipment being positioned outside a threshold distance relative to the camera, transmitting a notification to the operator to reposition the item of medical equipment; The method of claim 1 further comprising:

8. The method of claim 1, wherein the video stream is captured by a camera mounted on or within the item of medical equipment.

9. The method of claim 1, wherein the output of the second machine learning model includes a pixel map, the pixel map representing the amount of residue or type of residue per pixel on the surface of the item of medical equipment.

10. The method described in claim 9, wherein the cleaning protocol includes a required value for the wetness of the surface of the item of medical equipment.

11. Initiating the cleaning protocol for the item of medical equipment further comprises: determining a type of cleaning agent to be used during the cleaning protocol based at least in part on the first set of frames; and adjusting the criteria of the cleaning protocol based on the determined type of cleaning agent; The method of claim 1 , comprising:

12. A non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium including a memory with instructions encoded thereon, the instructions executable by one or more processors to perform operations, the instructions comprising: receiving a video stream; inputting a first set of frames of the video stream into a first machine learning model, the first machine learning model being trained to output whether the first set of frames corresponds to an activity that initiates a cleaning protocol for an item of medical equipment; inputting a second set of frames from the video stream into a second machine learning model in response to receiving an output that the first set of frames corresponds to an activity that initiates the cleaning protocol, the second machine learning model being trained to output whether the second set of frames satisfies criteria of the cleaning protocol; In response to the second set of frames not satisfying the criteria of the cleaning protocol, transmitting a notification to an operator to restart the cleaning protocol. A non-transitory computer-readable storage medium comprising instructions for performing

13. A computer-readable storage medium as described in claim 12, wherein the criteria for the cleaning protocol include a defect value representing a minimum or maximum value at which the cleaning protocol fails.

14. The second machine learning model generates an output for each frame of the set of frames, and the instructions further comprise: analyzing a given output of the second machine learning model corresponding to a given frame of the set of frames; In response to the given output being within a threshold range of the defect value, transmitting a notification to the operator to correct a defect in an attempt to clean the item of medical equipment.

14. The computer-readable storage medium of claim 13, comprising instructions for:

15. The instruction further comprises: monitoring an output of the second machine learning model corresponding to a frame subsequent to the given frame; transmitting a notification to the operator to restart the cleaning protocol in response to at least one of the monitored outputs failing to meet the fault value; and 15. The computer-readable storage medium of claim 14, comprising instructions for:

16. The computer-readable storage medium of claim 12, wherein the first machine learning model is a lightweight network that is periodically triggered on individual frames or short sets of frames to identify when a cleaning protocol should be initiated prior to heavy processing being performed downstream.

17. A system comprising: a memory with instructions encoded thereon; one or more processors, wherein the one or more processors, upon executing the instructions, receiving a video stream; inputting a first set of frames of the video stream into a first machine learning model, the first machine learning model being trained to output whether the first set of frames corresponds to an activity that initiates a cleaning protocol for an item of medical equipment; inputting a second set of frames from the video stream into a second machine learning model in response to receiving an output that the first set of frames corresponds to an activity that initiates the cleaning protocol, the second machine learning model being trained to output whether the second set of frames satisfies criteria of the cleaning protocol; In response to the second set of frames not satisfying the criteria of the cleaning protocol, transmitting a notification to an operator to restart the cleaning protocol. one or more processors configured to perform operations including A system comprising:

18. The system described in claim 17, wherein the criteria for the cleaning protocol include a defect value representing a minimum or maximum value at which the cleaning protocol fails.

19. The second machine learning model generates an output for each frame of the set of frames, and the one or more processors further: analyzing a given output of the second machine learning model corresponding to a given frame of the set of frames; In response to the given output being within a threshold range of the defect value, transmitting a notification to the operator to correct a defect in an attempt to clean the item of medical equipment.

20. The system of claim 18, wherein the system is caused to perform operations including:

Images