Method, system, and computer readable medium for automated evaluation of aseptic technique for compounding in a compounding hood

Abstract

A method for automated assessment of aseptic techniques for compounding in a compounding hood includes collecting data capable of determining the position, orientation, and movement of objects used in the aseptic compounding task using sensors positioned in or around the compounding hood. The method further includes providing the data to an automated aseptic techniques evaluator. The method further includes identifying stages of the aseptic compounding task from the data using the automated aseptic techniques evaluator. The method further includes automatically detecting, by the automated aseptic techniques evaluator, errors occurring during at least some of the stages. The method further includes generating and displaying, by the automated aseptic techniques evaluator, an output indicative of the errors.

Description

[Technical Field]

[0001] Priority claims This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 393,766, filed July 29, 2022, the entire disclosure of which is incorporated herein by reference.

[0002] Technical Field The subject matter described herein relates to tracking aseptic technique during compounding within a compounding hood, defined as a biological safety cabinet, laminar airflow workstation, or other similar device. More particularly, the subject matter described herein relates to automated tracking and characterization of aseptic technique for compounding within a compounding hood. [Background technology]

[0003] background In 2012, a tragic sterile compounding incident garnered national attention. The New England Compounding Center (NECC) compounded and distributed sterile methylprednisolone injections to 23 different states. Unfortunately, these medications became contaminated with a fungus, causing meningitis in patients who received the compounded product, making over 800 people ill and resulting in 76 deaths. 1 In addition to these shocking cases, there have been many other tragic compounding incidents that have not received the same amount of attention. 2-9 Since the 2012 NECC meningitis outbreak, the FDA has increased its oversight of compounding pharmacies. Prior to that incident, these operations were overseen by state boards of pharmacy and rarely inspected against U.S. Pharmacopeia guidance documents. In 2013, the Drug Quality and Safety Act was passed, expanding the authority given to the FDA to oversee compounding. This amended the Federal Food, Drug, and Cosmetic Act to create two categories of compounding products based on the patient's risk level: compounding pharmacies (Section 503a) and outsourcing facilities (Section 503b). Since the enactment of this law, understanding the various regulations, preparing pharmacies for compliance, and educating employees on proper aseptic technique have become major focuses for all employers. This includes both hospital and community pharmacies.

[0004] However, despite this increased attention, there continue to be problems with compounded products, as documented by the FDA through inspections. A warning letter (FDA Form 483) issued to the compounding pharmacy on February 5, 2019, indicated that FDA inspectors found serious deficiencies in the preparation of sterile drug products. Some of the violations noted from this inspection included: 10 Aseptic personnel were observed to have factors impeding primary air movement over and around opened vials.

[0005] Media filling was not performed under the most difficult or stressful conditions. Personnel engaged in aseptic processing were observed with exposed hands and wrists in the ISO5 area.

[0006] · Inspectors found a "lack of assurance" in the sterile manufacturing of pharmaceutical products at the facility. To prevent the compounding of contaminated products, an organization must have appropriate facilities, policies, and procedures, and, importantly, the ability to train, evaluate, and maintain employee competency in aseptic technique. While facilities and policies are easily evaluated, procedures to ensure that pharmacists and technicians (all employees) always follow strict aseptic technique for all preparations are difficult. The usual method of training, evaluating, and maintaining competency is for employees to be observed while performing various adjustments. Over time, employees are approved as competent and can then prepare compounded sterile products with little or no supervision. Observation has been proven to be an insufficient technique for ensuring compounding accuracy. 11 Each employee must also complete an annual media fill test, which simulates a different compound preparation. 12 These tests involve producing different products using tryptic soy broth as the medium. If employees demonstrate poor technique, the medium will become contaminated. While this concept seems reasonable, studies have demonstrated the difficulty of producing positive results. This study involved assessing the sensitivity and specificity of the medium fill test when poor technique is used. 13 A total of 250 mock compound preparations were prepared. The first run (25 preparations) followed best practice aseptic technique and sterile compounding procedures. Each of the four subsequent sets of runs eliminated an aspect of best practice aseptic technique, to the point that preparations were made without alcohol to sterilize vial septa, preparers did not use gloves, preparations were made outside the compounding hood, and uncapped vials were left in "dirty" air for 24 hours before preparation. Even utilizing such poor conditions, not a single preparation showed signs of turbidity, sedimentation, or visible microbial growth. A 0% contamination rate was recorded. If the only options for ensuring employee proficiency are through observation and media fill testing, both of which have been demonstrated to be insufficient markers of proper compounding processes, the false impression exists and the incorrect product likely reaches patients. 11,13 There is also a requirement that all pharmacy students, not just compounding pharmacy employees, be trained in aseptic technique. The 2016 Accreditation Council for Pharmacy Education (ACPE) Standards state that all pharmacy schools must provide aseptic education and training in: "the preparation of sterile and non-sterile prescriptions that are pharmaceutically accurate with respect to drug substance and dose, free from contamination, and properly formulated for safe and effective patient use." 14 The methods used and the amount of time focused on this important skill vary; a recent survey found that "only 59% of schools believe their students are adequately trained in compounding sterile preparations." 15 Summary of the Invention [Problem to be solved by the invention]

[0007] In light of these and other difficulties, there is a need for improved methods, systems, and computer-readable media for the automated evaluation of aseptic technique of compounding in a compounding hood. [Means for solving the problem]

[0008] overview A method for automated assessment of aseptic techniques for compounding in a compounding hood includes collecting data capable of determining the position, orientation, and movement of objects used in the aseptic compounding task using sensors positioned in or around the compounding hood. The method further includes providing the data to an automated aseptic techniques evaluator. The method further includes identifying stages of the aseptic compounding task from the data using the automated aseptic techniques evaluator. The method further includes automatically detecting, by the automated aseptic techniques evaluator, errors occurring during at least some of the stages. The method further includes generating and displaying, by the automated aseptic techniques evaluator, an output indicative of the errors.

[0009] According to one aspect of the subject matter described herein, collecting data using sensors positioned in or around the blending hood includes collecting data using cameras or other sensors positioned in or around the blending hood. Examples of other types of sensors that may be used include thermal, infrared, and acoustic sensors.

[0010] According to another aspect of the subject matter described herein, collecting data using sensors positioned in or around the blending hood includes collecting data using Internet of Things (IoT) sensors located on the blending equipment or the blending ingredients.

[0011] According to another aspect of the subject matter described herein, identifying the stage includes applying a computer vision-based object detection model to identify the object.

[0012] According to another aspect of the subject matter described herein, identifying the stage includes using a computer vision-based pose estimation model to create time series data representing the position, orientation, and motion of the object at different times.

[0013] According to another aspect of the subject matter described herein, identifying the stages includes utilizing task stage identification heuristics to identify each stage.

[0014] According to another aspect of the subject matter described herein, automatically detecting errors includes applying a classification model that generates a probability score indicating the likelihood of an error occurring at each stage based on the time series data and the predicted position, orientation, and movement of the object, and determining that an error has occurred if the score exceeds a threshold that distinguishes between the actual position, orientation, and movement of the object and the predicted position, orientation, and movement.

[0015] According to another aspect of the subject matter described herein, generating and displaying an output includes generating and displaying a dashboard interface that shows errors that occurred during performance of the sterile compounding task.

[0016] According to another aspect of the subject matter described herein, the dashboard interface displays best practice recommended actions for the student / user to perform during the sterile compounding task in real time while the sterile compounding task is being performed.

[0017] According to another aspect of the subject matter described herein, the automated aseptic techniques evaluator is implemented using at least one trained machine learning classifier.

[0018] According to another aspect of the subject matter described herein, there is provided a system for automated evaluation of aseptic technique of compounding in a compounding hood. The system includes at least one processor. The system further includes a plurality of sensors positionable in or around the compounding hood for collecting data from which the position, orientation, and movement of objects used in the aseptic compounding task can be determined. The system further includes an automated aseptic technique evaluator implemented using the at least one processor for receiving the data, identifying stages of the aseptic compounding task from the data, automatically detecting errors occurring during at least some of the stages, and generating and displaying an output indicative of the errors.

[0019] In accordance with another aspect of the subject matter described herein, the sensor comprises a camera or other optical sensor positionable in or around the blending hood.

[0020] According to another aspect of the subject matter described herein, the sensor comprises an Internet of Things (IoT) sensor that can be disposed on the dispensing device or the dispensing material.

[0021] According to another aspect of the subject matter described herein, the automated aseptic techniques evaluator is configured to identify objects using a computer vision-based object detection model.

[0022] According to another aspect of the subject matter described herein, the automated aseptic techniques evaluator is configured to use a computer vision-based pose estimation model to generate time series data representing the position and orientation of an object at different times.

[0023] According to another aspect of the subject matter described herein, the automated aseptic techniques evaluator is configured to identify each stage utilizing task stage identification heuristics.

[0024] According to another aspect of the subject matter described herein, the automated aseptic technique evaluator is configured to detect errors by applying a classification model that generates a probability score indicating the likelihood of an error occurring at each stage, and determining that an error has occurred if the score exceeds a threshold.

[0025] According to another aspect of the described subject matter, the automated aseptic techniques evaluator is configured to generate a dashboard interface that shows errors that occur during the performance of a sterile compounding task.

[0026] According to another aspect of the subject matter described herein, the automated aseptic techniques evaluator is implemented using at least one trained machine learning classifier.

[0027] According to another aspect of the subject matter described herein, a non-transitory computer-readable medium having stored thereon executable instructions that, when executed by a processor of a computer, control the computer to perform steps including collecting data using sensors positioned in or around a compounding hood from which position, orientation, and movement of objects used in a sterile compounding task can be determined, the steps further including providing the data to an automated aseptic techniques evaluator, the steps further including identifying stages of the sterile compounding task from the data using the automated aseptic techniques evaluator, the steps further including automatically detecting, by the automated aseptic techniques evaluator, errors occurring during at least some of the stages, and the steps further including generating and displaying, by the automated aseptic techniques evaluator, an output indicative of the errors.

[0028] The subject matter described herein can be implemented in software in combination with hardware and / or firmware. For example, the subject matter described herein can be implemented in software executed by a processor. In one exemplary embodiment, the subject matter described herein can be implemented using a non-transitory computer-readable medium having stored thereon computer-executable instructions that, when executed by a computer processor, control the computer to perform steps. Exemplary computer-readable media suitable for implementing the subject matter described herein include non-transitory computer-readable media such as disk memory devices, chip memory devices, programmable logic devices, and application-specific integrated circuits. In addition, computer-readable media implementing the subject matter described herein can be located on a single device or computing platform or can be distributed across multiple devices or computing platforms.

[0029] BRIEF DESCRIPTION OF THE DRAWINGS Exemplary embodiments of the subject matter described herein will now be described with reference to the accompanying drawings. [Brief explanation of the drawings]

[0030] [Figure 1] 10A-10C illustrate example images of a blending hood including cameras positioned at different locations within the cabinet to capture video of students / users performing blending tasks. [Figure 2] FIG. 1 is a schematic diagram illustrating exemplary tracking of student / user and equipment motion during a pharmaceutical compounding task, and mapping of the motion to a 3D coordinate system including path tolerances for the movement of the tracked objects. [Figure 3] FIG. 1 is a block diagram of an exemplary system for automated assessment of aseptic technique. [Figure 4] FIG. 1 illustrates an exemplary dashboard interface that may be displayed by an automated aseptic techniques evaluator. [Figure 5] 1 is a flowchart illustrating an exemplary process for automated assessment of aseptic technique. [Figure 6] FIG. 10 shows images of classified objects in a blending hood. [Figure 7A] FIG. 1 illustrates the identification of features of a user's gloved hand in performing a compounding task. [Figure 7B] FIG. 1 illustrates the identification of features of a user's gloved hand in performing a compounding task. [Figure 7C] FIG. 1 illustrates the identification of features of a user's gloved hand in performing a compounding task. DETAILED DESCRIPTION OF THE INVENTION

[0031] Detailed Description To address the above needs, the subject matter described herein includes an automated assessment of aseptic technique for compounding in a compounding hood. In one exemplary embodiment, aseptic technique is assessed by capturing video images of a student / user performing a pharmaceutical compounding task using a camera or other optical sensor positioned in or around the compounding hood. The video images are processed to identify sterile equipment, equipment movements, and student / user actions during compounding. An automated aseptic technique assessor implemented in software assesses the movements and actions using aseptic technique assessment rules, examples of which are described below. The automated aseptic technique assessor includes a dashboard interface that displays assessment results to the student / user.

[0032] In one study, equipment used during aseptic technique was tracked using chips containing accelerometers and gyroscopes placed on sterile equipment (e.g., syringes, vials) to record movement. It was determined that tolerance for aseptic technique did not emerge. Even though students / users are expected to follow the principles of aseptic technique, these steps are generalized and based on decades of opinion, as opposed to a robust body of literature that tests each step to determine performance indicators for what is and is not appropriate. To overcome this issue, subject matter experts were convened in focus groups to determine specific tolerances for each step. A modified Delphi technique was utilized to identify specific tolerances that guide each step of the process when compounding in a compounding hood. This is the first time quantitative criteria have been defined, allowing compliance to be determined, making this a component of the engine that drives automated aseptic technique assessors.

[0033] The automated aseptic technique evaluator can evaluate individuals as they compound sterile products during their normal workflow by documenting their movements and comparing them against established best practices as identified through the tolerances described above. After completing each product preparation, individuals receive a defensible performance report, including any omissions in technique and steps to address for future preparations. This documentation can be used to train new individuals, assess the ongoing competency of current employees, or provide documentation to regulatory agencies (state boards of pharmacy, hospital accreditation agencies, FDA) regarding ongoing quality assurance through aseptic technique. This real-time feedback also guides pharmacists as to whether a product should be remade or not administered to a patient due to a lapse in aseptic technique.

[0034] The need for quantification and assessment of aseptic technique centers around the concept that regulatory and financial pressures focused on compounding intravenous (IV) medications will continue to increase. Hospitals, compounding community pharmacies, and compounding manufacturers not only need to ensure the accuracy and precision of the final preparation, but also need to focus on the environment in which the product is manufactured, the steps individuals followed during compounding, and the maintenance of documentation to satisfy accreditation and regulatory agencies. One of the most critical activities is aseptic technique, the steps individuals use when preparing a product. Any lapse in this process could result in contamination of the final product.

[0035] In one example, an automated aseptic technique evaluator can be implemented using an artificial intelligence (AI) engine trained to determine whether captured video depicts aseptic technique evaluation rules that were followed and generate an output that quantifies the degree to which aseptic technique was followed. The output can be used for training, quality assurance, or other purposes. In one example, data collection and evaluation can include wireless transmission of captured video from a camera to a server, automating a kinematic data pipeline to evaluate the captured video using AI tools, and storing the data for easy access. The system can provide a time-series analysis of the video by calculating specified biomechanical variables (e.g., x, y, z, and pitch, yaw, and roll joint angles) at each time step.

[0036] In one exemplary embodiment, a commercially available USB camera can be placed within a standard compounding hood and used to capture movements during aseptic technique. One type of compounding hood that can be used is a horizontal airflow IV hood. Another type is a vertical airflow IV hood. The cameras can be placed in various locations within the compounding hood, such as the locations shown in FIG. 1 for a horizontal airflow IV hood. For each combination of three cameras, the origin of the motion capture system can be determined. The cameras can be used to capture video of the pharmaceutical compound preparation, and the video can be output in a computer-readable format, such as mpeg (.mp4) format. The subject matter described herein is not limited to using three cameras. For example, more or fewer than three cameras can be used without departing from the scope of the subject matter described herein. In another example, alternative detection methods to optical ones can be used without departing from the scope of the subject matter described herein.

[0037] Once the images are captured, machine learning can be applied to the captured data. Specific movements and actions can be marked so that the algorithm knows to distinguish between better and worse performance. The training itself can be done in cycles. In one exemplary implementation, as each batch of 25-30 videos is captured, the videos can be fed into a machine learning model for continuous refinement. After each refinement, the model can be integrated into a motion capture system-based library to create a generalizable database structure for extracting data and visualizing performance evaluations.

[0038] The automated aseptic technique evaluator may include templates for presenting data. Templates may be reusable across use cases (e.g., evaluating non-sterile compounding techniques or compounding topical or oral solutions (non-IV) and sterile (IV) techniques), but specific data visualization elements may need to be specified to properly search machine-learned results and clearly display them on a dashboard. In one example, motion data output from and post-processed by a motion capture system can be linked to show activity via a dynamic HTML reporting system. For example, a method can be specified to identify movements associated with airflow obstruction due to poor object positioning, relative to actions such as poor technique when extracting liquids or cleaning exposed surfaces. Metrics can be tracked during aseptic work (the arc of the movement, its dwell within or beyond the boundaries, and its accuracy depending on the given aseptic task; see Figure 2). The motion capture system may have the ability to display specific parameters of playback to inform visualizations (e.g., regions of interest, 3D model representations of human movement, etc.). The intent here is to visualize activity and tolerance boundaries, which can be refined by expert review. Data from the motion capture system can be mapped to a coordinate system and displayed along with path tolerances for sterile equipment movements (see Figure 2).

[0039] The automated aseptic technique assessor can display data in a dashboard format (listing all steps and an assessment of how the compounder performed on them) as well as in customized reports. This report can track how compounders performed over time (e.g., as a student / user license) or can be used to compile all compounders in a particular work setting (organization license). This information can be used for ongoing competency assessments to provide to instructors or regulatory authorities.

[0040] The solutions described herein can address both education and training of individuals regarding aseptic technique as well as quality assurance and compliance. The automated aseptic technique evaluator, in one example, may be implemented as a standalone product available via a software license. In another example, the automated aseptic technique evaluator may be integrated with a compounding hood or compounding hood that also includes a camera or other sensor for tracking movement.

[0041] An automated aseptic technique evaluator can be used to evaluate aseptic technique in horizontal and vertical compounding hoods. The automated aseptic technique evaluator can be continuously improved using machine learning. Machine learning can involve identifying an unsupervised learning mechanism to feed data, training the mechanism using a training set (75-80%) of captured .mp4 files and rubric assessments from an independent evaluator, and testing the learning using a test set (20-25%) of captured .mp4 files and comparing the output to the rubric assessment. It is important to note that this is a different type of machine learning than the one presented above. Here, the goal is to learn movement patterns that indicate better or worse performance, which may differ from expert-defined patterns without the need for final annotation. This testing also involves having the mechanism automatically determine tolerances that account for differences in aseptic technique practice, from obvious ones like horizontal vs. vertical airflow to more subtle features such as compounder characteristics. This testing can also include determining the feasibility of different camera configurations, as exemplified by the placement of a three-camera configuration versus a two-camera configuration. Additional features of the automated sterile technique evaluator may include a dashboard for instructor selection of activities and associated tolerances, automatic blurring of identifying details (e.g., faces), real-time or near-real-time capture and evaluation of sterile activities, and incorporation of IoT sensors into devices, compounding hoods, or students / users.

[0042] FIG. 3 is a block diagram illustrating an exemplary system for automated assessment of aseptic technique. Referring to FIG. 3, the system includes one or more sensors 300 disposed within or around a compounding hood 302. The system further includes a computing platform 304 including at least one processor 306 and memory 308. The system further includes an automated aseptic technique evaluator 310, which may be implemented using computer-executable instructions stored in the memory 308 and executed by the processor 306. The sensors 300 may include any suitable sensors for capturing images and movements of sterile equipment, compounds, and students / users within or around the compounding hood 302. In one example, the sensors 300 may include optical sensors such as cameras. In another example, the sensors 300 may include IoT devices integrated within or on sterile equipment such as syringes, vials, etc. The sensors 300 generate output data that is provided to the automated aseptic technique evaluator 310. The automated aseptic technique evaluator 310 assesses aseptic technique using one or more trained machine learning algorithms and generates an output indicative of the quality of aseptic technique performance. In one example, the automated aseptic techniques evaluator 310 may be implemented using a multi-stage machine learning classifier that performs the following operations.

[0043] 1) A computer vision (CV)-based object detection model is applied to each viewpoint video to create 2D or 3D position time series of task-relevant objects in the scene.

[0044] 2) A CV-based pose estimation model is applied to each viewpoint video to create time series data representing the position and orientation of relevant human behavior in the scene.

[0045] 3) Using object detection and pose estimation data, we define (e.g., time-stamp) stages of task performance using a rule set consisting of sterile technique task stage identification heuristics.

[0046] 4) For each identified task stage, apply a classification model that generates a probability score representing the likelihood that any task error from the a priori list occurred during the stage.

[0047] 5) Any errors whose probability score exceeds a set threshold (e.g., the most likely error to have occurred) are recorded and / or presented to the student / user.

[0048] For step 1, an example of a computer vision model is an AI model trained to recognize objects present in a compounding hood during aseptic technique. Examples of such objects are a student / user's hand, a vial, a syringe, chemical compounds, cleaning materials, etc. The position time series data includes the time-stamped location of the object in a coordinate system defined for each camera or other sensor. An algorithm for tracking the position, orientation, and movement of a user's gloved hand is described below.

[0049] For step 2, the pose estimation model estimates the pose and orientation of the object captured by the sensor. For example, the pose estimation model can estimate the orientation and position of a vial as the student / user moves it during a compounding task.

[0050] For step 3, for each identified object, timestamp data is added to the pose estimation data from step 2. Adding the timestamp data to the pose estimation data quantifies the movement of the identified object within the scene. For example, the object may be a syringe, and the tracked movement may be the injection of the needle attached to the syringe into a vial, the movement of the plunger to remove the liquid from the vial, and then the removal of the syringe with the needle intact from the vial. This process must ensure that the syringe with the needle intact faces the primary air and that the student / user's hand does not block the primary air flow toward the syringe or vial with the needle intact. Each step is performed to ensure the sterility of the dispensed product. One example of an aseptic technique task step identification heuristic is a heuristic for identifying compound withdrawal events.

[0051] For step 4, an example of rules that may be used to determine if an error has occurred is now described. This example relates to compounding using compounding food and includes the following steps: Syringe and needle assembly The goal is to prevent contamination from materials entering the compounding hood and ensure that only clean air is present around critical areas when assembling syringes.

[0052] Causes of errors include: · Attaching the needle to the syringe away from the HEPA filter (blocking sterile air at the needle attachment point).

[0053] ·The packaging or exterior of all syringes, needles, IV bags, and materials going into the compounding hood have not been wiped down with a 70% isopropyl solution or pre-soaked alcohol wipes.

[0054] · Positioning an object or hand in a way that blocks airflow to the syringe. · Open the syringe by opening the package towards the dispenser and not the HEPA filter.

[0055] Remove the liquid from the vial. The goal is to ensure the injection site is sterile, inject sterile air into the vial to match the volume withdrawn, and maintain clean air at all transfer sites.

[0056] Causes of errors include: · The inclined side of the syringe should face the dispenser.

[0057] - Not disinfected with alcohol. Inserting the needle outside the 10-110° range (which can contribute to coring).

[0058] · Leaving vials under positive or negative pressure. Do not change needles after each insertion.

[0059] ·Placing an object or hand in a way that blocks airflow to the injection site. · Positioning an object or hand in a way that blocks airflow to an uncapped vial.

[0060] · Positioning an object or hand in a way that blocks airflow to the rubber stopper of the vial. · Rubbing the vial stopper back and forth (which may introduce particulates).

[0061] · Remove the needle cap facing the dispenser to block airflow to the needle. Drawing the wrong amount of air into the syringe.

[0062] Withdraw the liquid but make sure there are air bubbles. · Injection into IV bags.

[0063] The goal is to safely remove air from the needle, infuse medication into the IV bag with clean air, and safely dispose of sharps.

[0064] Causes of errors include: · Ensure that the uncapped syringe is always facing away from the HEPA filter.

[0065] ·Failure to wipe down IV bags or infusion ports. ·Placing an object or hand in a way that blocks airflow to the injection site airflow.

[0066] · Using a needle to puncture outside the intended injection route. Additional areas of focus: o Perform all operations at least 6 inches from the inside and 3 inches from the back of the blend hood.

[0067] Place items in the compounding hood so that nothing gets between the HEPA filter and the sterile items, and avoid extraneous items in the compounding hood.

[0068] Proper placement of materials (needles in sharps container, syringes, paper, alcohol wipes, etc. in waste container).

[0069] The automated aseptic technique evaluator 310 can be trained to identify each of the above-referenced task steps for compounding in a horizontal airflow compounding hood, determine the probability that one or more of the above-referenced errors occurred, and, if the probability exceeds a threshold, determine that an error has occurred and generate an output indicating the errors that occurred (step 5 above). In one example, the output may be in a dashboard format, an example of which is shown in FIG. 4. In FIG. 4, the dashboard interface includes green, yellow, and red status indicators 400, 402, and 404, respectively, which indicate whether a particular compounding task step is performing well, needs improvement, or poorly, depending, for example, on the number and type of errors made. The dashboard interface further includes a compounding time display 406 that indicates the elapsed time since the start of the compounding task. The dashboard interface further includes a counter 408 of the total aseptic operations performed during the compounding task. The dashboard interface further includes a warning counter 410 that indicates the number of warnings generated for the compounding task. The dashboard interface further includes a continuous particle counter 412 that displays continuous particle counts detected by a particle counting sensor that may be integrated into the compounding hood. The interface further includes a best practice recommendation area 414 that displays to the student / user real-time compounding best practice recommended actions that can be taken to improve the current sterile compounding task. The dashboard interface further includes an action log 416 that displays actions taken and logged by the automated aseptic technique evaluator 310 during the compounding task. A status indicator is displayed adjacent to each displayed action, indicating whether the action has been performed, not performed, or still needs to be performed. It should be noted that the interface shown in FIG. 4 is an example, and the automated aseptic technique evaluator 310 may generate an interface having elements that are the same as or different from those shown in FIG. 4 without departing from the scope of the subject matter described herein.

[0070] Table 1, shown below, illustrates exemplary dashboard elements that the automated aseptic techniques evaluator 310 may display to a student / user via the dashboard interface.

[0071] [Table 1]

[0072] According to another aspect of the subject matter described herein, a smart compounding or research hood is provided. The smart compounding or research hood can include a housing forming an enclosure, such as that shown in FIG. 1, for pharmaceutical compounding or research. The smart compounding or research hood can include a fan for generating a desired directional airflow within the hood. The smart compounding or research hood can include one or more sensors for recording data from which the position and / or orientation of an object within the enclosure can be determined. In one example, the sensor can include a video camera for recording video of the object within the enclosure. The smart compounding or research hood can include or be associated with software for determining the position and / or orientation of an object within the hood based on data from the sensors and while a user is performing a compounding or research task. For example, the software can include a gloved hand tracker or other tracking model trained to track the position and orientation of an object within the enclosure. The output of the model can then be used to evaluate the compounding or research task being performed.

[0073] 5 is a flowchart illustrating an exemplary process for automated assessment of aseptic technique. Referring to FIG. 5, in step 500, the process includes using sensors positioned in or around the compounding hood to collect data that can determine the position, orientation, and movement of objects used in the aseptic compounding task. For example, cameras, other optical sensors, and / or IoT devices can be positioned in or around the compounding hood to record video or other data that can determine the identity, movement, and orientation of the objects.

[0074] In step 502, the process includes providing data to an automated aseptic techniques evaluator from which the position, orientation, and movement of an object can be determined. For example, a sensor can provide video data via a wireless or wired interface to the automated aseptic techniques evaluator 310 running on a computing platform to classify and evaluate the data. The video data can be raw video data and timing information associated with each video frame. From this data, the classification, orientation, and movement of the object can be determined.

[0075] In step 504, the process includes using an automated aseptic technique evaluator to identify stages of the sterile compounding task from the data. For example, the automated aseptic technique evaluator 310 can utilize the trained machine learning classifier described above to identify objects and recognize stages of the sterile compounding task. FIG. 6 shows an image of objects in a compounding hood classified using a machine learning classifier. In the illustrated example, the objects surrounded by green boxes have been classified and include a syringe, a vial, and an alcohol wipe. The student / user's hands and a portion of the compounding hood are also shown.

[0076] In step 506, the process includes automatically detecting errors that occur during at least some of the steps by the automated aseptic technique evaluator. For example, the automated aseptic technique evaluator 310 can implement the classifiers described above to determine whether errors occur during each step of the task.

[0077] In step 508, the process includes generating and displaying, by the automated aseptic techniques evaluator, an output indicative of the error. For example, the automated aseptic techniques evaluator 310 may generate and display an output, such as the dashboard interface shown in FIG. 5, that indicates the error detected during the sterile compounding task.

[0078] Gloved Hand Tracking Algorithm As mentioned above, one aspect of the subject matter described herein involves tracking the position, orientation, and movement of a user's gloved hands while performing a sterile compounding task. Tracking gloved hands presents a different challenge than tracking ungloved hands because gloves obscure some of the hand's features, making it more difficult to track the position, orientation, and movement. Performing gloved hand tracking involves several steps to generate a trained model for accurately tracking the position, orientation, and movement of a compounder's hands. One step is data collection and annotation. To train the model, a large dataset of hand movements is collected, ideally by detecting multiple compounders wearing gloves of different colors from multiple perspectives performing different techniques. The data may be from video frames or other types of sensors capable of capturing data from which the position and orientation of an object can be determined. This dataset is separated into individual frames, and then a large selection of frames is annotated with precise information about the hand's position and orientation in 3D space using software that allows for the identification of key positions of the fingers, wrist, and forearm. The annotation consists of keypoint, bounding box, and occlusion annotations performed using the Ground Truth annotation tool from Amazon Web Services, an online portal that allows for manual annotation of images.

[0079] 7A-7C show annotation results for three different frames showing a user's gloved hand performing a blending task. In FIGS. 7A-7C, circles overlaid on the images represent features of the user's hand whose position, orientation, and movement are tracked. For example, circle 700 represents the same tracked location on the back of the user's hand in different frames. Other circles represent similar features, such as knuckles, fingertips, joints, etc.

[0080] One task in selecting data for training a hand tracking model is to select frames or a sequence of frames that show various positions and orientations, including occlusions by or around objects in the environment. The next step is machine learning model development. Machine learning techniques, such as deep learning, are utilized to create the hand tracking model. The hand tracking model is trained using Nvidia's TAO toolkit, which provides a model zoo of pre-trained models for various applications (e.g., object detection, bounding boxes, keypoint estimation). The hand tracking model utilizes a fiducial point estimator as its base. In one example, the base model is fpenet_trainable_v1.0, available from Nvidia NGC. The model is then trained using transfer learning on task-specific video data, i.e., videos of users performing blending tasks within blending hoods.

[0081] The hand tracking model tracks the position, orientation, and movement of a user's gloved hand by identifying the features shown in Figures 7A-7C and tracking those features across frames. The dataset described in the previous paragraph is used to train a model that learns to recognize patterns of hand movement and the association between sensor data and the corresponding hand pose and orientation. The next step is model optimization. The trained model may require iterations and parameter fine-tuning to achieve better performance, reduce tracking error, and improve real-time responsiveness.

[0082] The trained model can be incorporated into an automated aseptic technique evaluator 310, which uses the model to track the position, orientation, and movement of a user's gloved hand as the user performs a compounding task. The automated aseptic technique evaluator 310 can use the tracked movement of the user's hand to identify stages of the aseptic compounding task. For example, the tracked positions of the user's thumb and index finger in Figures 7B and 7C can be used to identify the withdrawal of a compound from a vial using a syringe.

[0083] The disclosure of each of the following references is incorporated herein by reference in its entirety. References

[0084] [Table 2]

[0085] It will be understood that various details of the subject matter described herein can be changed without departing from the scope of the subject matter described herein. Moreover, the foregoing description is by way of example only, and not by way of limitation, as the subject matter described herein is defined by the appended claims.

Claims

1. 1. A method for automated assessment of aseptic technique of compounding in a compounding hood, comprising: using sensors positioned in or around the compounding hood to collect data capable of determining the position, orientation, and movement of objects used in sterile compounding tasks; providing said data to an automated aseptic technique assessor; identifying a stage of the sterile compounding task from the data using the automated aseptic technique evaluator; automatically detecting errors occurring during at least some of said steps with said automated aseptic technique evaluator; generating and displaying, by said automated aseptic technique evaluator, an output indicative of said error; A method comprising:

2. 10. The method of claim 1, wherein collecting the data using the sensor positioned in or around the blending hood comprises collecting the data using a camera or other optical or other type of sensor positioned in or around the blending hood.

3. 10. The method of claim 1, wherein collecting the data using the sensors positioned in or around the blending hood comprises collecting the data using Internet of Things (IoT) sensors located on blending equipment or blending ingredients.

4. The method of claim 1 , wherein identifying the stage comprises applying a computer vision-based or equivalent sensor-based object detection model to identify the object.

5. 5. The method of claim 4, wherein identifying the stage comprises using a computer vision-based or equivalent sensor-based pose estimation model to create time series data representing the position and orientation of the object at different times.

6. The method of claim 5 , wherein identifying the stages comprises utilizing task stage identification heuristics to identify each stage.

7. 10. The method of claim 1, wherein identifying the stage comprises identifying features of the user's gloved hand from the data collected from the sensor and tracking the position, orientation, and movement of the identified features in different frames of the data.

8. 2. The method of claim 1, wherein automatically detecting the error comprises applying a classification model that generates a probabilistic score indicating the likelihood of an error occurring at each stage, and determining that the error has occurred when the score exceeds a threshold.

9. The method of claim 1 , wherein generating and displaying the output comprises generating and displaying a dashboard interface illustrating the errors that occurred during performance of the sterile compounding task.

10. 10. The method of claim 9, wherein the dashboard interface displays best practice recommended actions for a student / user to take during the sterile compounding task in real time while the sterile compounding task is being performed, or displays an evaluation of performance during the sterile compounding task after performance of the sterile compounding task in a batch.

11. 10. The method of claim 1, wherein the automated aseptic techniques evaluator is implemented using at least one trained machine learning classifier.

12. 1. A system for automated assessment of aseptic technique of compounding in a compounding hood, comprising: at least one processor; a plurality of sensors positionable within or around the compounding hood for collecting data capable of determining the position, orientation, and movement of objects used in sterile compounding tasks; an automated aseptic technique evaluator implemented using the at least one processor for receiving the data, identifying stages of the sterile compounding task from the data, automatically detecting errors occurring during at least some of the stages, and generating and displaying an output indicative of the errors; A system comprising:

13. 13. The system of claim 12, wherein the sensor comprises a camera or other optical or other type of sensor positionable in or around the blending hood.

14. 13. The system of claim 12, wherein the sensor comprises an Internet of Things (IoT) sensor that can be placed on a dispensing device or a dispensing material.

15. 13. The system of claim 12, wherein the automated aseptic techniques assesser is configured to identify the objects using a computer vision-based or equivalent sensor-based object detection model.

16. 16. The system of claim 15, wherein the automated aseptic techniques assesser is configured to use a computer vision-based or equivalent sensor-based pose estimation model to create time series data representing the position and orientation of the object at different times.

17. 17. The system of claim 16, wherein the automated aseptic techniques evaluator is configured to identify each stage utilizing task stage identification heuristics.

18. 13. The system of claim 12, wherein the automated aseptic technique evaluator is configured to identify the stage by identifying characteristics of a user's gloved hand from the data collected from the sensor and tracking the position, orientation, and movement of the identified characteristics in different frames of the data.

19. 13. The system of claim 12, wherein the automated aseptic technique evaluator is configured to detect the error by applying a classification model that generates a probabilistic score indicating the likelihood of an error occurring at each stage, and determining that the error has occurred when the score exceeds a threshold.

20. 13. The system of claim 12, wherein the automated aseptic techniques evaluator is configured to generate a dashboard interface illustrating the errors that occurred during performance of the sterile compounding task.

21. 13. The system of claim 12, wherein the automated aseptic techniques evaluator is implemented using at least one trained machine learning classifier.

22. A smart formulation or research food, comprising: a housing defining an enclosure for performing a compounding or research task; a fan for generating airflow in a desired direction within said enclosure during said preparation or research task; at least one sensor disposed in or on the housing for recording data during the compounding or research task from which a position and orientation of an object within the enclosure during the compounding task can be determined; a tracking model trained to determine the position and orientation of the object within the enclosure during the compounding or research task from the data output from the sensor; Equipped with a hood.

23. A non-transitory computer-readable medium having stored thereon executable instructions that, when executed by a processor of a computer, control the computer to perform steps, the steps comprising: using sensors positioned in or around the compounding hood to collect data capable of determining the position, orientation, and movement of objects used in sterile compounding tasks; providing said data to an automated aseptic technique assessor; identifying a stage of the sterile compounding task from the data using the automated aseptic technique evaluator; automatically detecting errors occurring during at least some of said steps with said automated aseptic technique evaluator; generating and displaying, by said automated aseptic technique evaluator, an output indicative of said error; 1. A non-transitory computer-readable medium comprising:

Images