Surgical hand disinfection intelligent monitoring and compliance guarantee system based on behavior recognition

Abstract

The invention discloses a surgical hand disinfection intelligent monitoring and compliance guarantee system based on behavior recognition. The system comprises a multi-mode perception acquisition module, an edge calculation and behavior intelligent recognition module, a real-time enhancement interaction and feedback module and a data integration and cloud intelligent management module. According to the system, data of the hand washing process are synchronously collected through double-path depth vision, infrared thermal imaging and a quantitative disinfectant metering unit, step recognition, quality evaluation and coverage analysis of hand actions are achieved through an edge side AI model, and real-time visual guidance and voice correction are provided through augmented reality projection and directional audio. And the cloud platform establishes a digital file based on the user behavior data, and realizes precise intervention and management through a dynamic risk assessment model. According to the invention, the transformation from passive monitoring to active guidance and from post-event sampling inspection to whole-process real-time quality control is realized, the compliance and operation quality of surgical hand disinfection are effectively improved, and the hospital infection risk is reduced.

Description

Technical Field

[0001] This invention relates to the field of medical information technology and infection control technology, and in particular to an intelligent monitoring and compliance assurance system for surgical hand disinfection based on behavior recognition. Background Technology

[0002] Surgical hand disinfection is one of the most fundamental and crucial steps in preventing surgical site infections, and its effectiveness directly affects patient safety and the quality of medical care. The World Health Organization and medical guidelines in various countries have clearly stipulated a standardized surgical hand disinfection procedure centered on the "seven-step handwashing technique." This procedure has strict and detailed requirements regarding the order of steps, the duration of each step, the intensity of rubbing, the coverage area, and the sufficient amount of disinfectant used.

[0003] However, the traditional management model, which relies on the self-discipline of medical staff and subsequent spot checks, has a series of fundamental flaws that are difficult to overcome in practice, making hand disinfection compliance a consistently weak link in hospital infection control: 1. Currently, the main method relies on irregular manual inspections by infection control personnel. This approach not only consumes a large amount of professional manpower but also has limited coverage and random timing, making it impossible to achieve 24 / 7, full-process supervision. Furthermore, the supervision results are highly susceptible to the subjective judgment, attention span, and environmental factors of the inspectors, lacking objective and consistent evaluation standards.

[0004] 2. Current advanced automated monitoring technologies, such as the scheme disclosed in Chinese patent CN112494161B, mainly verify compliance through timers and periodic microbial cultures. Although they achieve partial automation of recording, they are essentially still a "end-point verification" model. Microbial culture results typically take 24 to 48 hours to obtain, by which time the surgery has long been completed. These results can only provide a retrospective evaluation of hand hygiene at a "past moment," and cannot provide any real-time intervention for the "ongoing" disinfection process, thus losing their preventative value.

[0005] 3. Most automated handwashing devices on the market can only monitor the total handwashing time (e.g., whether it reaches 3 minutes) and equate this with compliance. It's common for handwashing time to meet the standard but with completely incorrect steps, distorted movements, or omissions of crucial areas. For example, some devices may only perform a simple palm-to-palm rubbing while skipping more detailed steps like "clamping," "kneading," and "standing," or fail to clean critical areas such as fingertips, between fingers, and wrists. Current technology lacks the ability to deeply perceive and analyze the "intrinsic quality" of the handwashing process itself, leaving the most important process quality completely out of control.

[0006] 4. When medical staff operate independently, it's like practicing complex movements without a coach; they can only rely on memory and intuition. Once incorrect muscle memory and operating habits are formed, they are extremely difficult to correct later, especially for newly hired or under-trained personnel. Existing passive, punitive supervision models easily trigger resistance and fail to improve compliance and operational skills at their root.

[0007] In summary, this invention aims to fundamentally solve the aforementioned pain points and achieve a significant improvement in the quality of surgical hand disinfection by constructing a closed-loop management system that deeply integrates multimodal perception, edge intelligence, augmented reality, and data-driven approaches. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to overcome the limitations of the existing technology, which can only perform superficial duration recording and delayed microbial verification, and to provide a comprehensive monitoring and protection system and method that can perform multi-dimensional, fine-grained, intelligent perception, identification, guidance, evaluation and risk prediction of the surgical hand disinfection process itself.

[0009] To achieve the above objectives, the first aspect of this invention provides a surgical hand disinfection intelligent monitoring and compliance assurance system based on behavior recognition. This system is constructed within a highly integrated "intelligent sensing-interaction space" in the surgical handwashing area. Through the synchronous collection and fusion analysis of multi-source heterogeneous data, the invisible operational process is transformed into a quantifiable, assessable, and guideable digital twin model, thereby driving real-time interaction and precise management.

[0010] The system specifically includes the following collaborative modules: 1. Multimodal sensing and acquisition module This module is responsible for capturing dynamic information about the handwashing process from all aspects of the physical world. A multi-layered, complementary sensing network is constructed, including: Dual-channel high frame rate depth vision unit: Employs at least two high-precision depth cameras with global shutters (e.g., based on structured light or time-of-flight principles). The first camera (primary view camera) is vertically mounted above the sink, with its lens pointing vertically downwards, ensuring complete coverage of the entire sink area and the user's upper body and arm range of motion. The second camera (auxiliary view camera) is mounted laterally approximately 0.8-1.2 meters to the side of the sink, at a height of 1.2-1.5 meters, with its optical axis at a 30-60 degree angle to the user's standing direction. The two cameras are connected via a hardware synchronization signal cable to achieve time-stamp synchronization, and the synchronized RGB video stream and depth map stream are transmitted to an edge server via Gigabit Ethernet or USB interface. The core purpose of this unit is to acquire high-precision, real-time 3D point cloud data of the hand and forearm, providing a data foundation for subsequent high-precision motion tracking and 3D reconstruction.

[0011] High-sensitivity infrared thermal imaging unit: Employing an uncooled microthermometer infrared thermal imager, this unit is integrated and installed beside the main camera, with its optical axis also aligned with the core active area of ​​the hand. It not only operates in the visible light band but also continuously captures thermal imaging video sequences in the long-wave infrared band (8-14μm). Because surgical hand disinfectants (mostly alcohol-based or isopropanol-based) absorb heat intensely when evaporating on the skin surface, local skin temperature experiences a characteristic drop within a short period. By analyzing the spatiotemporal distribution patterns of temperature changes in different areas of the hand in the thermal imaging images, we can indirectly, intuitively, and quantitatively reflect the initial uniformity of disinfectant application and whether it continuously and adequately covers all key anatomical areas (such as nail edges, interdigital creases, and thenar eminence) during dynamic rubbing. The system enhances the interpretability of thermal imaging data through an adaptive baseline calibration mechanism: before each handwashing process begins, the system acquires an initial thermal image of the user's hands in a static state as an individualized temperature baseline. During disinfection, the system analyzes not only absolute temperature values ​​but also the relative temperature change rate and spatial temperature gradient distribution. For different types of disinfectants, the system incorporates multiple volatilization-endothermic models, which, through manual configuration, dynamically match the expected temperature decay curve patterns, enhancing the chemical and physical basis of coverage analysis.

[0012] The intelligent quantitative disinfectant delivery and metering unit integrates a high-precision miniature electromagnetic flowmeter or a strain gauge-based weighing sensor. Triggering methods are compatible with contactless infrared sensing, foot switches, or elbow touch. Upon each trigger, the unit accurately measures and records the output disinfectant volume or mass, and reports it in real-time to an edge server via Bluetooth 5.0 low-power or RS-485 industrial bus. This data is used to objectively determine whether the user's single dispensing amount meets the minimum standard required by infection control guidelines (typically 3-5 ml), preventing insufficient dosage from the outset.

[0013] 2. Edge computing and intelligent behavior recognition module This module is deployed on a rugged edge computing server (such as one based on the NVIDIA Jetson AGX Orin platform) near the handwashing area. It receives all high-speed data streams from the sensing module and runs lightweight yet powerful AI models to achieve real-time analysis and understanding with low latency. This includes: The real-time hand 3D digital twin submodule first preprocesses the dual-channel RGB-D data (denoising, alignment, background subtraction), and then uses an efficient stereo matching algorithm (such as SGM) or a deep learning-based multi-view approach. Figure 3 A 3D reconstruction network is used to generate detailed 3D mesh models of the hand and forearm in real time. A 3D pose tracking algorithm based on Kalman filtering or particle filtering is employed to continuously output the coordinates, rotation angles, and motion trajectories of all key points such as finger joints and wrist joints in 3D space.

[0014] Multi-stream spatiotemporal fusion behavior recognition AI engine: A heterogeneous multi-branch deep neural network architecture was designed. Spatial Flow Branch: Using RGB images as input, a lightweight 2D-CNN is used to extract static appearance features from each frame of the image, focusing on capturing spatial information such as hand shape, skin texture, and disinfectant reflection.

[0015] Temporal flow branch: Taking the dense optical flow field of multiple consecutive frames or the joint motion sequence calculated directly from the 3D reconstruction results as input, a 3D-CNN or temporal convolutional network (TCN) or Transformer encoder is used to specifically learn the dynamic temporal patterns of the action, such as the direction, rhythm and periodicity of kneading.

[0016] Multimodal fusion branch: This branch is further divided into two sub-streams: Thermal Imaging Substream: Processes thermal imaging video sequences, using CNN to extract spatial features of temperature distribution and their temporal evolution patterns. Sensor Data Substream: Inputs structured data such as disinfectant dosage scalars and radar-estimated intensity frequencies after encoding.

[0017] Fusion and Decision Layer: In the later stages of the network (feature level or decision level), the high-dimensional features extracted from all the above branches are adaptively weighted and fused. The fusion strategy can employ an attention mechanism, allowing the network to dynamically learn the importance of each modality in different scenarios. Ultimately, the network outputs multi-level, fine-grained recognition results. Step segmentation and sequence compliance judgment: The continuous video stream is precisely segmented into seven steps ("inner, outer, clamp, bow, big, stand, wrist") (or steps defined by local specifications), and it is judged whether the order of their occurrence is correct, whether there are omissions or skips.

[0018] Step execution quality scoring: For each identified step, it is compared with the standard action model defined by experts in four dimensions: "range of motion", "joint angle", "speed of movement" and "duration", and a quality score of 0-100 is given. For example, the "clamping" step requires the fingers of both hands to be fully interlaced, the palms and backs of the hands to be closely facing each other and have obvious reciprocating movements.

[0019] Disinfectant coverage uniformity analysis: The 3D mesh model of the hand is divided into dozens of key anatomical regions (zones). By fusing the spatiotemporal curve of thermal imaging temperature decrease with the corresponding 3D motion trajectory data of the region (such as the motion velocity of the point cloud in the region and the contact frequency with the contralateral skin), a "coverage confidence" model is constructed. The system can intelligently determine which areas have "poor coverage" due to insufficient movement or uneven application, and quantify its severity.

[0020] Overall compliance assessment: Taking into account the sequence of steps, quality scores of each step, coverage analysis results, and disinfectant usage data, an interpretable rule engine or lightweight classifier is used to give a final "compliant", "basically compliant (with flaws)", or "unqualified" rating for this handwashing operation, and generate a structured problem description.

[0021] 3. Real-time enhanced interaction and feedback module This module is responsible for transforming analytical data into intuitive, user-friendly, and efficient guidance, including: Augmented Reality (AR) Transparent Projection Unit: This unit uses a high-resolution, high-brightness ultra-short-throw laser projector, which is discreetly installed above the sink to project images precisely onto the mirror or specially made anti-fog transparent glass in front of the user, forming a virtual interactive screen that floats above the real scene.

[0022] Dynamic guidance interface: After the user starts washing their hands, the core area of ​​the interface renders a pair of semi-transparent, high-fidelity 3D animations of "virtual guiding hands" in real time. The animation strictly follows the standard process and is synchronized with the current step recognized by AI, demonstrating precise action details.

[0023] Real-time self-mirroring and visual feedback: Next to the "virtual hand," a color outline or semi-transparent image of the user's real hands, generated and keyed in real time by a depth camera, is simultaneously displayed. The system visualizes the analysis results using color coding: Green: The current step was executed well, and the corresponding area is adequately covered. Yellow: The action is basically correct but has minor flaws (such as being slightly too fast), or the area coverage is at a critical point. Flashing red: An error was detected, time was severely insufficient, or there is a clear coverage blind spot in a specific area. The red area will be overlaid on the corresponding position of the user's real hand image as a highlighted outline or heat map, accurately pointing out the problem.

[0024] Progress and performance display: The top of the interface displays the current step number / name, the remaining time for that step, and the "green, yellow, and red" status lights for each completed step. A brief evaluation report for this operation pops up immediately after handwashing is completed.

[0025] Spatial audio cues and voice guidance unit: A directional speaker array using beamforming technology focuses sound energy on the user's head area, minimizing noise interference with the surrounding environment.

[0026] Pre-instruction voice: Before each step begins, a concise step name and key points are played (e.g., "Step 2, rub palms and backs of hands together along the gaps between fingers").

[0027] Real-time voice correction: When AI detects a problem, it triggers targeted voice prompts. The prompts are precise and constructive, such as, "Please note that the temperature of your left fingertips is rising more slowly. Please increase the rotation and rubbing of your fingertips in your palm," rather than simply saying "Error."

[0028] Positive reinforcement voice prompts: When users correct mistakes in a timely manner or complete multiple steps well in a row, they are given short encouragement such as "Very good, please keep it up".

[0029] 4. Data Integration and Cloud-based Intelligent Management Module This module implements a closed loop of data aggregation, in-depth analysis, and management. It includes: Local data integration terminal: As a software component of the edge server, it is responsible for: Identity Binding: By connecting to facial recognition terminals, RFID readers, or interfacing with the hospital's HIS system, each handwashing operation is uniquely associated with the medical staff's identity at the start of the process. Structured Report Generation: After each handwashing session, a detailed electronic report is automatically generated, including: user ID, timestamp, total duration, time and quality score for each step, disinfectant usage, list of poorly covered areas, error types identified by AI, comprehensive score, and judgment result. The report uses structured formats such as JSON for easy subsequent processing. Local Encrypted Caching and Asynchronous Upload: Reports are encrypted and stored locally and asynchronously uploaded to the cloud when the network is stable, ensuring no data loss and reducing instantaneous network pressure.

[0030] Cloud-based intelligent management platform: Deployed in the hospital's data center or private cloud, providing a web management interface with a B / S architecture.

[0031] The full-domain data dashboard provides multi-dimensional, drill-down real-time data dashboards, displaying real-time compliance rates, average scores, distribution of high-frequency error types, disinfectant consumption trends, etc., for the entire hospital, each department, each operating floor, and even individuals.

[0032] Individual and collective digital profiles: Establish a continuously updated "hand hygiene behavior digital profile" for each medical staff member, fully record their operation data, and form an individual performance trend curve.

[0033] Dynamic risk assessment and precise intervention model: This is the core intelligence of the cloud platform. The model continuously analyzes individual profiles, with input variables including but not limited to: Recent compliance rate trends (highest weighted). Characteristics and stability of error patterns (e.g., persistently ignoring the "wrist"). Job-related risk level (e.g., cardiac surgeons face higher risks than outpatient physicians). Recent training and assessments received.

[0034] The model dynamically calculates each healthcare worker's current "hand hygiene risk coefficient" using machine learning algorithms (such as gradient boosting decision trees). Based on this coefficient, the platform ranks healthcare workers and automatically develops differentiated follow-up intervention strategies. Microbial sampling and testing plan: For individuals with high risk, the system automatically increases the frequency of microbial sampling (e.g., once every 5 operations); for individuals with low and stable risk, the frequency is reduced (e.g., once every 50 operations). This ensures that limited infection control and testing resources are precisely allocated to high-risk groups.

[0035] Intelligent early warning and task distribution: When the system detects a serious violation in a single operation, or a sharp increase in an individual's risk factor in a short period of time, it automatically generates an early warning work order and pushes it to the infection control specialist, department head, and the individual. It can also automatically link and push personalized retraining video courses or online assessment tasks.

[0036] Evidence-based management decision support: Long-term accumulated data can be used to analyze the effectiveness of different training methods, the actual coverage performance of different disinfectant brands, and even the potential correlation with surgical site infection rates, providing data-driven decision support for optimizing hospital infection control strategies.

[0037] Secondly, this invention provides a behavior recognition-based intelligent monitoring and compliance assurance method for surgical hand disinfection applied to the above-mentioned system. This method realizes a complete closed loop from perception to decision-making, including the following steps: S1: Intelligent Scene Activation and Identity Binding. When a user enters the handwashing area, a passive infrared sensor or depth camera detects the approaching person, waking the system from a low-power state. The user completes identity authentication via facial recognition or employee card recognition, and the system loads the user's personalized settings (such as voice preferences) and historical performance baseline.

[0038] S2: Synchronous Acquisition and Streaming of Multi-Source Heterogeneous Data. When the user starts an operation (such as pressing the liquid dispensing switch), the system's main control program sends a synchronization signal, and the depth vision unit, thermal imaging unit, metering unit, auxiliary sensors, etc., start simultaneously. Using a unified network time protocol clock as a reference, high-speed, continuous data acquisition begins, and the data is streamed to the edge computing server in real time via a high-speed bus.

[0039] S3: Real-time multi-dimensional behavior perception and understanding at the edge. Multiple processing threads run in parallel on the edge server: a) Perform real-time 3D reconstruction and joint tracking on RGB-D streams to construct a digital twin of the hand.

[0040] b) Input multimodal data streams (RGB frames, optical flow / motion sequences, thermal imaging frames, sensor data) into the deployed multi-stream fusion AI model for forward inference.

[0041] c) Real-time model output: current step classification, step quality score, hand area coverage confidence map, and global compliance status.

[0042] S4: Augmented Reality Guidance and Real-Time Process Intervention. While S3 is running, the edge server's graphics rendering engine, based on the AI ​​output: a) Drive the AR projection unit to update the "virtual guide" animation to the current standard procedure.

[0043] b) Compare the user's real hand image with the standard model, generate color-coded visual feedback, and overlay it.

[0044] c) Trigger corresponding spatial audio prompts according to preset rules (such as detecting a red alert).

[0045] This step achieves a rapid closed loop of "perception-analysis-feedback," enabling users to obtain the opportunity to correct errors immediately upon their occurrence.

[0046] S5: Generate electronic records and archive locally. Once the handwashing process is complete (either determined by AI or by the user), the system integrates all intermediate results and timestamps generated in stage S3, automatically generating a structured compliance assessment report, which is then linked to the user's identity information and stored locally with encryption.

[0047] S6: Data Cloud Aggregation and Intelligent Decision-Making Iteration. The local terminal uploads encrypted report files to the cloud management platform. After receiving the new data, the platform: a) Update individual and group profiles in the central database.

[0048] b) Trigger the dynamic risk assessment model to recalculate the risk coefficients of relevant personnel based on the latest data.

[0049] c) Based on the updated risk view, automatically optimize and output the precise intervention plan for the next cycle (such as adjusting the sampling list and generating training tasks).

[0050] d) Provide infection control management personnel with the latest statistical analysis and early warning information.

[0051] This completes the closed loop from monitoring a single operation to managing group risks.

[0052] Compared with the prior art, the beneficial effects of the present invention are: 1. It has shifted from the traditional "time-based + post-event training" model to a deep process quality control of the operational behavior itself in "all steps, all dimensions, and all real-time", solving key problems in infection control.

[0053] 2. By integrating three-dimensional vision, infrared thermal imaging, quantitative measurement, and auxiliary sensing, the system evaluation no longer relies on a single signal, has strong anti-interference capabilities (such as changes in light or water mist obstruction), and greatly improves the scientific validity and credibility of the evaluation conclusions.

[0054] 3. By seamlessly integrating standard specifications with users' self-images through AR technology, an unprecedented immersive training experience is provided. Real-time, precise visual and auditory feedback transforms passive supervision into proactive guidance, significantly improving medical staff's learning interest, operational compliance, and skill mastery.

[0055] 4. By establishing digital profiles of individual behaviors and dynamic risk prediction models, the precise allocation of infection control resources (such as manpower and testing costs) has been achieved. The management strategy has been upgraded from a "one-size-fits-all" periodic random inspection to precise intervention that is "tailored to the individual and the risk," greatly improving management efficiency and prevention and control effectiveness.

[0056] 5. Employing an edge computing architecture, core analysis is completed locally, ensuring real-time performance, reducing network dependence, and enhancing data privacy. The modular design allows the system to flexibly adapt to various newly built or renovated handwashing station environments. The cloud platform facilitates centralized, intelligent, and visualized management for hospitals. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the overall hardware layout and connection architecture of the system of the present invention.

[0058] Figure 2 A schematic diagram illustrating the principle of hand 3D reconstruction and tracking using a dual-path depth vision unit.

[0059] Figure 3 This is a block diagram of the algorithm network structure of a multi-stream spatiotemporal fusion behavior recognition AI engine.

[0060] Figure 4 Architecture diagram for augmented reality (AR) real-time interactive interface.

[0061] Figure 5 This is a flowchart of the dynamic risk assessment and precise intervention decision-making process for a cloud-based intelligent management platform. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] To make the objectives, technical solutions, and advantages of this invention clearer, the following will use the surgical handwashing area of ​​a newly built clean operating room in a large tertiary hospital as a typical application scenario to describe in detail the specific implementation of the system of this invention. Those skilled in the art will understand that, without departing from the core principles of this invention, adaptations can be made according to specific site conditions.

[0064] Please see the appendix Figures 1-5 Example 1: System Hardware Deployment and Integration 1.1 Site Survey and Planning First, a detailed survey of the target handwashing area is required. Record the dimensions of the handwashing basin, the material of the countertop, the size and location of the mirror or glass above it, the ceiling height, lighting conditions, and the locations of power and network interfaces. Based on the site conditions, if necessary, use 3D design software to plan the installation locations and wiring paths for all equipment, ensuring no visual obstruction, ergonomic design, and a neat and aesthetically pleasing appearance. Key planning points include: the fields of view of the two depth cameras must completely cover the area from the fingertips to 10cm above the elbow, with sufficient overlap between their fields of view; the thermal imager must be protected from direct exposure to strong ambient heat sources (such as heater vents); and the AR projection image must be projected completely and without distortion onto the visible area directly in front of the user.

[0065] 1.2 Hardware Equipment Selection and Installation Depth vision unit: For example, two Intel RealSense D455 depth cameras are selected. They feature a global shutter, stable performance in complex lighting conditions, and support hardware synchronization.

[0066] Install the main camera (A): Make an opening above the sink ceiling and install an adjustable universal bracket. Fix camera A to the bracket with the lens pointing vertically downwards. Adjust the height so that it is about 1.8 meters away from the sink countertop, ensuring that its field of view covers a square area with sides of about 1 meter centered on the sink.

[0067] Install auxiliary camera (B): Mount another bracket approximately 1.3 meters high on the wall in front of the sink (usually to the user's right). Adjust the angle of camera B so that its optical axis points to the center of the sink, forming an angle of approximately 45 degrees with the user's face. Connect the synchronization interfaces of cameras A and B using the provided synchronization cable.

[0068] Two active extension cables are led out from cameras A and B respectively, laid in conduits, and finally connected to the edge server rack.

[0069] Infrared thermal imaging unit: A FLIR Lepton 3.5 miniature thermal imager module is selected and integrated into a custom-designed small protective housing. It is mounted side-by-side with camera A on the same ceiling bracket, approximately 10cm apart, with the optical axis roughly parallel and downward. It connects to the edge server via a USB cable through its onboard SPI-to-USB interface.

[0070] Intelligent disinfectant delivery and metering unit: Select an intelligent dispensing device compatible with the hospital's existing disinfectants. Install it in the original dispensing device location. Its internal flow meter signal is connected to an RS-485 to USB converter inside the cabinet via an RS-485 communication cable. Install the foot switch or infrared sensor according to the instruction manual.

[0071] Edge computing server: The NVIDIA Jetson AGX Orin 64GB developer kit was selected, which has powerful AI computing capabilities and rich I / O interfaces. It was installed in an industrial-grade wall-mounted chassis with a cooling fan, which was fixed to the wall of the equipment room near the sink, no more than 10 meters away from the sink, to shorten the cable length.

[0072] AR projection unit: Use a laser projector (ultra-short throw version) or similar product. Create a concealed mounting slot for the projector in the ceiling above the sink, exposing only the lens. Precisely adjust the projector's position, tilt, and focus to project the image onto the anti-fog high-definition glass in front of the sink, ensuring a clear and square image.

[0073] Audio Unit: Two Bluetooth ceiling speakers with beamforming capabilities are selected and installed above the sink on the left and right sides, respectively, facing the user's head area. The speakers are connected to the audio output port of the edge server via Bluetooth receivers.

[0074] Identity recognition unit: A face recognition terminal (such as Hikvision DS-K1T671) is installed on the side wall of the sink. It is connected to the hospital's intranet switch via a network cable and communicates with the edge server via a serial port or network API.

[0075] Structured cabling: All power and data cables (USB, HDMI, audio, network cables) are flame-retardant, low-smoke halogen-free cables, run through flexible metal conduits or PVC cable trays, and are neatly laid along ceilings and walls, ultimately converging at the edge server chassis. High-voltage and low-voltage cables are laid separately to avoid interference. All equipment is equipped with suitable UPS (Uninterruptible Power Supply) to handle brief power outages.

[0076] 1.3 System Calibration and Initialization After the hardware is installed, precise system-level calibration must be performed.

[0077] Depth camera calibration: Internal parameter calibration: Using a checkerboard calibration board, monocular camera calibration was performed on cameras A and B respectively to obtain their respective internal parameters such as focal length, principal point, and distortion coefficient.

[0078] Extrinsic parameter calibration (stereo calibration): The calibration board is placed at different positions and angles in the sink area, and multiple sets of images are simultaneously captured by cameras A and B. Using the stereo calibration toolkit in OpenCV or MATLAB, the rotation matrix and translation vector of camera B relative to camera A are calculated, i.e., the spatial positional relationship between the two. This step is crucial and forms the basis for subsequent 3D reconstruction.

[0079] Spatial Coordinate System 1: Define the center of the washbasin countertop as the origin of the world coordinate system. Through measurement, transform the extrinsic parameter matrices of cameras A and B to this world coordinate system. Simultaneously, through measurement, determine the three-dimensional coordinates of the four corner points of the AR projector's image in the world coordinate system, establishing the projector's "projection matrix."

[0080] Thermal imager spatial registration: Since the thermal imager and camera A are installed in close but not identical positions, coarse registration is required. An object with temperature variations (such as a cup of warm water) that can be seen by both the RGB camera and the thermal imager is placed in the sink area. Through feature point matching, the homography transformation matrix between the thermal imager image and the camera A image is estimated so that the thermal imaging data can be mapped onto the 3D hand model.

[0081] Background model learning: In an unmanned state, the system collects and stores depth background images, RGB background images and thermal imaging background images for a period of time, which are used for subsequent foreground segmentation to reduce static environmental interference.

[0082] Example 2: Construction, Training, and Deployment of Software Algorithm Models 2.1 Data Collection and Labeling Build a prototype system similar to the field in a laboratory or pilot operating room. Recruit dozens of volunteers (including infection control experts, surgeons, nurses, and students) to collect a large amount of multimodal data on surgical hand disinfection procedures with informed consent.

[0083] Data collection scenarios: covering both standard operations and intentionally designed non-standard operations (such as skipping steps, distorted movements, insufficient time, uneven simulation coverage, etc.).

[0084] Data synchronization: Develop data acquisition software to ensure that the timestamps of RGB-D video stream, thermal imaging video stream, and sensor data stream are strictly synchronized and saved as a sequence file.

[0085] Professional annotation: Use professional video annotation tools (such as CVAT or custom tools) to perform multi-level annotation on the collected data: Step-level annotation: Annotate the handwashing step to which each frame of the image belongs ("inside", "outside"... "wrist" or "background").

[0086] Action quality rating: For each step segment, a rating of 0-100 is given from four dimensions: "amplitude", "angle", "speed" and "duration".

[0087] Region Coverage Labels: On the thermal image or RGB image of the keyframe, anatomically divide the hand into regions (e.g., "left thumb pad", "right finger crease 1-2", etc.) and label whether the region is "fully covered", "partially covered" or "not covered" in this step.

[0088] Global compliance labeling: Label each complete handwashing procedure as "compliant", "basically compliant", or "unqualified".

[0089] 2.2 Design and Training of Multi-Stream Fusion AI Model Design an end-to-end neural network called "HandWashNet".

[0090] Input layer: Spatial Stream: Input 3 consecutive RGB images (224x224). Temporal Stream: Input the dense optical flow field (224x224, 2 channels) corresponding to the RGB frames. Thermal Imaging Stream: Input 5 consecutive thermal images (scaled to 112x112). Sensor Stream: Input a 4D vector: [Disinfectant dosage (ml), current step duration (s), radar strength index, radar frequency index].

[0091] Feature extraction backbone network: Spatial Flow: The first few layers of EfficientNet-B0 pre-trained on ImageNet are used to extract spatial feature maps Fs. Temporal Flow: A lightweight 3D CNN (such as R(2+1)D-18) is used to extract spatiotemporal features Ft. Thermal Imaging Flow: A simple 4-layer 2D CNN is used to extract thermal features Fth.

[0092] Multimodal fusion: Fs and Ft are concatenated along the channel dimension, and then passed through a convolutional layer and a spatial attention module to obtain the visual fusion feature F_vis. The sensor vector is expanded into a feature map through a fully connected layer, concatenated with Fth along the channel dimension, and then passed through a convolutional layer to obtain the sensor fusion feature F_sens. Before the decision head of the network, F_vis and F_sens are concatenated again, passed through a fully connected layer and a Dropout layer, and the final output is: Step classification probability (8 classes: 7 steps + background).

[0093] Step quality regression value (1 scalar).

[0094] Heatmap of poorly covered areas (a low-resolution 2D map that can be upsampled and mapped back to the hand).

[0095] Model training: Use the PyTorch framework.

[0096] The loss function is a multi-task weighted sum: cross-entropy loss for step classification + smoothing L1 loss for quality regression + Dice loss for region heatmap prediction.

[0097] The collected dataset is divided into training, validation, and test sets. The AdamW optimizer is used for thorough training until the performance on the validation set converges.

[0098] Data augmentation techniques (such as random cropping, color dithering, and simulated water mist) are used to improve the robustness of the model.

[0099] 2.3 Model Optimization and Edge Deployment Model optimization: NVIDIA's TensorRT tool was used to optimize the trained PyTorch model for inference. This included FP16 quantization, layer fusion, and automatic kernel tuning. This significantly improved inference speed and reduced latency.

[0100] Package the optimized TensorRT engine files, model configuration files, and related preprocessing and post-processing code into a Docker image or deploy it directly to the Jetson AGX Orin edge server. Develop a persistent inference service process that receives real-time streams from the data acquisition module via gRPC or ZeroMQ and returns the recognition results.

[0101] Example 3: System Software Integration and Interaction Logic Implementation 3.1 Edge-side main control software development The master control software, developed using C++ and Python, runs on the edge server and is responsible for the process scheduling and data flow management of the entire system.

[0102] Multi-threaded data acquisition module: Creates independent threads to read data from the depth camera, thermal imager, and serial port (liquid dispenser), adds a unified timestamp, and puts it into a shared memory circular buffer.

[0103] Data synchronization and preprocessing module: A high-priority thread retrieves data from each buffer and performs alignment and synchronization based on timestamps. It performs distortion correction and alignment on RGB and depth images (ensuring a one-to-one correspondence between RGB pixels and depth pixels). A registration matrix is ​​applied to the thermal image to coarsely align it to the RGB image space.

[0104] AI Inference Client Module: Packages the synchronized multimodal data, calls the deployed "HandWashNet" inference service, and obtains real-time recognition results.

[0105] State Machine and Business Logic Module: Implement a finite state machine to manage the entire handwashing process: Waiting -> Identity Recognition -> Preparation -> Step 1 -> ... -> Step 7 -> Complete. State transitions are driven by AI inference results, and corrective feedback is triggered as needed.

[0106] AR rendering and feedback control module: Develop a lightweight rendering engine using OpenGL or Vulkan, or integrate the Unity engine runtime.

[0107] Based on the current status and AI results, dynamically render the 3D animation of the "virtual guide" (animation resources are pre-made).

[0108] It receives hand contour or 3D mesh data from AI inference and renders an image of the user's hand.

[0109] Based on the coverage analysis results and step quality, a color map (green / yellow / red) is calculated and applied to the corresponding area of ​​the hand image.

[0110] The rendered final image is then output to the projector via HDMI.

[0111] Voice dispatch module: Maintains a voice segment library. Based on instructions from the business logic module, it calls the audio API to play the corresponding pre-recorded or TTS-generated voice prompts.

[0112] Local report generation and upload module: At the end of the process, it collects all timestamp data, AI intermediate results, and final judgment, generates a JSON format report, encrypts it using AES, and saves it to a local SQLite database. Another background thread periodically checks the network and securely uploads any unuploaded reports to the cloud platform's designated API interface via HTTPS.

[0113] 3.2 Cloud Management Platform Development Develop cloud backends using Java Spring Boot or Python Django frameworks, and develop frontends using Vue.js or React.

[0114] Backend services: Data Receiving API: Provides a secure RESTful API to receive encrypted reports uploaded from various edge terminals, decrypts them, verifies data integrity, and stores them in a PostgreSQL or TimescaleDB (for time-series data) database.

[0115] Data analytics engine: Implements dynamic risk assessment models. For example, using Python's scikit-learn library, features (such as compliance rate in the past 7 days, main error types, and risk position weights) are extracted from all recorded personnel every hour over the past 30 days. An XGBoost classification model is then trained or updated to predict the probability of being "high-risk" as a risk coefficient. The results are stored in a database.

[0116] Business logic service: Automatically generates future microbial sampling schedules based on risk coefficients and preset rules. Manages early warning rules and task distribution.

[0117] Front-end management interface: Data Dashboard: Utilizes chart libraries such as ECharts to display real-time monitoring dashboards, historical trend analysis, department rankings, and individual details.

[0118] Risk management interface: Displays a list of high-risk personnel, risk change trends, and a sampling plan automatically generated by the system for administrator review and confirmation.

[0119] Archives and Reports Inquiry: Supports searching historical handwashing records and detailed reports by personnel, time, department, and other criteria.

[0120] System configuration interface: Administrators can configure AI model confidence thresholds, feedback trigger rules, risk model parameters, and alert recipients, etc.

[0121] Example 4: Complete Workflow Example Using a preoperative handwashing procedure performed by Chief Physician Wang of the Cardiac Surgery Department as an example, the complete operation process of the system is illustrated: Activation and Login: At 2 PM, Director Wang walked towards the smart handwashing station in Operating Room 3. When he was about 1 meter away from the sink, the depth camera detected movement, and the system woke up from standby. A gentle prompt, "Please verify your identity," appeared on the mirror / glass. Director Wang aligned his face with the facial recognition terminal to the side, and it was instantly recognized. The mirror displayed: "Hello, Director Wang! Please begin your surgical hand disinfection." The system simultaneously retrieved Director Wang's recent average score (92 points) from the cloud as a reference baseline for this assessment.

[0122] Liquid dispensing and data acquisition initiation: Director Wang pressed down the lever of the intelligent dispensing device with his elbow. The internal micro-pump of the dispensing device started working, and the high-precision flow meter measured in real time, automatically stopping after dispensing 4.2ml of disinfectant. The measurement data "4.2ml, sufficient" was sent to the edge server via Bluetooth. The server recorded this event as a process start signal and immediately sent a synchronization start command to all sensors.

[0123] Process execution and real-time guidance: Director Wang applied disinfectant to his hands and began to rub them together.

[0124] Step 1 (Internal): He rubs his palms together. The dual cameras above capture his hand movements, and the AI ​​model on the edge server quickly analyzes them: the step classification is correct, the joint movement amplitude meets the standard, and the duration has reached 15 seconds (exceeding the minimum requirement of 10 seconds). Simultaneously, the thermal image shows the palm area temperature decreasing uniformly. AR Interface: The main area's "virtual hand" is demonstrating the palm-rubbing animation; the actual outline of Deputy Director Wang's hand is displayed in green. The system plays a voice message: "Step one, palms together, completed well." Step 3 (Clamping): As the fingers interlock and the palm rubs the back of the hand, the AI's temporal flow branch analysis of the movement trajectory reveals that the left hand's interlocking angle is slightly smaller than the right hand's, and the rubbing frequency is slightly faster. Simultaneously, thermal imaging fusion analysis shows that the temperature drop curve in the area near the little finger on the back of the right hand (a common blind spot) is gradual, indicating low coverage confidence. Real-time feedback trigger: In the AR interface, the outline of Director Wang's left hand turns yellow, and the area on the little finger side of the back of his right hand becomes a flashing red highlight. At the same time, a directional speaker plays: "Please pay attention to the interlocking angle of your left hand and intensify the rubbing of the outer side of the back of your right hand." Director Wang notices the red prompt, immediately adjusts his movements, increasing the angle and focusing on rubbing the red area. Two seconds later, the red alert disappears, and the outline returns to green.

[0125] Subsequent steps: The system continues to perform this type of synchronous monitoring and guidance. In the "Stand" (fingertips rotating in the palm) step, the flexible pressure-sensitive film (if already installed) detects the fingertip pressure distribution, providing additional strong evidence for the AI's judgment.

[0126] End and Report Generation: After completing the wrist rubbing, Director Wang held his hands above the sink. The AI ​​model detected two consecutive seconds without any specific rubbing motion, and all steps were recognized, thus determining the process to be complete. Total time: 4 minutes and 50 seconds. The handwashing report popped up on the mirror: "Total time: 4 minutes and 50 seconds; Step completeness: 100%; Average score for action standardization: 94; Area coverage: Excellent (one area corrected in real time); Disinfectant usage: Sufficient. Overall score: 96 points. Data recorded. Wishing you a successful surgery." The report then faded out from the screen.

[0127] Data Upload and Cloud Processing: The edge server uploads this encrypted report to the hospital's infection control data center via the operating room's dedicated network. Upon receiving the report, the cloud platform then... Update Director Wang's personal file and add a new record with a score of 96. His recent (7-day) average score has increased from 92 to 92.5.

[0128] The dynamic risk assessment model was recalculated based on its latest data, and its risk coefficient slightly decreased from 0.15 (low risk) to 0.14.

[0129] According to the latest risk coefficient ranking of all staff in the hospital, Director Wang is ranked low in the list of microbial sampling for the next week (100 people are planned to be sampled), and the system has not included him in the high-frequency sampling list.

[0130] Zhang, a specialist in the Infection Control Department, logged into the management platform on his office computer and checked the "Today's Warnings" panel. There were no new warnings. He switched to the "Risk Dashboard" and saw that the overall compliance rate of the cardiac surgery department, where Director Wang worked, was among the best and was on an upward trend, which pleased him.

[0131] Example 5: System Maintenance, Calibration, and Expansion Routine Maintenance: The system is designed for low maintenance. Daily power-on self-test checks the connection status of all sensors. Weekly cleaning of the camera lens and projector lens with alcohol wipes. Monthly backup of the local database.

[0132] Periodic calibration: Simplified external parameter calibration is required every quarter or when the camera position may have been disturbed. This can be quickly done using a simple calibration board. A comprehensive system calibration should be performed annually.

[0133] Model Iteration and Updates: The cloud platform continuously collects anonymized operational data (with authorization). When the data accumulates to a certain scale and new error patterns are discovered, the "HandWashNet" model can be incrementally trained or retrained on the cloud using the new data. The optimized new model is then pushed to various edge terminals in stages through a secure channel for updates, making the system "smarter" with use.

[0134] System Expansion: This system can be easily expanded to all surgical handwashing stations, ICU handwashing stations, and even endoscopy disinfection rooms within a hospital. The cloud platform can centrally manage hundreds or thousands of nodes. Furthermore, the system interface can be opened to hospital HIS and surgical anesthesia systems, enabling broader integration. For example, hand disinfection compliance data can be automatically archived as part of the "preoperative preparation" section in the patient's electronic medical record; or it can be integrated with the surgical scheduling system to automatically check and prompt the doctor about their most recent hand disinfection record before surgery.

[0135] As can be seen from the detailed embodiments described above, this invention is not merely a collection of technical devices, but rather the construction of a complete intelligent ecosystem encompassing "perception-cognition-interaction-decision." It deeply integrates cutting-edge sensor technology, artificial intelligence algorithms, human-computer interaction design, and professional infection control standards, creatively solving the modern management challenges of surgical hand disinfection—an age-old and crucial medical procedure—and providing a solid technical guarantee for improving medical safety and reducing the risk of hospital-acquired infections.

[0136] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A surgical hand disinfection intelligent monitoring and compliance assurance system based on behavior recognition, characterized in that, It includes a multimodal sensing and acquisition module deployed in the handwashing area, an edge computing and behavior intelligence recognition module, a real-time enhanced interaction and feedback module, and a data integration and cloud-based intelligent management module; The multimodal perception and acquisition module is used to collect multi-source synchronous data in real time during the user's hand disinfection process, and transmit the data to the edge computing and behavior intelligence recognition module; The edge computing and behavior intelligence recognition module is used to process and fuse the multi-source data in real time, identify the compliance of the disinfection steps, the execution quality and the disinfectant coverage status, and output the recognition results. The real-time augmented interaction and feedback module receives the recognition results and provides real-time visual and auditory guidance to the user through augmented reality projection and spatial audio; The data integration and cloud-based intelligent management module is used to receive and store disinfection behavior data from the edge, establish personal digital profiles based on user historical data, and perform dynamic risk assessments and precise intervention decisions.

2. The system according to claim 1, characterized in that, The multimodal sensing and acquisition module includes: The dual-channel high frame rate depth vision unit includes at least two hardware-synchronized depth cameras, which are respectively vertically installed at a preset angle above and to the side of the sink. They are used to simultaneously acquire RGB images and depth images of the hand and forearm, and transmit them to the edge computing and behavior intelligence recognition module via gigabit Ethernet or USB interface. A high-sensitivity infrared thermal imaging unit is installed next to the vertical camera to continuously acquire thermal imaging video sequences of the hand in the long-wave infrared band. The intelligent quantitative disinfectant delivery and metering unit integrates a high-precision flow meter or weighing sensor to measure and output the volume or mass in real time when the user triggers the use of disinfectant, and reports it to the edge server.

3. The system according to claim 2, characterized in that, In the dual-channel high frame rate depth vision unit, the two depth cameras are connected through a hardware synchronization signal line to achieve timestamp synchronization.

4. The system according to claim 1, characterized in that, The edge computing and behavior intelligence recognition module is deployed on the edge computing server and includes: The real-time hand 3D digital twin submodule is used to perform stereo matching or deep learning reconstruction on dual-channel RGB-D data to generate a 3D mesh model of the hand, and continuously output the 3D motion trajectory of the joints based on the filtering algorithm. A multi-stream spatiotemporal fusion behavior recognition AI engine, consisting of a multi-branch deep neural network, including: Spatial flow branch, processing RGB images to extract hand shape and texture features; Temporal flow branch: Process optical flow fields or joint motion sequences to extract motion dynamic features; The multimodal fusion branch integrates thermal imaging features with sensor data; The AI ​​engine outputs include: step classification and sequence judgment, step execution quality score, hand area coverage confidence map, and global compliance judgment result.

5. The system according to claim 4, characterized in that, The quality score for each step is derived by comparing it with a standard motion model based on four dimensions: range of motion, joint angle, movement speed, and duration. The coverage confidence map quantifies the disinfectant coverage status of each anatomical area of ​​the hand by fusing thermal imaging temperature change curves with three-dimensional motion trajectory data.

6. The system according to claim 1, characterized in that, The real-time enhanced interaction and feedback module includes: The augmented reality transparent projection unit uses an ultra-short-throw laser projector installed above the sink to project images onto a mirror or transparent glass. The dynamic guidance interface renders virtual guide hand animations and the user's real hand contours in real time, and uses color coding to provide feedback on the execution status: green indicates compliance, yellow indicates defects, and flashing red indicates errors or blind spots. The spatial audio prompting unit uses a directional speaker array to play pre-guidance voice, real-time correction voice, or positive stimulus voice based on the recognition results.

7. The system according to claim 1, characterized in that, The data integration and cloud-based intelligent management module includes: The local data integration terminal, deployed on the edge server, is used to bind user identity information with handwashing operation data, generate a structured electronic report containing timestamps, step quality scores, disinfectant usage, and coverage analysis results, and perform local encryption and asynchronous uploading. A cloud-based intelligent management platform, deployed in the hospital's data center or private cloud, provides: The data dashboard is used to visualize compliance rates, high-frequency error types, and disinfectant consumption trends across the hospital, departments, and individuals. Personal digital profile, continuously recording user's data from all operations; The dynamic risk assessment model calculates the hand hygiene risk coefficient based on the user's recent compliance rate, error patterns, and job risk level, and automatically generates differentiated intervention strategies.

8. The system according to claim 7, characterized in that, The differentiated intervention strategies include: The microbial sampling frequency is automatically adjusted based on the risk factor. When a serious violation or a sharp increase in risk is detected, an early warning work order is automatically generated and pushed to relevant personnel; Link and push personalized retraining content or assessment tasks.

9. A method for intelligent monitoring and compliance assurance of surgical hand disinfection based on behavior recognition, applied to the system described in any one of claims 1-8, characterized in that, Includes the following steps: S1: Intelligent scene activation and identity binding, which identifies the user's identity through facial recognition or work card and loads their historical performance baseline; S2: Synchronous acquisition of multi-source heterogeneous data, activation of depth vision, thermal imaging, and disinfectant metering units, and streaming transmission to the edge server with a unified time reference; S3: Real-time behavior perception and understanding at the edge, using 3D reconstruction and multi-stream fusion AI models to identify step compliance, execution quality and coverage status in real time; S4: Augmented reality guidance and real-time intervention, updating the AR interface and voice prompts based on recognition results to achieve real-time correction; S5: Generate structured electronic reports and archive them locally with encryption; S6: Upload the report to the cloud, update user profiles, recalculate risk coefficients, and output a precise intervention plan.

10. The method according to claim 9, characterized in that, In step S3, the multi-stream fusion AI model adopts a deep network structure with spatial stream, temporal stream and multimodal fusion branches. It dynamically weights and fuses the features of each modality through an attention mechanism, and outputs step segmentation, quality score and coverage analysis results.

Images