Surgical hand disinfection intelligent monitoring and evaluation system based on multi-modal image fusion

Abstract

The application is a surgical hand disinfection intelligent monitoring and evaluation system based on multi-modal image fusion, which is composed of five modules of multi-modal data acquisition, data synchronization and preprocessing, multi-modal feature fusion analysis, intelligent evaluation and decision, real-time interaction and feedback, synchronously acquires RGB vision, infrared thermal imaging and three-dimensional depth data, realizes three-modal feature fusion and cross-validation through a self-attention mechanism, constructs a ternary collaborative analysis model, conducts non-contact, high-precision and quantitative evaluation on the whole process of surgical hand disinfection, and provides real-time audio-visual feedback and structured report. The application solves the problems of strong subjectivity of traditional manual monitoring, high false alarm rate of single visual system, inability to quantify disinfectant coverage and rubbing effectiveness, etc., realizes monitoring-evaluation-feedback-improvement closed-loop management, significantly improves the qualified rate of surgical hand disinfection, reduces the risk of surgical infection, and can be widely applied to operating room infection control monitoring and medical operation training.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring technology for medical infection control, specifically to an intelligent monitoring and evaluation system for surgical hand disinfection based on multimodal image fusion. Background Technology

[0002] Surgical hand disinfection is a crucial measure for preventing surgical site infections (SSIs), and its quality directly impacts surgical quality and patient safety. However, current surgical hand disinfection quality monitoring systems have significant shortcomings. Traditional monitoring methods primarily rely on manual visual inspection by infection control personnel or colleagues. This method is labor-intensive, subjective, difficult to achieve comprehensive coverage and continuous recording, and prone to causing resistance among medical staff, making it difficult to guarantee the objectivity and consistency of quality evaluation.

[0003] In recent years, with the widespread application of digital technologies in the medical field, a few intelligent systems have emerged that attempt to monitor hand hygiene. However, in the highly demanding field of surgical hand disinfection, the application of digital technologies is still in its early stages. For example, existing representative systems (such as Sure Wash) are mainly based on a single visual image classification technology, which has limited monitoring dimensions, is easily affected by factors such as ambient light and occlusion, and suffers from insufficient accuracy and reliability, making it difficult to widely apply in real and complex clinical scenarios. More importantly, these existing technical solutions generally suffer from problems such as a single data collection modality and a lack of multi-dimensional scientific quantification of evaluation indicators, making it impossible to conduct a refined assessment of the rubbing duration, force, uniformity of disinfectant coverage, and actual cleaning effect on various anatomical sites. At the same time, existing systems lack real-time, intelligent interactive feedback capabilities, failing to provide immediate guidance and correction during the operation.

[0004] In summary, current methods, whether traditional manual or early attempts at intelligent systems, have failed to effectively address the three core pain points in surgical hand disinfection quality monitoring: high subjectivity in assessment, low quantitative accuracy, and lack of process guidance. Therefore, there is an urgent need for an innovative technological solution that can integrate multi-dimensional information to achieve high-precision quantitative assessment and intelligent real-time feedback. Summary of the Invention

[0005] The purpose of this invention is to overcome the problems existing in the prior art and provide an intelligent monitoring and evaluation system for surgical hand disinfection based on multimodal image fusion. This system aims to address the limitations of traditional methods and AI vision technologies, which suffer from single-dimensionality and lack of real-time monitoring and feedback of the hand disinfection process. The core of the system is to comprehensively perceive the hand disinfection process from three dimensions: visual recognition, thermodynamic effects, and hand motion recognition, through the collaborative work of multiple heterogeneous sensors. It then utilizes a deep learning fusion network for cross-validation and joint inference to ultimately achieve objective, accurate, and real-time intelligent evaluation and guidance.

[0006] To achieve the above-mentioned technical objectives and effects, the present invention is implemented through the following technical solution: A surgical hand disinfection intelligent monitoring and evaluation system based on multimodal image fusion is disclosed. The system includes a multimodal data acquisition module, a data synchronization and preprocessing module, a multimodal feature fusion analysis module, an intelligent evaluation and decision-making module, and a real-time interaction and feedback module. The multimodal data acquisition module is used to simultaneously acquire RGB images of the hand, temperature field distribution, and three-dimensional depth data; The data synchronization and preprocessing module is used to achieve spatiotemporal alignment of multi-source data, hand region segmentation, key point detection, and data cropping. The multimodal feature fusion analysis module is used to complete cross-modal joint reasoning through parallel feature extraction and a self-attention fusion network; The intelligent assessment and decision-making module is used to generate multi-dimensional quantitative scores and compliance judgments based on standard processes; The real-time interaction and feedback module is used to provide visual guidance feedback, voice correction, structured reports, data upload, and traceability reports.

[0007] Furthermore, the multimodal data acquisition module includes a visible light camera, an infrared thermal imager, and a depth sensor, wherein: The visible light camera is used to capture visual information of the hand surface, including hand shape outline, joint position, and skin texture. This is the dimension closest to human observation and provides a basis for step recognition and area localization. The infrared thermal imager is used to capture information invisible to the naked eye, including the temperature field distribution on the skin surface. By observing the spatiotemporal changes in the temperature field, it indirectly determines the uniformity of disinfectant coverage and the adequacy of rubbing. It overcomes the inherent limitations of traditional methods and visual technologies that cannot directly "see" whether disinfectant coverage is uniform or rubbing is adequate, and provides a new and effective indirect measurement method. The depth sensor is used to capture three-dimensional spatial information, including the precise spatial position of each point on the hand, surface curvature, and relative motion trajectory. By measuring the point cloud density changes and deformations of the three-dimensional hand model, it estimates the hand-washing action and rubbing direction and force, providing a geometric basis for estimating the percentage of area contact, effective coverage time, and average rubbing force.

[0008] Furthermore, the infrared thermal imager is used to indirectly infer the adequacy and uniformity of hand disinfection by observing the dynamic changes in the temperature field caused by the evaporation of disinfectant and the friction from rubbing.

[0009] Furthermore, the data synchronization and preprocessing module includes a trigger and a spatial calibration module, wherein: First, spatiotemporal alignment is performed, which involves hardware synchronization and spatial calibration to ensure pixel-level alignment of three types of data at the same hand position at the same time. These three types of data include: 1) Hand region segmentation: Using a lightweight model pre-trained on a large amount of hand data, the RGB image is processed to accurately segment the hands and wrist regions. 2) Key point detection: Use the model to detect key points of both hands, providing initial 2D pose information.

[0010] 3) Mapping and data cropping: The hand mask and key points on the RGB image are mapped onto the infrared and depth images through a calibration matrix, and the corresponding thermal imaging area of ​​the hand is cropped out, which greatly reduces the amount of subsequent calculations.

[0011] Furthermore, the multimodal feature fusion analysis module consists of a feature extraction network. It uses three parallel network branches to extract the feature vectors unique to each modality from RGB images, infrared thermal image sequences, and depth data, respectively. Then, feature fusion is performed. An attention-based fusion network is used to allow the model to automatically learn the weights of each modality feature in different scenarios.

[0012] Furthermore, the specific process of extracting characteristic parameters of the hand disinfection process from infrared thermal image sequences includes: 1) Calculation of regional temperature rise rate: For each anatomical region of the hand in the infrared video, calculate the temperature change gradient over a specific time period. This indicator reflects whether the region has been continuously and effectively rubbed. The frictional heat generated by continuous rubbing will cause the temperature to rise. If a region has no temperature rise for a long time, it means that it has not been rubbed sufficiently. 2) Temperature distribution uniformity index: Calculate the statistical variance of the temperature values ​​of the whole hand or a specific area and track its changes over time. This index reflects the uniformity of disinfectant application. Disinfectant evaporation will cause local cooling. When applied evenly, the temperature of the whole hand should show a relatively consistent downward trend. The raw thermal images are transformed into quantitative indicators with clear physical meaning and clinical relevance, providing a basis for subsequent fusion analysis and evaluation decisions.

[0013] Furthermore, the specific process of feature extraction and fusion is as follows: 1) Parallel feature extraction network: The input is a cropped RGB image sequence. The network is based on 3DCNN and temporal convolution and outputs the following feature vectors: classification probability of each step, confidence of color coverage of each anatomical region, and overall hand motion vector. It is trained using video data that includes the application of disinfectant and drying stages. Thermal Imaging Branch: The input is the registered infrared thermal image sequence, which analyzes the spatiotemporal evolution of the temperature field and outputs feature vectors: average hand temperature and temperature rise gradient of each region. 2) Feature fusion and decision network: The self-attention mechanism is adopted. The temporal features of the three channels are aligned in the time dimension and then input into the Transformer encoder. The multi-head self-attention mechanism is used to perform dynamic weight allocation and cross-modal association modeling. Finally, a multi-dimensional state vector is output to describe the hand-disinfection state of each preset area at the current time t. Each state is a comprehensive score that integrates coverage, rubbing time and thermal effect.

[0014] Furthermore, in the multimodal feature fusion analysis module, an association modeling and joint reasoning method for infrared thermal imaging features and other modal features is adopted. Specifically, it includes an infrared-visible light-depth three-modal cross-validation mechanism, which comprehensively utilizes information from three dimensions to perform consistency checks in order to reduce environmental interference and misjudgment rate. This can give full play to the unique advantages of thermal imaging technology, while making up for its weakness that may be affected by ambient temperature through multimodal fusion, achieving a 1+1+1>3 effect.

[0015] Furthermore, the intelligent assessment and decision-making module consists of a standardized knowledge base and a quantitative scoring model. Its quantitative scoring indicators include: effective coverage time of each anatomical region, rubbing uniformity index, disinfectant distribution uniformity index, regional contact area percentage, estimated average rubbing force, and temperature change consistency index. The real-time interaction and feedback module consists of a display unit and a voice interaction unit, and is used for: 1) Real-time feedback: Through interactive interface and voice, the evaluation results are transformed into actionable guidance, directly intervening in the current operation and forming an immediate closed loop of monitoring-evaluation-correction; 2) Data accumulation: All process data and personalized assessment reports are uploaded to the infection control platform to form a long-term management closed loop of operation-record-analysis-improvement.

[0016] A method for intelligent monitoring and evaluation of surgical hand disinfection based on multimodal image fusion, comprising the following steps: Step S1: Simultaneously acquire RGB images of the hand, infrared thermal imaging data, and 3D depth data; Step S2: Perform spatiotemporal alignment, manual segmentation, and preprocessing on the multi-source data; Step S3: Extract features from each modality and complete fusion inference using a self-attention mechanism; Step S4: Compare the hand disinfection process with the standard hand disinfection procedure to obtain a quantitative score and a compliance assessment; Step S5: Output audiovisual feedback in real time and generate and upload a structured report.

[0017] The beneficial effects of this invention are: 1. This invention employs indirect measurement and multimodal cross-validation. Thermal imaging technology is used to perceive the disinfection process, that is, to indirectly infer the adequacy and uniformity of hand disinfection by the dynamic changes in the temperature field caused by the evaporation of disinfectant and the friction of rubbing. Visual technology is used to identify rubbing actions, that is, to estimate the hand disinfection actions and the direction and force of rubbing by the changes in point cloud density and deformation of the three-dimensional model of the hand. An infrared-visible light-depth three-modal cross-validation mechanism is adopted to comprehensively use information from the three dimensions for consistency verification, so as to reduce environmental interference and misjudgment rate.

[0018] 2. The system structure of this invention is an end-to-end closed-loop intelligent system. A complete closed-loop system from data collection and intelligent analysis to real-time intervention and long-term management; forward channel: multimodal perception → fusion analysis → real-time feedback → guidance for current operations; backward channel: operational data → evaluation report → statistical analysis → improvement of training strategies; this closed-loop design of monitoring-evaluation-feedback-improvement enables the system to not only solve current operational quality problems, but also provide data support for long-term quality improvement, realizing the unity of technical and management value.

[0019] 3. This invention uses a multi-dimensional quantitative indicator system for evaluation. A completely new quantitative evaluation index system has been established, surpassing the traditional "time + number of steps, visual recognition" approach:

[0020] Based on these multi-dimensional quantitative indicators, an unprecedented level of precision is provided for the scientific measurement of hand disinfection quality. Attached Figure Description

[0021] Figure 1 This is a data flow diagram of the core algorithm of this invention; Figure 2 This is a schematic diagram illustrating the system deployment and configuration of the present invention. Detailed Implementation

[0022] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] 1. System hardware deployment and configuration like Figure 2 As shown, this embodiment integrates and deploys the system in the handwashing basin of the hospital operating room.

[0024] A hand motion recognition module is installed on the top of the handwashing sink: it uses an 8-megapixel global shutter professional camera with a wide-angle lens to ensure that the hands and forearms of medical staff can be fully captured within a working distance of 30-80cm, with a frame rate of 60fps to adapt to rapid hand movements.

[0025] An infrared thermal imaging unit is installed on the top of the handwashing sink: an infrared thermal imager with a resolution of 256x192 and a thermal sensitivity of <40mK@25°C is used. Its wavelength range (8-14μm) can accurately detect the subtle temperature changes of the skin caused by the evaporation of hand sanitizer and friction.

[0026] An embedded AI edge computing terminal is deployed below the sink. This device is responsible for running all core algorithms and has powerful parallel computing capabilities to process video streams in real time.

[0027] A 27-inch mirrored touchscreen display unit is installed directly in front of the sink. It integrates an identity recognition module and a speaker to display guidance feedback interface, voice prompts and alarms.

[0028] Access the hospital's intranet via Wi-Fi 6 or Gigabit Ethernet to communicate with the infection control data center.

[0029] 2. Software System and Algorithm Implementation The system software is built on the Linux framework, and the algorithm module mainly uses the PyTorch deep learning framework.

[0030] Step 1: Initialization and Triggering When medical staff stand in front of the handwashing station, the system detects the presence of a human target using a depth sensor, automatically activates, and the mirror touch display unit lights up, showing the prompt "Please take a sufficient amount of disinfectant." At the same time, the system enters the preparation state.

[0031] Step 2: Synchronous Acquisition and Preprocessing of Multimodal Data Hardware synchronization: By sending signals to the sensors, the multimodal data of visual features, thermal features and 3D geometric / motion features acquired at the same time are strictly aligned in time (error <1ms).

[0032] Real-time preprocessing: 1) Hand region segmentation: Using a lightweight model pre-trained on a large amount of hand data, the RGB image is processed to accurately segment the hands and wrist regions.

[0033] 2) Key point detection: Use the model to detect key points (knuckles, fingertips, etc.) of both hands, providing initial 2D pose information.

[0034] 3) Mapping and data cropping: The hand mask and key points on the RGB image are mapped onto the infrared and depth images through a calibration matrix, and the corresponding thermal imaging area of ​​the hand is cropped out, which greatly reduces the amount of subsequent calculations.

[0035] Step 3: Multimodal Feature Fusion Analysis 1) Parallel feature extraction network: The input is a cropped RGB image sequence. The network is based on 3DCNN and temporal convolution and outputs the following feature vectors: classification probability of each step, confidence of color coverage of each anatomical region, and overall hand motion vector. It is trained using video data that includes stages such as "applying disinfectant" and "drying".

[0036] Thermal Imaging Branch: The input is a sequence of registered thermal images. It analyzes the spatiotemporal evolution of the temperature field and outputs feature vectors: average hand temperature and temperature rise gradient of each region (such as palm and between fingers).

[0037] 2) Feature fusion and decision network: The system employs a self-attention mechanism, aligning the temporal features of the three channels along the time dimension before inputting them into a Transformer encoder. The core of this mechanism is a multi-head self-attention mechanism, which can simultaneously perform dynamic weight allocation and cross-modal correlation modeling. For example, when determining whether "fingers are disinfected," a higher weight is assigned to the "finger temperature rise feature of the infrared branch," while the weight of the "visual feature branch" is reduced. The final output is a multi-dimensional state vector describing the "hand disinfection state" of each preset area at the current time t. Each state is a comprehensive score (0-1) that integrates coverage, rubbing time, and thermal effect.

[0038] Step 4: Intelligent Assessment and Real-time Feedback Compliance checks and quantification: The system has a built-in surgical hand disinfection knowledge base, which specifies the minimum duration of the action (apply an appropriate amount of hand disinfectant to each part of both hands, the forearm and the lower 1 / 3 of the upper arm, and rub thoroughly for 3 to 5 minutes).

[0039] The system compares the sequence with the template in real time: when the status score of a region is above a set threshold (e.g., >0.8), the region is considered to be "effectively disinfecting hands".

[0040] Calculates and displays in real time: total time, time for the current step, and cumulative effective hand sanitization time for each area.

[0041] If an incorrect step sequence is detected (such as skipping to the back of the hand before completing the palm rubbing), or if the effective hand sanitizing time in a certain area is severely insufficient, it will be immediately marked as a "non-compliant item".

[0042] Visualization and voice feedback: The hand disinfection action is displayed in real time on the screen.

[0043] Area color coding: areas that have met the standards are displayed in green, areas that are being manually cleaned but have not yet met the standards are displayed in purple, and areas that have been ignored or have not been cleaned sufficiently are displayed in red and flash.

[0044] Dynamic guidance: Highlights the area to be cleaned next and displays a countdown for that step.

[0045] Voice prompts: "Now please clean the back of your left hand, rotating and rubbing," "Please note that the right thumb has not been cleaned long enough."

[0046] Step 5: Report Generation and Data Upload After the surgical hand disinfection process is completed (either the user leaves or the process is manually ended), the system automatically generates a structured report, which includes: 1) Comprehensive score: a weighted calculation based on the effective cleaning time of each area, the correctness of the sequence, and the overall uniformity (out of 100); 2) Quantitative indicator details: total time, time of each step, coverage of each area (%), average rubbing frequency, etc.; 3) Problem diagnosis: a list of all non-compliant items, linked to the video clips at the time of occurrence (multimodal synchronous playback); 4) Operator ID: linked via facial recognition; 5) The report is uploaded to the hospital infection control management platform via the network and stored in the database for long-term traceability, department ranking, and personal training records.

[0047] 3. System Training and Optimization Data collection: In the operating room of the East Wing, surgical hand disinfection data of medical staff were collected, and hand disinfection video data was recorded simultaneously and finely annotated (standard and non-standard action single frame).

[0048] Model training: First, pre-training is performed on self-collected data. Then, the complete "feature extraction + fusion decision" network is trained end-to-end using self-collected multimodal time series data.

[0049] Personalized adaptation: The system can learn common hand sizes and habitual movement patterns of different users, and improve the accuracy of user assessment by fine-tuning with a small amount of initial data. The system software is built on the Linux framework, and the algorithm module mainly uses the PyTorch deep learning framework.

[0050] 4. System robustness and privacy security design To ensure the system operates stably and reliably in real, complex clinical environments and complies with medical data regulations, this embodiment has been specifically designed as follows: 1) Enhanced Robustness: To address common interferences such as changes in operating room lighting and momentary occlusion caused by personnel movement, the system employs data augmentation techniques (such as simulating lighting changes and randomly occluding certain areas) during the model training phase to improve the algorithm's adaptability. Furthermore, the deployment of edge computing terminals ensures that core monitoring and feedback functions can continue to operate independently locally even during network fluctuations.

[0051] 2) Privacy and Data Security: The system strictly adheres to medical data security and privacy protection standards. All collected visual and thermal imaging data undergoes real-time anonymization (e.g., blurring faces) before transmission and storage. Only irreversibly anonymized data is associated with the operator's employee ID for quality analysis and management. All data is transmitted through the hospital's internal encrypted network and stored encrypted in the infection control data center. Access permissions are strictly controlled to ensure the privacy and security of patients and medical staff.

[0052] 5. Experimental verification and effect analysis To verify the effectiveness and reliability of this system, preliminary experimental verification was conducted: Test setup: In a simulated operating room handwashing environment, 30 medical staff (including surgeons and operating room nurses) were recruited to use this system to perform standard surgical hand disinfection procedures, and multimodal data was recorded synchronously throughout the process.

[0053] Gold Standard and Evaluation Metrics: Two senior hospital infection control experts were invited to independently conduct blind reviews of randomly selected surgical hand disinfection process videos. Their results served as the "gold standard" for evaluation. The system output results were compared with the gold standard, and the accuracy and Cohen's Kappa coefficient were calculated to assess consistency and reliability.

[0054] Verification results: 1) Step completeness judgment: The system's overall accuracy in judging whether each step of the "six-step handwashing method" is complete reaches 94.7% compared with the expert gold standard.

[0055] 2) Consistency of quantitative assessment: For the core quantitative indicator of "coverage in each region", the Kappa value of the system score and the expert score is 0.85, indicating that the two are highly consistent.

[0056] 3) Problem detection capability: The system successfully identified that 73.3% (22 / 30) of the testers had the common problem of "insufficient rubbing time in the finger gap area" in the first test, which is consistent with the expert observation conclusion.

[0057] Conclusion: Experimental results show that the system of the present invention can not only automatically and accurately complete the compliance judgment of surgical hand disinfection, but its quantitative assessment results are also highly consistent with expert judgments. Furthermore, it can effectively reveal operational details and defects that are easily overlooked in manual inspections, laying the foundation for the accuracy and practicality of clinical applications.

[0058] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A surgical hand disinfection intelligent monitoring and evaluation system based on multimodal image fusion, characterized in that, The system includes a multimodal data acquisition module, a data synchronization and preprocessing module, a multimodal feature fusion analysis module, an intelligent evaluation and decision-making module, and a real-time interaction and feedback module; among which: The multimodal data acquisition module is used to simultaneously acquire RGB images of the hand, temperature field distribution, and three-dimensional depth data; The data synchronization and preprocessing module is used to achieve spatiotemporal alignment of multi-source data, hand region segmentation, key point detection, and data cropping. The multimodal feature fusion analysis module is used to complete cross-modal joint reasoning through parallel feature extraction and a self-attention fusion network; The intelligent assessment and decision-making module is used to generate multi-dimensional quantitative scores and compliance judgments based on standard processes; The real-time interaction and feedback module is used to provide visual guidance feedback, voice correction, structured reports, data upload, and traceability reports.

2. The intelligent monitoring and evaluation system for surgical hand disinfection based on multimodal image fusion according to claim 1, characterized in that, The multimodal data acquisition module includes a visible light camera, an infrared thermal imager, and a depth sensor, wherein: The visible light camera is used to capture visual information of the hand surface, including hand shape outline, joint position, and skin texture, to provide basic step recognition and area localization basis; The infrared thermal imager is used to capture information invisible to the naked eye, including the temperature field distribution on the skin surface, and indirectly judges the uniformity of disinfectant coverage and the thoroughness of rubbing by the spatiotemporal changes of the temperature field; The depth sensor is used to capture three-dimensional spatial information, including the precise spatial position of each point on the hand, surface curvature, and relative motion trajectory. By measuring the point cloud density changes and deformations of the three-dimensional hand model, it estimates the hand-washing action and rubbing direction and force, providing a geometric basis for estimating the percentage of area contact, effective coverage time, and average rubbing force.

3. The intelligent monitoring and evaluation system for surgical hand disinfection based on multimodal image fusion according to claim 2, characterized in that, The infrared thermal imager is used to indirectly infer the adequacy and uniformity of hand disinfection by observing the dynamic changes in the temperature field caused by the evaporation of disinfectant and the friction from rubbing.

4. The intelligent monitoring and evaluation system for surgical hand disinfection based on multimodal image fusion according to claim 2, characterized in that, The data synchronization and preprocessing module includes a trigger and a spatial calibration module, wherein: First, spatiotemporal alignment is performed, which involves hardware synchronization and spatial calibration to ensure pixel-level alignment of three types of data at the same hand position at the same time. These three types of data include: 1) Hand region segmentation: Using a lightweight model pre-trained on a large amount of hand data, the RGB image is processed to accurately segment the hands and wrist regions. 2) Key point detection: Use the model to detect key points of both hands, providing initial 2D pose information; 3) Mapping and data cropping: The hand mask and key points on the RGB image are mapped onto the infrared and depth images through a calibration matrix, and the corresponding thermal imaging area of ​​the hand is cropped out, which greatly reduces the amount of subsequent calculations.

5. The intelligent monitoring and evaluation system for surgical hand disinfection based on multimodal image fusion according to claim 4, characterized in that, The multimodal feature fusion analysis module consists of a feature extraction network. It uses three parallel network branches to extract the feature vectors unique to each modality from RGB images, infrared thermal image sequences, and depth data, respectively. Then, feature fusion is performed. An attention-based fusion network is used to allow the model to automatically learn the weights of each modality feature in different scenarios.

6. The intelligent monitoring and evaluation system for surgical hand disinfection based on multimodal image fusion according to claim 5, characterized in that, The specific process of extracting feature parameters of the hand disinfection process from infrared thermal image sequences includes: 1) Calculation of regional temperature rise rate: For each anatomical region of the hand in the infrared video, calculate the temperature change gradient over a specific time period; 2) Temperature distribution uniformity index: Calculate the statistical variance of temperature values ​​for the whole hand or a specific area and track its changes over time.

7. The intelligent monitoring and evaluation system for surgical hand disinfection based on multimodal image fusion according to claim 5, characterized in that, The specific process of feature extraction and fusion is as follows: 1) Parallel feature extraction network: The input is a cropped RGB image sequence. The network is based on 3DCNN and temporal convolution and outputs the following feature vectors: classification probability of each step, confidence of color coverage of each anatomical region, and overall hand motion vector. It is trained using video data that includes the application of disinfectant and drying stages. Thermal Imaging Branch: The input is the registered infrared thermal image sequence, which analyzes the spatiotemporal evolution of the temperature field and outputs feature vectors: average hand temperature and temperature rise gradient of each region. 2) Feature fusion and decision network: The self-attention mechanism is adopted. The temporal features of the three channels are aligned in the time dimension and then input into the Transformer encoder. The multi-head self-attention mechanism is used to perform dynamic weight allocation and cross-modal association modeling. Finally, a multi-dimensional state vector is output to describe the hand-disinfection state of each preset area at the current time t. Each state is a comprehensive score that integrates coverage, rubbing time and thermal effect.

8. The intelligent monitoring and evaluation system for surgical hand disinfection based on multimodal image fusion according to claim 5, characterized in that, In the multimodal feature fusion analysis module, an association modeling and joint reasoning method for infrared thermal imaging features and other modal features is adopted. Specifically, it includes an infrared-visible light-depth three-modal cross-validation mechanism, which comprehensively utilizes information from three dimensions to perform consistency checks in order to reduce environmental interference and misjudgment rate.

9. The intelligent monitoring and evaluation system for surgical hand disinfection based on multimodal image fusion according to claim 7, characterized in that, The intelligent assessment and decision-making module consists of a standardized knowledge base and a quantitative scoring model. Its quantitative scoring indicators include: effective coverage time of each anatomical area, rubbing uniformity index, disinfectant distribution uniformity index, percentage of area contact area, estimated value of average rubbing force, and temperature change consistency index. The real-time interaction and feedback module consists of a display unit and a voice interaction unit, and is used for: 1) Real-time feedback: Through interactive interface and voice, the evaluation results are transformed into actionable guidance, directly intervening in the current operation and forming an immediate closed loop of monitoring-evaluation-correction; 2) Data accumulation: All process data and personalized assessment reports are uploaded to the infection control platform to form a long-term management closed loop of operation-record-analysis-improvement.

10. A method for intelligent monitoring and evaluation of surgical hand disinfection based on multimodal image fusion, characterized in that, The method includes the following steps: Step S1: Simultaneously acquire RGB images of the hand, infrared thermal imaging data, and 3D depth data; Step S2: Perform spatiotemporal alignment, manual segmentation, and preprocessing on the multi-source data; Step S3: Extract features from each modality and complete fusion inference using a self-attention mechanism; Step S4: Compare the hand disinfection process with the standard hand disinfection procedure to obtain a quantitative score and a compliance assessment; Step S5: Output audiovisual feedback in real time and generate and upload a structured report.

Images