Hand hygiene qualified behavior subject recognition method based on video stream

Abstract

The present invention discloses a method for identifying qualified hand hygiene behavior subjects based on video streams, comprising: obtaining a sequence of gait, posture, and hand movement feature image frames; performing hand position detection and tracking through a target detection model and an optical flow method; classifying the hand movements in each image frame through a fine-grained feature extraction model; judging the validity of the action classification results through a sliding window method, and outputting a final action classification sequence; performing user identity identification through an identity recognition network fusing gait, posture, and hand movement features to obtain user identity information; finally, combining the action classification sequence and related time statistics to comprehensively score the hand hygiene process, and associating the score with the user identity information to screen out qualified hand hygiene behavior subjects. The present invention can achieve high-precision, real-time hand hygiene behavior detection and user identity information identification, providing a more comprehensive solution for hand hygiene assessment and management.

Description

Technical Field

[0001] The present invention belongs to the field of medical and health detection technology, and more specifically, relates to a hand hygiene qualified behavior subject identification technology based on video stream. Background Art

[0002] Hand hygiene is a key measure for preventing hospital-acquired infections (HAIs) and controlling the spread of disease. In healthcare settings, medical staff's compliance with hand hygiene practices directly impacts patient safety. However, traditional hand hygiene monitoring relies heavily on manual oversight, which can lead to blind spots and inconsistent monitoring. To improve hand hygiene compliance and reduce the risk of HAIs in healthcare settings, automated hand hygiene behavior detection technology is crucial.

[0003] In existing automated detection technologies, hand hygiene behavior detection mainly uses target detection algorithms or video processing methods. Target detection algorithms analyze each frame of an image to quickly identify hand position and movement type, offering the advantage of real-time feedback. For example, the invention patent application with publication number CN118115953A proposes an intelligent monitoring method and device for surgical hand disinfection suitable for use in operating rooms. However, simple target detection algorithms are deficient in the accuracy of classifying and detecting similar gestures. For example, the subtle differences between the two similar gestures of "palms facing each other, fingers together, rubbing each other" and "palms facing each other, fingers crossed, rubbing each other" can easily lead to misjudgment. This is especially true in complex environments (such as lighting changes, hand occlusion, etc.), where missed or false detection of hand movements is likely to occur. In these cases, detection accuracy is significantly reduced, and even false positives can directly impact work progress and reduce system adaptability.

[0004] Video processing methods usually require first forming a complete video, then analyzing the entire video, and obtaining overall information about hand movements by processing all frames. For example, the invention patent application with publication number CN112836641A proposes a hand hygiene monitoring method based on machine vision. Although this video processing method has improved detection accuracy and adaptability, it still faces technical challenges such as difficulty in distinguishing similar gestures and unstable hand position tracking. In addition, the processing results usually have a large delay, and it is impossible to promptly feedback compliance information of hand hygiene operations to users.

[0005] In addition, most existing technologies ignore the relationship between hand hygiene behavior and individual identity. Due to the lack of identity information, it is impossible to accurately track and manage each individual's hand hygiene compliance, it is impossible to screen out subjects with qualified hand hygiene behavior, and it is difficult to provide personalized improvement suggestions and adaptive management based on user behavior characteristics.

[0006] Patent application CN113723241A proposes a hand hygiene monitoring system based on posture estimation, which uses an identity recognition module to identify medical staff. This approach requires the installation of hardware equipment such as entrance and exit identifiers, sensors near the handwashing sink and hand sanitizer, and signal receivers. Medical staff are also required to wear smart badges at all times, which are linked to their medical staff IDs. This increases hardware costs and makes it difficult to ensure that medical staff always wear the required badges and can be effectively identified. In environments where medical equipment is concentrated, radio frequency interference from the additional electronic equipment is also a factor that prevents its widespread application.

[0007] Therefore, it is necessary to develop a new hand hygiene behavior detection method that can solve the problems of missed detection and false detection of hand movements at a limited time cost, improve detection accuracy while providing real-time feedback, and identify the identity information of the hand hygiene subject without adding additional devices, so as to obtain the information of the hand hygiene qualified behavior subject in real time for personalized management and data analysis. Summary of the Invention

[0008] 1. Technical problems to be solved

[0009] In response to the problems in existing hand hygiene behavior detection technologies, such as missed or false detection of hand movements leading to decreased accuracy, and the inability to efficiently screen out qualified subjects of hand hygiene behavior due to the lack of identity information of hand hygiene subjects, the present invention proposes a method for identifying qualified subjects of hand hygiene behavior based on video streams. Videos of hand hygiene subjects walking towards a sink and completing the hand washing process are collected. First, hand movements are detected frame by frame using a target detection algorithm. Then, the optical flow method is applied to solve the problem of missed hand movement detection. The visual detection results are then refined using time series analysis technology to achieve high-precision, real-time hand movement detection. Simultaneously, the video stream is input into a multi-dimensional feature fusion network to extract the gait, posture, and hand movement features of the hand hygiene subject, and accurately identify the user identity information in the video stream; the hand hygiene behavior is associated with the user identity information, and the identity information of the qualified subjects of hand hygiene behavior is screened out, providing an automated solution for hand hygiene assessment and management. Furthermore, the present invention also provides electronic equipment and computer-readable storage media for implementing the above method.

[0010] 2. Technical solution

[0011] The purpose of the present invention is achieved through the following technical solutions.

[0012] The first aspect of the present invention discloses a method for identifying a subject of hand hygiene compliance based on a video stream, comprising the following steps:

[0013] Acquire a video stream that can reflect the user's gait information, hand washing posture information, and hand movement information, and form a gait feature image frame sequence, a posture feature image frame sequence, and a hand movement feature image frame sequence;

[0014] Input the hand motion feature image frame sequence into the pre-built object detection model to detect the hand position frame by frame, and obtain the detection frame used to represent the hand position information in the image frame;

[0015] Based on the detection frame, combined with the optical flow method, real-time tracking of hand position changes is performed. The tracking frame is used to represent the hand position in the frames that were missed during the hand position detection step. Missed frames are image frames where the hand position information is not detected despite the hand position being detected and the detection frame being obtained. This is mainly caused by occlusion or other reasons.

[0016] Extracting a hand ROI from an image frame based on the detection frame and the tracking frame to form a hand image, and arranging the hand image in time sequence to obtain a hand image frame sequence;

[0017] Inputting the hand image frame sequence into the fine-grained feature extraction model to perform fine-grained feature extraction, obtaining action classification results and action classification confidences of the hand actions in the image frames, and arranging the action classification results in time sequence to obtain an action classification result sequence;

[0018] The sliding window method is used to judge the validity of the actions in the action classification result sequence, and the time series analysis results are obtained. The time series analysis results are arranged in time sequence and the final action classification sequence is output;

[0019] After the hand hygiene process is completed, the total duration of the hand hygiene process and the duration of each hand movement are counted;

[0020] The gait feature image frame sequence, the posture feature image frame sequence, and the hand image frame sequence are input into the identity recognition network to perform user identity recognition and obtain user identity information; the identity recognition network is a multi-dimensional feature fusion network constructed based on gait, posture, and hand movement features;

[0021] The user's hand hygiene behavior is evaluated based on the final action classification sequence, the total duration of hand hygiene behavior, and the duration of each hand action. The evaluation results are associated with the user's identity information to screen out the hand hygiene behavior subjects whose evaluation results are qualified.

[0022] Preferably, the feature image frame acquisition step also includes: extracting image frames in the video stream according to a preset frame interval to obtain key image frames, arranging the key image frames in time sequence to obtain a gait key image frame sequence, a posture key image frame sequence, and a hand movement key image frame sequence, and using the above key image frame sequence as input for the user identity information recognition and hand position detection steps.

[0023] Preferably, in the hand position detection step, the target detection model is a YOLO model, and the detection frame (x, y, w, h) of the hand area in each image frame is output through forward propagation, where x and y are the coordinates of the center point of the detection frame, and w and h are the width and height of the detection frame. The image frame where the hand is not detected is a suspected missed detection frame;

[0024] Furthermore, the hand position change tracking step specifically includes:

[0025] S301, calculating the suspected missed frames F in the hand position detection step i The temporal gradient and spatial gradient of the previous K frames, K ≥ 1; the hand detection frame (x i-1 ,y i-1 , w i-1 , h i-1 ) to generate the suspected missed detection frame F i The optical flow tracking position of the center point coordinate of the detection frame (x i ,y i ), and obtain the tracking frame (x i ,y i , w i , h i );

[0026] S302, if the optical flow tracking position (x i ,y i ) falls within the image frame size range, and the current suspected missed frame F is determined i If it is a missed frame, the tracking frame of the missed frame (x i ,y i , w i , h i ) i and h i The value of w i-1 and h i-1 , get the tracking frame (x i ,y i , w i-1 , h i-1 ); On the contrary, if the optical flow tracking position (χ i ,y i ) falls outside the image frame size, it is judged that the hand has left the detection range and is not detected, so there is no need to calculate the tracking frame.

[0027] Furthermore, in the hand ROI extraction step, for the image frame in which the hand area is detected in the hand position detection step, the detection frame in the image frame is extracted as the ROI, and a hand image is formed; for the image frame in which the hand area is not detected in the hand position detection step, the tracking frame is extracted as the ROI, and a hand image is formed.

[0028] Furthermore, the validity of the actions in the action classification result sequence is determined by a sliding window method, specifically including:

[0029] Define a time window, including Z consecutive image frames, Z>2;

[0030] In the time window, the action classification detection results of each image frame are counted, the number of occurrences of each action in the time window is calculated, and the action with the highest number of occurrences A is recorded. j ;

[0031] Action A j The average confidence level wei avg，j The calculation formula is:

[0032]

[0033] Where Z is the number of frames in the window, wei l is the confidence of the classified action obtained in the lth frame, count(A j ) is action A in the time window j The number of occurrences of jl is the Kronecker function, A l is the hand movement corresponding to the lth frame;

[0034] If action A j The average confidence level wei avg，j If the value is higher than the set threshold and the cumulative number of occurrences within the time window exceeds the preset number, action A will be recorded separately. j and the average confidence wei avg，j A is the time series analysis result sequenced and the confidence level of time series analysis wei2, 0<wei2≤1;

[0035] On the contrary, if action A j If the average confidence of is lower than the set threshold, or the cumulative number of occurrences in the time window is lower than the preset number, the action classification results A are recorded respectively. detected And the corresponding confidence wei1 is the time series analysis result A sequenced and time series analysis confidence level wei2.

[0036] Preferably, in the hand movement classification step, the image frames in the hand image frame sequence are standardized and resized before being input into a fine-grained feature extraction model: the fine-grained feature extraction model is a lightweight convolutional neural network, which is lightweight through group convolution, structural reparameterization and random inactivation.

[0037] Specifically, lightweight convolutional neural networks include:

[0038] The input stage is constructed as follows: 3×3 convolution is used for downsampling to extract the spatial features of the input image and obtain the input feature map;

[0039] The fine-grained feature extraction stage is constructed as follows: at least one first intermediate module and at least one second intermediate module are configured to extract fine-grained features of hand movements; the first intermediate module and the second intermediate module adopt a structural reparameterization strategy;

[0040] The input stage is constructed as follows: a 1×1 convolutional layer is used to enhance the information interaction between channels, and then a global average pooling layer and two fully connected layers are used in sequence to output the action classification results and action classification confidence.

[0041] In the lightweight convolutional neural network, a network backbone for extracting fine-grained features of hand movements is configured with at least one first intermediate module and at least one second intermediate module;

[0042] The first intermediate module increases the dimension of the input feature map through 3×3 convolution, and then passes it through a 1×1 grouped convolution branch, a 3×3 grouped convolution branch, and a branch that only performs batch normalization. The outputs of the three branches are summed, and the dimension is reduced through a 1×1 convolution. Then, a random dropout layer is used for regularization, and finally a residual connection is performed with the input feature map to obtain the first output feature map;

[0043] The second intermediate module first passes the input feature map through a 1×1 grouped convolution branch, a 3×3 grouped convolution branch, and a branch that performs batch normalization only. The outputs of the three branches are then summed. The compression and excitation attention modules are then introduced to obtain the intermediate feature map. The intermediate feature map is then passed through 1×1 convolution modules for dimension increase and dimension decrease in sequence. It is then regularized through a random dropout layer and finally residually connected with the intermediate feature map to obtain the second output feature map.

[0044] The first intermediate module and the second intermediate module adopt a structural reparameterization strategy, so that the 1×1 grouped convolution branch, the 3×3 grouped convolution branch and the branch that only performs batch normalization in the first intermediate module and the second intermediate module are combined into a single 3×3 grouped convolution channel.

[0045] The identity network includes:

[0046] A feature extraction module is constructed to include a gait feature extraction branch, a posture feature extraction branch, and a hand movement feature extraction branch;

[0047] The feature fusion coding module is constructed to fuse the gait feature map, posture feature map and hand movement feature map output by the feature extraction module to form an individual feature code;

[0048] The code comparison module is constructed to compare the individual feature code with the template feature code in the pre-built feature code template library, and perform user identity recognition based on the comparison result.

[0049] The gait feature extraction branch uses the GaitSet model backbone to extract features from the preprocessed gait feature image frame sequence, and outputs the first gait feature map and the second gait feature map through the convolutional neural network part and the multi-layer global pipeline part respectively.

[0050] The posture feature extraction branch uses the OpenPose convolutional neural network model backbone to extract features from the image frames in the posture feature image frame sequence and outputs a posture feature map;

[0051] The hand motion feature extraction branch uses the OpenPose convolutional neural network model backbone to extract features from the hand images in the normalized resized hand image frame sequence and outputs a hand motion feature map.

[0052] Furthermore, the OpenPose convolutional neural network model backbone has a partial affinity field module and a key point heat map module. Through iterative optimization of the partial affinity field module and the key point heat map module, a feature map is obtained for two-dimensional representation of the position and correlation of key points of posture or hand movements.

[0053] During the iteration process, the L2 norm loss function is used to evaluate the iterative effects of some affinity field modules and key point heat map modules;

[0054] Among them, the loss function fL of the partial affinity field module is expressed as:

[0055]

[0056] Where C is the number of affinity fields, c is the affinity field index, p is the position of each pixel in the image, W(p) is the spatial weighted mask function, if there is no label at pixel p, then W(p) = 0, otherwise W(p) = 1, L c (p) is the predicted value of the affinity field of type c at pixel p, is the true value of the affinity field of type c at pixel p, ||·||2 represents the L2 norm;

[0057] Among them, the loss function f of the key point heat map s Expressed as:

[0058]

[0059] Where D is the number of key points, d is the key point index number, S d (p) is the predicted value of the key point of category d at pixel p, is the true value of the key point of category d at pixel p.

[0060] Among them, the feature fusion coding module is constructed as follows: fusing the first gait feature map, the second gait feature map, the posture feature map and the hand movement feature map, using horizontal pyramid mapping to optimize feature expression, mapping the fused features to a low-dimensional space, and forming individual feature coding.

[0061] During the training process, the feature fusion coding module learns the association between gait, posture and hand movement features and adjusts the model parameters by optimizing the triplet loss function value; the triplet loss function f TripletLoss The calculation formula is:

[0062]

[0063] Where, is the feature encoding of the anchor sample, Encode the features of the positive samples that belong to the same identity as the anchor point, It is the feature encoding of negative samples that belong to different identities from the anchor point, γ is the boundary value used to ensure that there is enough interval between positive and negative samples, and max means taking the maximum value.

[0064] The coding comparison module is constructed to: calculate the cosine similarity between the individual feature code and the template feature code in the pre-built feature code template library, and output the template identity corresponding to the template feature code with the highest similarity as the identification identity;

[0065] The calculation formula for cosine similarity is:

[0066]

[0067] Formula and, and Represent individual feature coding and template feature coding respectively.

[0068] The second aspect of the present invention further provides an electronic device, including a processor and a memory, wherein the processor is used to call a computer program stored in the memory to execute the method for identifying hand hygiene qualified behavior subjects based on video streams provided by the first aspect of the present invention.

[0069] The third aspect of the present invention further provides a computer-readable storage medium for storing program code, wherein the program code is used to execute the method for identifying hand hygiene qualified behavior subjects based on video streams provided by the first aspect of the present invention.

[0070] 3. Beneficial effects

[0071] Compared with the prior art, the present invention has the following advantages:

[0072] (1) Existing methods for detecting hand hygiene behaviors mostly rely on video processing, which requires analysis of the entire video. This requires a large amount of computation and often leads to processing delays and insufficient real-time feedback. Target detection algorithms are prone to false detection or missed detection when identifying similar gestures, especially in complex environments such as those with occlusion, insufficient lighting, or difficulty distinguishing the foreground and background of hand washing. The present invention first processes the video stream frame by frame using a target detection model, then tracks the missed frames using an optical flow method, and finally applies time series analysis technology to further analyze and confirm the classification results. This achieves real-time and accurate detection of hand movements at a relatively low time cost, and enables real-time and accurate classification of hand hygiene behaviors.

[0073] The present invention utilizes optical flow to accurately track and estimate hand positions in frames suspected of missed detection. This effectively compensates for missed detections, temporary occlusions, and detection failures often encountered by target detection models in complex backgrounds, significantly improving performance and adaptability across a wide range of application scenarios. Furthermore, optical flow offers the advantages of real-time performance and high computational efficiency, adapting to diverse hand movements and reducing reliance on single-frame detection algorithms, ensuring efficient and reliable hand tracking in complex and dynamic environments.

[0074] The present invention also incorporates time series analysis, combining detection results from multiple frames using a sliding window method to effectively determine the continuity and consistency of an action, significantly reducing the impact of single-frame detection errors. Furthermore, this approach, based on continuous frame analysis, improves the accuracy of classifying similar gestures and enhances the reliability of action recognition.

[0075] Furthermore, the present invention uses a lightweight convolutional neural network model in the hand motion classification process, adopting lightweight strategies such as group convolution, structural reparameterization, and random inactivation. In particular, the structural reparameterization strategy adopted enhances feature extraction and expression capabilities while ensuring computational efficiency, making it particularly suitable for detailed recognition of hand movements. Moreover, this model design can also enhance the generalization ability of the network, accelerate the inference process, enable it to process hand images in large-scale data sets in real time, effectively reduce computational complexity and memory consumption, and achieve the simultaneous completion of hand motion detection and classification during the hand hygiene process.

[0076] (2) The identification of the subject of hand hygiene based on video streams needs to cope with the challenges of the complex actual conditions and environmental factors of the medical environment. On the one hand, the handwashing sink is located indoors, surrounded by narrow passages or small spaces. The gait and posture video collected is limited in length in seconds, and the images are usually at a certain angle. On the other hand, the wrapped clothing of medical staff blurs the outline of the body, most of the body area is blocked, and the clothing of different individuals in the same environment is highly consistent, making user identification extremely difficult. The single feature identification method is not applicable.

[0077] To this end, the present invention constructs an identity recognition network through multi-dimensional feature fusion technology, and comprehensively utilizes three identity features of gait, posture and hand movements. It can effectively reduce the errors caused by environmental interference and ensure that the accuracy of identity recognition can be maintained in complex environments. The robustness and recognition accuracy are greatly improved.

[0078] Furthermore, the gait feature extraction branch in the identity recognition network of the present invention adopts the GaitSet model backbone comprising a convolutional neural network part and a multi-layer global pipeline part to extract two sets of gait features. On the one hand, it can enrich the hierarchy and diversity of feature representation; on the other hand, it can also make the gait features occupy a higher proportion in the entire identity recognition network, thereby improving the influence of gait features on the identity recognition results.

[0079] Furthermore, the posture feature extraction branch and hand movement feature extraction branch in the identity recognition network of the present invention use the OpenPose convolutional neural network model backbone that includes a partial affinity field module and a key point heat map module, which can effectively capture and understand the spatial and directional relationship between various detection parts of the human body, realize accurate modeling of human posture / hand movement, and accurately locate the key points of human posture / hand movement, thereby obtaining human posture / hand movement features.

[0080] Furthermore, the identity feature fusion coding module in the identity recognition network of the present invention adopts horizontal pyramid mapping to further optimize feature expression, maps the fusion features to a low-dimensional space, and the formed individual feature coding is compact and has recognition capability.

[0081] Furthermore, the encoding comparison module in the identity recognition network uses cosine similarity for feature comparison. Cosine similarity can measure the directional consistency of two feature vectors in high-dimensional space, avoid errors caused by differences in the size of feature vectors, and enable the network to focus on optimizing the semantic matching of features rather than just the numerical proximity, thereby improving the accuracy of the comparison.

[0082] (3) The present invention can detect hand movements in real time and complete user identity recognition, associating hand hygiene behavior with their identity information. This not only enables accurate tracking and evaluation of each user's hand hygiene behavior, screening out medical staff with qualified hand hygiene behavior, but also provides medical institutions with detailed behavioral data and accountability tracing to ensure the implementation of hand hygiene compliance. Moreover, based on the combination of identity information, it can identify the behavioral patterns of specific individuals, discover problems in a timely manner and conduct targeted interventions, thereby improving hand hygiene compliance, optimizing the behavior management of medical institutions, and providing reliable technical support for public health management. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 Schematic diagram of the process of the method for identifying subjects of hand hygiene compliance based on video streams of the present invention;

[0084] Figure 2 Schematic diagram of the framework of a lightweight convolutional neural network in an embodiment;

[0085] Figure 3 This is a structural diagram of the first intermediate module in the embodiment;

[0086] Figure 4 This is a schematic structural diagram of the second intermediate module in the embodiment;

[0087] Figure 5 Schematic diagram of hand hygiene behavior in the embodiment;

[0088] Figure 6 Schematic diagram of the structure of the GaitSet model feature extraction network in the embodiment;

[0089] Figure 7 Schematic diagram of the structure of the OpenPose model feature extraction network in the embodiment;

[0090] Figure 8 This is a structural diagram of a feature fusion coding module in an embodiment;

[0091] Figure 9 This is a flow chart of the identity recognition algorithm in the embodiment;

[0092] Figure 10 Schematic diagram of the network structure for implementing the method for identifying hand hygiene qualified behavior subjects based on video streams in an embodiment. DETAILED DESCRIPTION

[0093] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0094] The relevant technical terms involved in the present invention are explained as follows:

[0095] Camera: A device used to capture, record and process visual images, usually consisting of an image sensor, lens assembly, image processing unit, data storage and transmission interface, and power management module.

[0096] Hand hygiene: A general term for hand washing, hygienic hand disinfection and surgical hand disinfection performed by medical staff during their professional activities.

[0097] Hand hygiene behaviors: Hand hygiene behaviors specifically include the following 16 hand movements: 1. Wet your hands and wrists under running water; 2. Take soap or hand sanitizer; 3. Place your palms together and rub your fingers together; 4. Place your left palm against the back of your right hand and rub them together along the interdigital space; 5. Place your right palm against the back of your left hand and rub them together along the interdigital space; 6. Place your palms together and rub your fingers together; 7. Place your knuckles on top of each other and rub them together; 8. Bend your left knuckles and place them on your right palm. 9. Bend your right knuckles and rub them in a rotating motion in the palm of your left hand; 10. Rotate your left thumb in the palm of your right hand and rub it in a rotating motion; 11. Rotate your right thumb in the palm of your left hand and rub it in a rotating motion; 12. Put your five fingers together and rub your left fingertips in the palm of your right hand and rub them in a rotating motion; 13. Put your five fingers together and rub your right fingertips in the palm of your left hand and rub them in a rotating motion; 14. Rub your right wrist with your left hand; 15. Rub your left wrist with your right hand; 16. Rinse the soap or hand sanitizer foam from your fingertips toward your wrist.

[0098] Feature maps: These represent the multidimensional structure of data activations or responses across different dimensions in a feature space. Feature maps map raw data into a high-dimensional feature space using various feature extraction methods, capturing and describing the data's inherent characteristics and patterns. The multidimensional representation of feature maps can simultaneously express multiple feature information, preserve the spatial or structural information of the data, and enable hierarchical feature abstraction in multi-layer models.

[0099] Feature vector: An ordered vector that represents the numerical description of an object or data instance in each feature dimension in the feature space.

[0100] The embodiment of the present invention proposes a method for identifying qualified hand hygiene actors based on video streams, which aims to effectively identify the implementation of hand hygiene and associate it with the identity information of the actor. By combining a multi-stage processing flow of hand position detection, optical flow tracking and fine-grained feature extraction, real-time and accurate hand hygiene monitoring can be achieved in complex medical environments. At the same time, the identity information of the actor is identified based on gait, posture and hand washing movements. The present invention matches hand hygiene monitoring data with identity information, which helps in the daily behavior management of medical institutions and provides important technical support for improving hand hygiene compliance and public health management.

[0101] like Figure 1 As shown, the method for identifying subjects of hand hygiene compliance based on video streams proposed in an embodiment of the present invention mainly includes the following steps:

[0102] S100, Feature Image Frame Acquisition: A camera installed in front of the sink captures the user's gait as they approach the sink and their posture while washing their hands, generating video stream 1. A camera installed directly above the sink captures the user's hand movements while washing their hands, generating video stream 2. Different video segments or image frame sequences are intercepted from video stream 1 and video stream 2 to form corresponding feature image frame sequences.

[0103] The hand hygiene process begins when the image in video stream 2 begins to change, and ends when the image in video stream 2 remains unchanged for P seconds (P > 0). Specifically, the method for extracting different video segments or image frame sequences from video stream 1 and video stream 2 is as follows: The video segment Q seconds (Q ≥ 5) before the start of the hand hygiene process in video stream 1 is captured as the gait feature video segment, the video segment from the start to the end of the hand hygiene process in video stream 1 is captured as the posture feature video segment, and the video segment from the start to the end of the hand hygiene process in video stream 2 is captured as the hand movement feature video segment.

[0104] Output each frame of image (i.e., hand motion feature image frame) according to the hand motion feature video segment to form a hand motion feature image frame sequence {F1, F2, ..., F i ,......,F m The image frame sequence is fed into the subsequent motion detection process for processing and hand motion recognition. Each frame of the gait feature video segment, posture feature video segment, and hand motion feature video segment (gait feature image frame, posture feature image frame, and hand motion feature image frame) continuously output forms a gait feature image frame sequence, a posture feature image frame sequence, and a hand motion feature image frame sequence. These image frame sequences are then fed into the subsequent identity recognition process for processing and use in identity recognition.

[0105] Preferably, image frames are extracted from each video stream at a frame interval L to obtain key image frames, which are then arranged in chronological order to form a key image frame sequence. The key image frame sequence replaces the continuous frame-by-frame image frame sequence and serves as the gait feature image frame sequence, posture feature image frame sequence, and hand movement feature image frame sequence for subsequent hand hygiene behavior and identity recognition analysis. Preferably, the value of L is 3. Replacing the continuous frame-by-frame image frame sequence with the key image frame sequence further improves the real-time performance of hand movement detection and identity recognition.

[0106] Subsequent hand movement recognition uses hand movement image sequences from the handwashing process, combined with object detection and optical flow tracking technology, to extract and analyze these features to accurately identify and confirm the user's handwashing behavior in real time. Identity recognition is a system based on multidimensional feature fusion technology. By extracting and integrating three identity features: gait, posture, and hand movement, combined with feature extraction branches, feature fusion modules, and encoding modules, it achieves highly robust and accurate individual identity recognition in complex environments.

[0107] S200, hand position detection: During the hand hygiene process, the hand movement feature image frames obtained in S100 are analyzed in real time to perform hand position detection.

[0108] Here, hand position refers to the hands and forearms of the subject in the hand motion feature image frame. Before hand position detection, an initial object detection model should be constructed and trained for hand shape detection. After training, the model can be used to identify hand positions in each image frame. The object detection model can be built based on models such as YOLO and SSD. Preferably, the YOLO model is used.

[0109] The target detection model is used to perform preliminary detection of the hand position in each frame or key frame. The specific method is as follows: the hand movement feature image frame sequence is input into the target detection model, and through forward propagation, the detection box (x, y, w, h) of the hand area in each image frame is output, where x and y are the coordinates of the center point of the detection box, and w and h are the width and height of the detection box.

[0110] S300, hand position tracking: Based on the hand detection frame obtained in S200, the hand position change is tracked in real time in combination with the optical flow method, and the hand position in the missed detection frame where the hand is not detected is represented by the tracking frame.

[0111] In the hand position detection stage S200, for a continuous image frame sequence {F1, F2, ..., F i ,......,F m}, usually some image frames fail to detect hand information, which are called suspected missed detection frames. In this embodiment, the frame number of the suspected missed detection frame is set to i, F i There are two main reasons for missed detection: one is due to complex situations such as lighting changes, blurred hand boundaries, or occlusion, which lead to missed detection by the target detection model; the other is that the hand leaves the detection range and the hand position is not detected. i Tracking processing is performed to achieve continuous detection of hand position and movement.

[0112] The specific method of hand position tracking includes the following sub-steps:

[0113] S301, calculate the suspected missed detection frame F i The temporal gradient and spatial gradient of the previous K frames (K≥1) are used to estimate the hand detection frame (x i-1 ,y i-1 , w i-1 , h i-1 ) to generate the suspected missed detection frame F i The optical flow tracking position of the center point coordinate of the detection frame (x i ,y i ), and obtain the tracking frame (x i ,y i , wi , h i ).

[0114] S302, if the optical flow tracking position (x i ,y i ) falls within the image frame size range, the current suspected missed frame is judged to be a missed frame, then the tracking frame (x i ,y i , w i , h i ) i and h i The value of w i-1 and h i-1 , get the tracking frame (x i ,y i , w i-1 , h i-1 ), and subsequent processing continues according to the tracking frame of the missed frame. Conversely, if the optical flow tracking position falls outside the image frame size range, it is determined that the hand has left the detection range and was not detected, and the tracking frame and subsequent processing are not calculated.

[0115] The detailed process of the optical flow tracking algorithm in sub-step S301 is as follows:

[0116] S3011, for the previous K frames of image {F i-K , F i-K+1 ,......,F i-1} and the current frame image F i Grayscale conversion and Gaussian blur processing are performed to reduce the influence of noise.

[0117] S3012, calculate the two adjacent frame images F s-1 and F s The spatial gradient of (s=i-K+1, i-K+2, ..., i) is used to identify the edges and hand contours in the image and is calculated as:

[0118] E s，x =F s-1 *G x ,

[0119] E s，y =F s-1 *G y ,

[0120] Where, E s，x and E s,y are the spatial gradients in the x and y directions, G x and G y are the convolution kernels in the x and y directions respectively, and * is the convolution calculation.

[0121] Optional,

[0122] S3013: Calculate the temporal gradient between the current frame and the previous frame to represent the intensity change of the hand between consecutive frames. The calculation method is:

[0123] E s，t =F s-1 -F s ,

[0124] Where, E s，t is the temporal gradient between the previous frame and the current frame image.

[0125] Assuming that the optical flow of all points in a small neighborhood is the same, the optical flow constraint equation is established for each pair of adjacent frames according to the optical flow constraint equation:

[0126] E s,x u+E s,y v+E s,t =0,

[0127] Where u and v are the optical flow components of each pixel in the image in the x and y directions, respectively.

[0128] S3014, stack the optical flow constraint equations of the first K pairs of adjacent frames to form an overdetermined linear equation system:

[0129]

[0130] Where,

[0131] S3015, using the weighted least squares method to solve the optimal optical flow vector (u, v) of the overdetermined equations, the calculation method is:

[0132]

[0133] Where W is the diagonal weight matrix, which is used to assign different weights to the gradients between different frames.

[0134] Preferably, W=diag(1, 2, ..., K), where diag represents a diagonal matrix.

[0135] S3016, extract the optical flow vectors of all pixels in the detection frame and calculate the average motion speed of the pixels in the detection frame:

[0136]

[0137] Where k is the index of the pixel in the detection box, u avg and v avg are the average motion speeds of all pixels in the detection frame in the x and y directions, M is the number of pixels in the detection frame, and u k and v kis the motion speed of each pixel index k.

[0138] S3017, using the calculated average motion vector to update the hand position in the current frame:

[0139] x i =x i-1 +u avg ,

[0140] y i =y i-1 +v avg ,

[0141] Where x i-1 and y i-1 are the center point coordinates of the hand position in the previous frame respectively.

[0142] Then the missed detection frame F is not generated i The hand position tracking frame is (x i ,y i , w i-1 , h i-1 ), w i-1 and h i-1 are the width and height of the hand position detection box in the previous frame respectively.

[0143] It is worth noting that the present invention uses the optical flow method to track the position of the hand, which can ensure that the hand is i Tracking the exact hand position in the scene not only improves tracking stability but also effectively handles temporary occlusions and detection failures. Furthermore, optical flow offers the advantages of real-time performance and high computational efficiency, adapting to diverse hand movements and reducing reliance on single-frame detection algorithms. This ensures efficient and reliable hand tracking in complex and dynamic environments, significantly improving the performance of this method in a variety of application scenarios.

[0144] S400, Region of Interest (ROI) extraction: Based on the detection frame obtained in S200 and the tracking frame obtained in S300, the hand ROI in each image frame is extracted and sorted to obtain a hand image frame sequence.

[0145] Specifically, for image frames where S200 detects the hand area, the ROI is the detection frame area generated by the target detection model, i.e., the detection frame; for image frames where S200 does not detect the hand area, the position of the hand is tracked by the S300 optical flow method. If the optical flow tracking position is within the image area, the tracking frame is regarded as the ROI, and then the image frame is cropped according to the ROI to obtain a hand image to focus on the local area related to the hand washing action; the hand image is arranged in time sequence to obtain a hand image frame sequence.

[0146] It's worth noting that these ROIs primarily serve to ensure that subsequent analysis focuses solely on the relevant areas, minimizing background interference. The ROIs are dynamically updated across successive image frames to ensure consistent tracking of hand position. By tracking hand position using optical flow, the system can adjust the ROIs in real time, maintaining continuous focus on hand hygiene behavior.

[0147] The present invention extracts the region of interest, namely the hand area, in the image frame through ROI extraction technology. In the subsequent hand movement classification and user identity information recognition process, the background interference is reduced, which can effectively improve the image processing efficiency and movement detection accuracy.

[0148] S500, hand action classification: Through the fine-grained feature extraction model, the hand image extracted in S400 is deeply analyzed to identify the specific details of the hand action, obtain the hand action performed in the current image frame, and obtain the action classification result A detected And the corresponding action classification confidence wei1 (0<wei1<1), arrange the action classification results corresponding to each image frame in time sequence, and obtain the action classification result sequence.

[0149] It is understandable that before feature extraction of the hand image, since the size of the target detection frame in each frame is different, it is usually necessary to standardize the size of the hand image so that the image size is consistent in the model input stage and an image suitable for the input model is obtained.

[0150] In the embodiment, the fine-grained feature extraction model specifically uses a lightweight convolutional neural network, using lightweight strategies such as group convolution, structural reparameterization, and random inactivation. Its structure is as follows: Figure 2 As shown in the figure, it specifically includes the following contents: the input stage of the fine-grained feature extraction model uses 3×3 convolution for downsampling to preliminarily extract the spatial features of the input image and obtain the input feature map; the backbone of the fine-grained feature extraction model consists of the first intermediate module and the second intermediate module to extract the fine-grained features of the hand movements; the output stage uses a 1×1 convolution layer to enhance the information interaction between channels, and then passes through the global average pooling layer and two fully connected layers in sequence to finally output the action classification result A detected And the action classification confidence wei1. Among them, the backbone is composed of N1 first intermediate modules and N2 second intermediate modules connected in sequence, N1, N2 ≥ 1. The two intermediate modules adopt a structural reparameterization strategy, which can effectively reduce the computational complexity of the inference stage.

[0151] like Figure 3As shown in the figure, during the model training phase, the first intermediate module increases the dimension of the input feature map through 3×3 convolution, and then passes through the 1×1 group convolution branch, the 3×3 group convolution branch, and the branch that only performs batch normalization. The outputs of the three branches are then summed, and the dimension is reduced through 1×1 convolution. Then, regularization is performed through a random dropout layer, and a residual connection is performed with the input feature map to obtain the first output feature map. Figure 4 As shown in the figure, the second intermediate module first passes the input feature map through a 1×1 grouped convolution branch, a 3×3 grouped convolution branch, and a branch that performs only batch normalization, then adds the outputs of the three branches, and finally introduces a Squeeze and Excitation (SE) attention module to obtain an intermediate feature map. The intermediate feature map is sequentially passed through 1×1 convolution modules for dimension increase and dimension decrease, and then regularized through a random dropout layer. Finally, a residual connection is performed with the intermediate feature map to obtain the second output feature map.

[0152] like Figure 3 and Figure 4 As shown in Figure 1, the inference phase (i.e., application phase) is different from the model training phase. In order to improve the efficiency of hand movement recognition, the present invention adopts a targeted structural reparameterization strategy, so that the 1×1 grouped convolution branch, the 3×3 grouped convolution branch, and the branch that only performs batch normalization in the first and second intermediate modules are combined into a single 3×3 grouped convolution channel. For the g-th feature group I of the grouped convolution, g , the structural reparameterization process can be expressed as:

[0153]

[0154] Where G o g 、B o g are the convolution kernel and bias after structural reparameterization, G 3×3 g It is the convolution kernel of the 3×3 convolution branch before the structure is reparameterized, G 1×1，padding g It is the convolution kernel after filling the 1×1 convolution branch before the structure is reparameterized, G BN，structured g is the normalized convolution kernel constructed by the branch that only performs batch normalization before the structure is reparameterized, B 3×3 g 、B 1×1 g 、B BN g are the biases of the first three branches of the structural reparameterization.

[0155] It is worth noting that structural reparameterization plays a role in optimizing the network structure and improving inference efficiency in this model. By adopting a complex multi-branch structure during the training phase to enhance feature extraction capabilities, the structural reparameterization strategy can convert these multiple branches into single-branch convolution operations when the model is converted to the inference phase. This process effectively reduces the amount of computation and memory usage during the inference phase while maintaining the feature expression capabilities learned by the model during the training phase. In other words, structural reparameterization ensures that the model does not add additional computational burden during the inference process, making recognition more real-time and efficient, meeting the purpose of simultaneously completing hand movement detection and classification during the hand hygiene process.

[0156] The hand image is input into the fine-grained feature extraction model, and the action classification result A is output. detected And the action classification confidence wei1. The classification results include 16 types of hand action classifications in the terminology explanation section, which can be found in Figure 5 As shown. The lightweight convolutional neural network used in the present invention, combined with the structural reparameterization process, enhances the feature extraction and expression capabilities while ensuring computational efficiency, and is particularly suitable for detailed recognition of hand movements. Based on this model design, the present invention not only improves the generalization ability of the network, but also accelerates the reasoning process, enabling it to process hand images in large-scale data sets in real time, effectively reducing computational complexity and memory consumption, avoiding overfitting problems, and greatly improving the performance of the model in fine-grained image classification tasks. Especially on devices with limited computing resources or in application scenarios that require rapid convergence, this model performs particularly well, ensuring high precision and high efficiency in hand movement detection, and greatly improving the overall reliability and practicality of hand hygiene behavior evaluation.

[0157] S600, time series analysis operation: Use the sliding window method to determine the validity of the hand action classification results in the action classification result sequence, and obtain the time series analysis result A sequenced , and then output the final action classification sequence after arranging them in time sequence.

[0158] The specific method of the sliding window method to judge the effectiveness of the action is as follows: After the hand action classification is completed, a time window is defined, which usually includes Z (Z>2) consecutive image frames. Within the time window, the action classification detection results of each image frame are counted, and the number of occurrences of each hand action in the window is calculated, and the action with the highest number of occurrences A is recorded. j , where j is the hand action index with the highest number of occurrences in the window. By accumulating the detection results, we can get the action A j The average confidence level wei avg，j :

[0159]

[0160] Where Z is the number of frames in the window, wei l is the confidence of the classified action obtained in the lth frame, count(A j ) is action A in the time window j The number of occurrences of jl is the Kronecker function, A l is the hand movement corresponding to the lth frame.

[0161] If action A j The average confidence level wei avg，j If the value is higher than the set threshold and the cumulative number of occurrences within the window exceeds the preset number, action A is recorded separately. j and the average confidence wei avg，j A is the time series analysis result sequenced And the time series analysis confidence wei2 (0<wei2≤1). On the contrary, if action A j If the average confidence of is lower than the set threshold, or the cumulative number of occurrences in the window is lower than the preset number, the hand action classification results A are recorded respectively. detected And the corresponding confidence wei1 is the time series analysis result A sequenced The time series analysis results of each image frame in the hand hygiene process are arranged in time sequence A sequenced , output the final action classification sequence. Through this step, the overall recognition failure caused by single frame error can be avoided, thereby improving the accuracy of recognition and the stability of hand action detection.

[0162] S700, time statistics: After the hand hygiene process is completed, the total duration of the hand hygiene process and the duration of each hand movement are recorded.

[0163] The total duration of the hand hygiene process is calculated, that is, the duration of the hand motion feature video segment extracted in S100 is calculated, thereby obtaining the total duration of the hand hygiene process.

[0164] The duration of each hand movement in the final action classification sequence is counted to obtain the duration of each hand movement. The statistical method for the duration of each hand movement is as follows: according to the image frame interception frame rate fps, calculate the representative time t of each image frame fra =(1 / fps), count the number of times each action image frame appears in the final action classification sequence (A n ), where n represents the hand movement index, and then the duration of each hand movement is calculated as t A，n =t fra ·count(A n ).

[0165] S800, user identity recognition: the gait feature image frame sequence and the posture feature image frame sequence obtained in S100 and the hand movement feature image frame sequence formed in S400 are input into the identity recognition network to perform user identity recognition and obtain user identity information.

[0166] In real-world applications, identity verification must cope with a variety of complex real-world conditions and environmental factors, which can severely impact the accuracy of traditional authentication methods. This is particularly true in confined spaces, where camera placement and viewing angles are often limited, leading to image tilt, blur, and lighting variations. Furthermore, the uniform protective gear worn by medical staff (such as masks, gloves, and goggles) often obscures facial features, making facial recognition difficult.

[0167] The input for gait recognition is a sequence of walking video images, with the primary features extracted being the motion of each joint of the human body. This makes data collection non-invasive and acceptable. However, due to the large amount of data in the sequence images and the high computational complexity, processing is also difficult. Despite extensive research on gait in biomechanics, gait-based identity verification remains immature. Furthermore, in the medical environment used for hand hygiene testing, there is a high frequency of simultaneous presence of individuals wearing similar clothing styles. Factors such as the dynamic behavior of medical staff, obstructions from clothing, and environmental interference can also lead to errors in the detection of features such as posture and gait. Specifically, for handwashing, the walking gait video segment is only a few seconds long, providing limited information.

[0168] Given these practical difficulties, identification methods based solely on gait, posture, or hand movements alone cannot meet practical needs. To address this challenge, this paper uses multi-dimensional feature fusion technology to comprehensively utilize gait, posture, and hand movements to construct a more robust and efficient identification network. This network effectively reduces errors caused by environmental interference and ensures that identification accuracy is maintained even in complex environments.

[0169] The identity recognition network provided by the present invention is a multi-dimensional feature fusion network constructed based on gait, posture and hand movement features. It extracts the features of gait, posture and hand movement respectively through three different branches and feature extraction branches of different frameworks. Then, by fusing these features, the feature expression is further optimized using deep separable convolution and fully connected layers to obtain feature encoding representing individual identity.

[0170] At the same time, by collecting individual gait, posture and hand movement videos during registration, template feature coding is formed as the basis for recognition and comparison. By utilizing the limited nature of samples, the accuracy of identity authentication in a limited population is further improved.

[0171] It should be noted that there is no strict time sequence between this step and the aforementioned hand motion detection steps S500 to S600 and time statistics step S700 , and they can usually be performed simultaneously.

[0172] like Figure 9 As shown in Figure 2, the identity recognition network mainly consists of a feature extraction module, a feature fusion encoding module, and an encoding comparison module. The feature extraction module consists of three branches: gait feature extraction branch, posture feature extraction branch, and hand movement feature extraction branch. Each branch uses different frameworks (GaitSet and OpenPose) to extract gait, posture, and hand movement features respectively.

[0173] Among them, the gait feature extraction branch uses the GaitSet model backbone, and its structure is as follows Figure 6 As shown in the figure, it mainly includes a convolutional neural network part and a multi-layer global pipeline part. Before inputting the model, the gait feature image frame sequence in the video stream 1 is first preprocessed, including alignment, normalization and binarization, to obtain an image that meets the network input requirements. The preprocessed image is input into the convolutional neural network, and high-level frame-level features are extracted through multi-layer convolution and nonlinear activation functions to generate feature maps with channel, height and width dimensions; then, the set pooling (SP) operation is used to aggregate these frame-level features into gait set-level features to achieve invariance to the input order and adaptability to different numbers of inputs. The convolutional neural network part generates gait features. Figure 1 At the same time, the Multilayer Global Pipeline (MGP) part extracts and fuses the set-level features under different receptive fields at multiple convolutional levels to further enrich the hierarchy and diversity of feature representation. The Multilayer Global Pipeline part generates gait features. Figure 2 It is worth noting that, through the above design, the gait feature extraction branch outputs two feature maps, which makes the gait features occupy a higher proportion in the entire identity recognition network, thereby improving the influence of gait features on the identity recognition results.

[0174] Among them, the posture feature extraction branch uses the OpenPose convolutional neural network model backbone to extract the posture features of each image frame. Its structure is as follows Figure 7As shown in the figure, it mainly includes the VGG-19 network, the Part Affinity Fields (PAFs) module and the Confidence Maps module. Among them, the main function of the PAFs module is to capture and understand the spatial and directional relationship between the various detected parts of the human body, so as to achieve accurate modeling of the human posture; the main task of the Confidence Maps module is to locate the key points of the human body in the input feature image. First, the posture feature image in the video stream is adjusted to a predetermined size and then input into the convolutional neural network for feature extraction. Specifically, the model uses the first ten layers of the pre-trained VGG-19 network as the initial feature extractor to generate a high-dimensional feature map; the generated feature map is then input into the PAFs module. In the PAFs module, the model parameters are optimized through multiple iterations to improve the prediction accuracy of the partial affinity field; then, the intermediate feature map generated by the PAFs module is input into the Confidence Maps module, and the model parameters are optimized through multiple iterations to accurately locate the key points of the human body.

[0175] Throughout the network's multiple iterations, the predictions for PAFs and keypoint heatmaps are gradually refined. At each iteration, the predictions from the current stage are fused with the initial feature map and processed through successive convolutional modules (3×3 convolutional layers). The outputs of each convolutional module are cascaded to enhance the network's ability to capture multi-scale features.

[0176] In addition, at the end of each iteration, loss functions are applied to PAFs and key point heatmaps respectively. Specifically, the L2 norm loss function is used to evaluate the iterative effect of PAFs and key point heatmaps to ensure the accuracy of the association between parts and the accuracy of key point positioning. Among them, the loss function f of the PAFs module is L Expressed as:

[0177]

[0178] Where C is the number of affinity fields, c is the affinity field index, p is the position of each pixel in the image, W(p) is the spatial weighted mask function, if there is no label at pixel p, then W(p) = 0, otherwise W(p) = 1, L c (p) is the predicted value of the affinity field of type c at pixel p, is the true value of the affinity field of type c at pixel p, and ||·||2 represents the L2 norm.

[0179] Among them, the loss function f of the key point heat map module s Expressed as:

[0180]

[0181] Where D is the number of key points, d is the key point index number, S d (p) is the predicted value of the key point of category d at pixel p, is the true value of the key point of category d at pixel p.

[0182] Finally, after all iterative stages, a feature map representing the position and correlation of two-dimensional posture key points is generated for each frame. These frame-level features are aggregated into set-level features using the pooling operation and output as a posture feature map.

[0183] Among them, the hand motion feature extraction branch also uses the OpenPose convolutional neural network model backbone to extract the hand motion features of each image frame. The specific network structure and characteristics will not be repeated here. It should be noted that before performing hand motion feature extraction based on the hand image extracted by S400, the hand image should first be standardized and resized. The proportion of the hand image is guaranteed to remain unchanged by padding the excess part with zeros, so as to obtain an image suitable for the input model. The adjusted image is then input into the hand motion feature extraction branch. Each image is processed by the OpenPose iterative stage to generate a feature map for each frame representing the position and correlation of the key points of the two-dimensional hand motion. The set pooling operation is used to aggregate these frame-level features into hand motion set-level features to output the hand motion feature map.

[0184] Identity feature fusion encoding module: extract gait features Figure 1 , gait characteristics Figure 2 , posture feature maps and hand movement feature maps are fused, and horizontal pyramid mapping is used to further optimize feature expression. The fused features are mapped to low-dimensional space to form a compact and recognizable individual feature code.

[0185] Specifically, feature fusion is to divide each feature map into several strips through horizontal pyramid mapping (HPM). The feature fusion process is as follows: Figure 8 Specifically, for the horizontal pyramid map with a scaling scale of S, it first undergoes horizontal pyramid pooling (HPP), that is, at scales s∈1, 2, ..., S, the feature map is split into 2 in the height dimension. s-1 strips, that is, each feature map has a total of strips; then global pooling is applied to the three-dimensional strips to obtain one-dimensional features. Global pooling is the sum of global maximum pooling and global average pooling. Then, since strips of different scales depict features of different receptive fields, and different strips of each scale depict features of different spatial positions, an independent fully connected layer is used to map the features to the discriminant space. Finally, the identity feature vector, that is, the individual feature encoding, is generated through feature flattening and a fully connected layer.

[0186] It is worth noting that, considering that gait features have a strong correlation with the identification of user identity information, the identity feature vector obtained by the present invention integrates two gait feature maps, a posture feature map and a hand feature map, and contains comprehensive information of gait, posture and hand, and the proportion of gait information is greater than that of posture and hand information, which means that gait is the main detection basis, and posture and hand information play an auxiliary detection role. The fused identity feature vector is used to encode individual identity features to form feature fusion coding, that is, individual feature coding.

[0187] Coding comparison module: Match the obtained individual feature code with the template feature code stored in the database. Specifically, the cosine similarity function can be used to measure the similarity between feature vectors (that is, between the fused individual feature code and the template feature code in the feature code template library). The template identity corresponding to the template feature code with the highest cosine similarity to the individual feature code is output as the identification identity, thereby determining the identity information of the individual performing the action.

[0188] It is worth noting that cosine similarity can measure the directional consistency of two feature vectors in high-dimensional space, which can avoid errors caused by differences in the size of feature vectors, thereby allowing the network to focus on optimizing the semantic matching of features rather than just the proximity of numerical values. The calculation formula for cosine similarity is:

[0189]

[0190] Where, and Represent individual feature coding and template feature coding respectively.

[0191] It is understandable that during the model training process, the model parameters of the gait, posture, and hand movement feature extraction branches are frozen to maintain the extraction capabilities of the corresponding features of each branch, and only the model parameters of the feature fusion encoding stage are trained. The training process is combined with the triplet loss function to ensure that samples of the same identity in the feature space have a high similarity, while samples of different identities have a low similarity. By optimizing the triplet loss function value, the network can learn the association between gait, posture, and hand movement features, and gradually adjust the model parameters to improve the accuracy and robustness of feature extraction. Triplet loss function fTripletLoss The calculation formula is:

[0192]

[0193] Where, is the feature encoding of the anchor sample, Encode the features of the positive samples that belong to the same identity as the anchor point, It is the feature encoding of negative samples that belong to different identities from the anchor point, γ is the boundary value used to ensure that there is enough interval between positive and negative samples, and max means taking the maximum value.

[0194] It is understandable that the template feature code is formed during the user registration phase. During registration, the system uses a camera installed in front of the washbasin to capture video data of the user walking through the washbasin passage to the designated washbasin and completing the handwashing process, from which individual gait and posture information is obtained; and a camera installed above the washbasin captures video data of the user's handwashing process, from which individual handwashing process information is obtained. The above process information is forward-propagated through the trained identity recognition network to obtain the template feature code of each user. The template feature code is stored in the database to form a feature code template library, which is used as feature data that uniquely identifies its identity during the identity verification process.

[0195] It's worth noting that these template feature codes can be obtained by fusing multiple measurements of the same user. This involves collecting gait, posture, and handwashing data from the same individual multiple times, sampling at different times, locations, and environments, and then fusing these data using a weighted average to achieve a diverse and stable set of gait features. The template feature code is a unique feature vector in the database. This vector contains the numerical vectors of all the identity features extracted from the user and the relationships between each identity feature, enabling differentiation between different users.

[0196] S900, comprehensive evaluation and identity information association: The hand hygiene process is comprehensively evaluated based on the final action classification sequence, the total duration of the hand hygiene process, and the duration of each hand hygiene process. The comprehensive evaluation data is then associated with the identified user identity information to screen out user information with qualified hand hygiene behavior.

[0197] After the hand hygiene process is completed, the final action classification sequence output by S600, as well as the total duration of the hand hygiene process and the duration of each hand hygiene process obtained by S700, are combined to score the hand hygiene process using a pre-set comprehensive scoring system. Then, combined with the user identification information obtained by S800, a comprehensive score data for the user's hand hygiene is obtained. This score data can be counted as a hand hygiene data item and updated to the database in a timely manner. At the same time, user information with qualified hand hygiene behavior can be screened out through threshold comparison.

[0198] Furthermore, the present invention can also simultaneously record the score of each action in the hand hygiene data of this time, compare the comprehensive score with the preset threshold, and determine the user whose hand hygiene behavior is unqualified if it is lower than the specific threshold, and remind the specific user to perform hand hygiene again through the electronic screen and the sound speaker. If the comprehensive score shows that the time of a specific hand action is low and lower than the threshold corresponding to the action, the specific medical staff will be prompted to focus on training and strengthening during the next hand hygiene process. If the comprehensive score is higher than the passing score and lower than the specified score, an early warning reminder will be fed back when uploading the information. If the number of early warning reminders exceeds a certain threshold, the corresponding user will be prompted to relearn the hand hygiene method. It can be understood that in other embodiments, other evaluation methods besides comprehensive scoring can also be used.

[0199] In summary, the method for identifying hand hygiene qualified behavior subjects based on video streams provided by the present invention can Figure 10 To implement the network diagram, deep learning and optical flow tracking technology are combined to ensure accuracy and real-time performance in complex environments. First, multi-angle cameras are used to capture video streams of medical staff's hand hygiene behavior, extract key frames to reduce the amount of calculation, and combine with deep learning models to detect and track hand positions, effectively avoiding the problem of missed target detection; secondly, the fine-grained feature extraction network enhances feature expression capabilities and improves recognition accuracy while ensuring computational efficiency; in addition, time series analysis improves the stability of motion detection, avoiding overall failure due to single-frame misjudgment; finally, identity recognition integrates gait, posture, and hand motion features to achieve efficient and accurate identity authentication. This method has the advantages of high efficiency, accuracy, and strong robustness, and is suitable for practical application scenarios such as medical staff hand motion detection and user identity recognition.

[0200] The present invention combines hand motion classification data with user identity information, which can effectively improve hospital management efficiency, individual performance and postoperative infection control. For example, first, identity recognition can be linked with the operating room access control system to ensure that the identity of each person entering the operating room is clear and prevent unauthorized entry. At the same time, it automatically records the hand washing behavior of each medical staff member before and after the operation and associates this data with their identity information, thereby ensuring that medical staff strictly follow hand hygiene standards during each operation and reduce the risk of cross-infection; secondly, the combination of identity recognition and hand hygiene data can be used to improve individual performance. Customized improvement suggestions or training programs can be provided based on the hand hygiene operation of each medical staff member to help them improve their hand washing habits, thereby improving the standardization and safety of medical operations. In addition, by associating each hand washing behavior with specific surgical operations and postoperative patient infection data, it can provide hospitals with more accurate comprehensive analysis. This data-driven analysis not only helps to identify potential infection risks, but also optimizes the relationship between hand washing operations and operations, thereby providing a strong basis for the formulation of infection control strategies and reducing the incidence of postoperative infections in patients. In general, the combination of hand motion detection and identity recognition can bring significant benefits in individual behavior supervision, hospital management and infection control, and improve the quality of medical services and patient safety.

[0201] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the program code in the electronic device and computer-readable storage medium described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0202] Finally, it should be noted that the above schematically describes the invention and its implementation methods, which is not restrictive. Without departing from the spirit or basic characteristics of the invention, the invention can be implemented in other specific forms. What is shown in the accompanying drawings is only one of the implementation methods of the invention. The actual structure is not limited to this, and any figure marks in the claims should not limit the claims involved. Therefore, if a person of ordinary skill in the art is inspired by it, without departing from the purpose of the invention, a structural method and embodiment similar to the technical solution are designed without creativity, they should all fall within the scope of protection of this application. In addition, the word "including" does not exclude other elements or steps, and the word "one" before an element does not exclude the inclusion of "multiple" elements. The multiple elements stated in the product claim can also be implemented by one element through software or hardware. Words such as first and second are used to indicate names, and do not indicate any specific order.

[0203] Although the embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields. The above-mentioned specific embodiments are merely illustrative and instructive, and are not restrictive. A person skilled in the art, guided by this specification, may devise various forms without departing from the scope of protection of the claims of the present invention, all of which fall within the scope of protection of the present invention.

Claims

1. A method for identifying qualified hand hygiene behavior subjects based on video streams, characterized in that: The following steps are involved: Acquire a video stream that can reflect the user's gait information, hand washing posture information, and hand movement information, and form a gait feature image frame sequence, a posture feature image frame sequence, and a hand movement feature image frame sequence; Input the hand motion feature image frame sequence into the pre-built object detection model to detect the hand position frame by frame, and obtain the detection frame used to represent the hand position information in the image frame; Based on the detection frame, the optical flow method is combined to track the hand position changes in real time, and the tracking frame is used to represent the hand position in the frames that were missed in the hand position detection step; Extracting a hand ROI from an image frame based on the detection frame and the tracking frame to form a hand image, and arranging the hand image in a time sequence to obtain a hand image frame sequence; Input the hand image frame sequence into the fine-grained feature extraction model to perform fine-grained feature extraction, obtain the action classification results and action classification confidence of the hand actions in the image frames, arrange the action classification results in time sequence, and obtain the action classification result sequence; Determine the validity of actions in the action classification result sequence by a sliding window method to obtain a time series analysis result, arrange the time series analysis result in time sequence, and output a final action classification sequence; After the hand hygiene process is completed, the total duration of the hand hygiene process and the duration of each hand movement are counted; Inputting the gait feature image frame sequence, the posture feature image frame sequence, and the hand image frame sequence into an identity recognition network to perform user identity recognition to obtain user identity information; The identity recognition network is a multi-dimensional feature fusion network built based on gait, posture and hand movement features; The identity recognition network includes: a feature extraction module, which is constructed to have a gait feature extraction branch, a posture feature extraction branch, and a hand movement feature extraction branch; a feature fusion encoding module, which is constructed to fuse the first gait feature map, the second gait feature map, the posture feature map, and the hand movement feature map output by the feature extraction module, optimize the feature expression using horizontal pyramid mapping, and map the fused features to a low-dimensional space to form an individual feature code; a code comparison module, which is constructed to compare the individual feature code with the template feature code in a pre-built feature code template library and perform user identity recognition based on the comparison results; The user's hand hygiene behavior is evaluated based on the final action classification sequence, the total duration of hand hygiene behavior, and the duration of each hand action. The evaluation results are associated with the user's identity information to screen out the hand hygiene behavior subjects whose evaluation results are qualified.

2. The method for identifying hand hygiene qualified behavior subjects based on video stream according to claim 1, characterized in that: In the hand position detection step, a hand region detection frame (x, y, w, h) is output, where x and y are the coordinates of the center point of the detection frame, and w and h are the width and height of the detection frame; image frames in which the hand is not detected are suspected missed detection frames; The steps for tracking hand position changes include: S301, calculating the suspected missed frames F in the hand position detection step i The temporal gradient and spatial gradient of the previous K frames, K ≥ 1; the hand detection frame (x i-1 ,y i-1 , w i-1 , h i-1 ) to generate the suspected missed detection frame F i The optical flow tracking position of the center point coordinate of the detection frame (x i ,y i ), and obtain the tracking frame (x i ,y i , w i , h i ); S302, if the optical flow tracking position (x i ,y i ) falls within the image frame size range, and the current suspected missed frame F is determined i If it is a missed frame, the tracking frame of the missed frame (x i ,y i , w i , h i ) i and h i The value of w i-1 and h i-1 , get the tracking frame (x i-1 ,y i-1 ,w i-1 ,h i-1 ); On the contrary, if the optical flow tracking position (x i ,y i ) falls outside the image frame size, it is judged that the hand has left the detection range and is not detected, so there is no need to calculate the tracking frame.

3. The method for identifying qualified hand hygiene behavior subjects based on video streams according to claim 1, characterized in that: The determining the validity of the actions in the action classification result sequence by the sliding window method specifically includes: Define a time window, including Z consecutive image frames, Z>2; In the time window, the action classification detection results of each image frame are counted, the number of occurrences of each action in the time window is calculated, and the action with the highest number of occurrences A is recorded. j ; Action A j The average confidence level wei avg,j The calculation formula is: Where Z is the number of frames in the window, wei l is the confidence of the classified action obtained in the lth frame, count(A j ) is action A in the time window j The number of occurrences of jl is the Kronecker function, A l is the hand movement corresponding to the lth frame; If action A j The average confidence level wei avg,j If the value is higher than the set threshold and the cumulative number of occurrences within the time window exceeds the preset number, action A will be recorded separately. j and the average confidence wei avg,j A is the time series analysis result sequenced and the confidence level of time series analysis wei2, 0<wei2≤1; On the contrary, if action A j If the average confidence of is lower than the set threshold, or the cumulative number of occurrences in the time window is lower than the preset number, the action classification results A are recorded respectively. detected And the corresponding confidence wei1 is the time series analysis result A sequenced and time series analysis confidence level wei2.

4. The method for identifying hand hygiene qualified behavior subjects based on video stream according to claim 1, characterized in that: In the hand movement classification step, the image frames in the hand image frame sequence are standardized and resized before being input into the fine-grained feature extraction model; the fine-grained feature extraction model is a lightweight convolutional neural network, which achieves lightweightness through group convolution, structural reparameterization and random inactivation.

5. The method for identifying qualified hand hygiene behavior subjects based on video streams according to claim 4, characterized in that: The gait feature extraction branch uses the GaitSet model backbone to extract features from the image frames in the preprocessed gait feature image frame sequence, and outputs the first gait feature map and the second gait feature map through the convolutional neural network part and the multi-layer global pipeline part contained therein respectively; The posture feature extraction branch uses the OpenPose convolutional neural network model backbone to extract features from image frames in the posture feature image frame sequence and outputs a posture feature map; The hand motion feature extraction branch uses the OpenPose convolutional neural network model backbone to extract features from the hand images in the hand image frame sequence after standardized size adjustment, and outputs a hand motion feature map.

6. The method for identifying hand hygiene qualified behavior subjects based on video stream according to claim 5, characterized in that: The OpenPose convolutional neural network model backbone comprises a partial affinity field module and a key point heat map module, and through iterative optimization of the partial affinity field module and the key point heat map module, a feature map for two-dimensionally representing the positions and correlations of key points of postures or hand movements is obtained; During the iterative process, the L2 norm loss function is used to evaluate the iterative effects of the partial affinity field module and the key point heat map module; Among them, the loss function f of some affinity modules L Expressed as: Where C is the number of affinity fields, c is the affinity field index, p is the position of each pixel in the image, W(p) is the spatial weighted mask function, if there is no label at pixel p, then W(p) = 0, otherwise W(p) = 1, L c (p) is the predicted value of the affinity field of type c at pixel p, is the true value of the affinity field of type c at pixel p, ‖·‖2 represents the L2 norm; Among them, the loss function f of the key point heat map S Expressed as: Where D is the number of key points, d is the key point index number, S d (p) is the predicted value of the key point of category d at pixel p, is the true value of the key point of category d at pixel p.

7. The method for identifying qualified hand hygiene behavior subjects based on video streams according to claim 6, characterized in that: During the training process, the feature fusion coding module learns the association between gait, posture and hand movement features and adjusts the model parameters by optimizing the triplet loss function value; the triplet loss function f TripletLoss The calculation formula is: Where, is the feature encoding of the anchor sample, Encode the features of the positive samples that belong to the same identity as the anchor point, It is the feature encoding of negative samples that belong to different identities from the anchor point, γ is the boundary value used to ensure that there is enough interval between positive and negative samples, and max means taking the maximum value.

8. The method for identifying hand hygiene qualified behavior subjects based on video stream according to claim 7, characterized in that: The coding comparison module is constructed to: calculate the cosine similarity between the individual feature code and the template feature code in the pre-built feature code template library, and output the template identity corresponding to the template feature code with the highest similarity as the identification identity; The calculation formula of the cosine similarity is: Where, and Represent individual feature coding and template feature coding respectively.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor is used to call a computer program stored in the memory to execute the method for identifying hand hygiene qualified behavior subjects based on video streams according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store program code, and the program code is used to execute the hand hygiene qualified behavior subject identification method based on video stream according to any one of claims 1 to 8.

Images