Video stream-based hand hygiene qualification behavior subject identification method

Abstract

The invention discloses a video stream-based hand hygiene qualified behavior subject identification method for a hand hygiene behavior. The method comprises the following steps of: acquiring a gait, posture and hand action feature image frame sequence; hand position detection and tracking are performed through a target detection model and an optical flow method; classifying hand actions in each image frame through a fine-grained feature extraction model; the effectiveness of the action classification result is judged through a sliding window method, and a final action classification sequence is output; user identity recognition is carried out by fusing the gait, the posture and the hand action features through an identity recognition network, and user identity information is obtained; and finally, counting a comprehensive score of the hand hygiene process in combination with the action classification sequence and the related time, associating the score with the user identity information, and screening out a hand hygiene behavior subject with a qualified evaluation result. According to the invention, high-precision and real-time hand hygiene behavior detection and user identity information identification can be realized, and a more comprehensive scheme is provided for hand hygiene evaluation and management.

Description

Technical Field

[0001] The present invention belongs to the field of medical and health detection technology, and more specifically, relates to a hand hygiene qualified behavior subject recognition technology based on video stream. Background Art

[0002] Hand hygiene is a key measure to prevent hospital infection and control the spread of disease. In the medical environment, the hand hygiene compliance of medical staff directly affects the safety of patients. However, traditional hand hygiene monitoring mostly relies on manual supervision, which has problems such as blind spots and inconsistent supervision. In order to improve hand hygiene compliance in the medical environment and reduce the risk of hospital infection, automated hand hygiene behavior detection technology is particularly important.

[0003] In the existing automated detection technology, hand hygiene behavior detection mainly adopts target detection algorithm or video processing method. The target detection algorithm analyzes each frame of the image to quickly identify the hand position and action type, and has the advantage of real-time feedback. For example, the invention patent application with the publication number CN118115953A proposes an intelligent monitoring method and device for surgical hand disinfection suitable for operating rooms, but the simple target detection algorithm is insufficient in the classification and detection accuracy of similar gestures. For example, the slight difference between the two similar gestures of "palms facing each other, fingers together and rubbing each other" and "palms facing each other, fingers crossed and rubbing each other" can easily lead to misjudgment, especially in complex environments (such as lighting changes, hand occlusion, etc.), it is easy to miss or misdetect hand movements. At this time, the detection accuracy is significantly reduced, and even false alarms and failures directly affect the progress of work and the adaptability of the system becomes poor.

[0004] Video processing methods usually need to first form a complete video, then analyze the entire video, and process all frames to obtain overall information about hand movements. For example, the invention patent application with publication number CN112836641A proposes a hand hygiene monitoring method based on machine vision. Although this video processing method has improved detection accuracy and adaptability, it still faces technical challenges such as difficulty in distinguishing similar gestures and unstable hand position tracking. In addition, the processing results usually have a large delay, and it is impossible to promptly feedback the compliance information of hand hygiene operations to users.

[0005] In addition, most existing technologies ignore the relationship between hand hygiene behavior and individual identity. Due to the lack of identity information, it is impossible to accurately track and manage the hand hygiene compliance of each individual, it is impossible to screen out subjects with qualified hand hygiene behavior, and it is difficult to provide personalized improvement suggestions and adaptive management based on user behavior characteristics.

[0006] The invention patent application with the publication number CN113723241A proposes a hand hygiene monitoring system based on pose estimation, which identifies the identities of medical staff through an identity recognition module. This method requires the on-site installation of hardware devices such as entrance and exit identifiers, sensors beside washbasins and hand disinfectants, and signal receivers. Medical staff also need to wear smart name tags at all times and bind them to their staff numbers. On the one hand, this increases the hardware cost, and at the same time, it is difficult to ensure that medical staff always wear the name tags as required and can be effectively identified. In an environment where medical equipment is relatively concentrated, the radio frequency interference of additional electronic devices is also one of the factors that prevent its widespread application.

[0007] Therefore, it is necessary to develop a new method for detecting hand hygiene behavior, which can solve the problems of missed detection and false detection of hand movements at a limited time cost, provide real-time feedback while improving the detection accuracy, and identify the identity information of the hand hygiene subject without adding additional devices, so as to obtain the information of the qualified hand hygiene behavior subjects in real time for personalized management and data analysis. Summary of the Invention

[0008] 1. Technical Problems to be Solved

[0009] Aiming at the problems in the existing hand hygiene behavior detection technology, such as the decrease in accuracy caused by missed detection or false detection of hand movements, and the lack of identity information of the hand hygiene subject, resulting in the inability to efficiently screen out the qualified hand hygiene behavior subjects, the present invention proposes a method for identifying qualified hand hygiene behavior subjects based on video streams. Videos of hand hygiene subjects walking towards the washbasin and completing the handwashing process are collected. First, the hand movements are detected frame by frame through a target detection algorithm, then the optical flow method is applied to solve the problem of missed detection of hand movements, and finally, time series analysis technology is used to refine the visual detection results to achieve high-precision and real-time hand movement detection. Synchronously, the video stream is input into a multi-dimensional feature fusion network to extract the gait, posture, and hand movement features of the hand hygiene subject, and accurately identify the user identity information in the video stream; associate the hand hygiene behavior with the user identity information, and screen out the identity information of the qualified hand hygiene behavior subjects to provide an automated solution for hand hygiene assessment and management. Further, the present invention also provides an electronic device and a computer-readable storage medium for implementing the above method.

[0010] 2. Technical Solutions

[0011] The object of the present invention is achieved through the following technical solutions.

[0012] The first aspect of the present invention discloses a method for identifying qualified hand hygiene behavior subjects based on video streams, including the following steps:

[0013] Obtain a video stream that can reflect the user's gait information, handwashing posture information, and hand movement information, and form a gait feature image frame sequence, a posture feature image frame sequence, and a hand movement feature image frame sequence;

[0014] Input the hand movement feature image frame sequence into a pre-constructed object detection model to detect the hand position frame by frame, and obtain a detection box for characterizing the hand position information in the image frame;

[0015] Based on the detection box, combine the optical flow method to real-time track the change of the hand position, and characterize the hand position in the missed detection frames in the hand position detection step through a tracking box. The missed detection frames are image frames that should have detected the hand position to obtain a detection box but failed to detect the hand position information. This is mainly caused by occlusion or other reasons.

[0016] Based on the detection box and the tracking box, extract the hand ROI in the image frame, form a hand image, arrange the hand images in time sequence, and obtain a hand image frame sequence;

[0017] Input the hand image frame sequence into a fine-grained feature extraction model for fine-grained feature extraction, obtain the action classification result and action classification confidence of the hand movement in the image frame, arrange the action classification results in time sequence, and obtain an action classification result sequence;

[0018] Judge the validity of the actions in the action classification result sequence through the sliding window method, obtain the time series analysis result, arrange the time series analysis results in time sequence, and output the final action classification sequence;

[0019] After the hand hygiene process ends, count the total duration of the hand hygiene process and the duration of each hand movement;

[0020] Input the gait feature image frame sequence, the posture feature image frame sequence, and the hand image frame sequence into an identity recognition network for user identity recognition to obtain user identity information; the identity recognition network is a multi-dimensional feature fusion network constructed based on gait, posture, and hand movement features;

[0021] Combine the final action classification sequence, the total duration of the hand hygiene behavior, and the duration of each hand movement to evaluate the user's hand hygiene behavior, associate the evaluation result with the user identity information, and screen out the hand hygiene behavior subjects with qualified evaluation results.

[0022] Preferably, the feature image frame acquisition step further includes: extracting image frames in the video stream at a preset frame interval to obtain key image frames, arranging the key image frames in time sequence respectively to obtain a gait key image frame sequence, a posture key image frame sequence, and a hand movement key image frame sequence, and using the above key image frame sequences as the input for the user identity information recognition and hand position detection steps.

[0023] Preferably, in the hand position detection step, the target detection model is the YOLO model, and the detection box (x, y, w, h) of the hand area in each image frame output by forward propagation is obtained, where x and y are the coordinates of the center point of the detection box, and w and h are the width and height of the detection box. An image frame without a detected hand is a suspected undetected frame;

[0024] Further, the hand position change tracking step specifically includes:

[0025] S301, calculate the temporal gradient and spatial gradient of the K frames of images before the suspected undetected frame F i in the hand position detection step, where K ≥ 1; estimate the average motion vector of the detection box (x i-1 , y i-1 , w i-1 , h i-1 ) of the hand by the optical flow method, and generate the optical flow tracking position (x i , y i ) of the center point of the detection box of the suspected undetected frame F i to obtain a tracking box (x i , y i , w i , h i ) representing the hand position in the suspected undetected frame;

[0026] S302, if the optical flow tracking position (x i , y i ) falls within the image frame size range, determine that the current suspected undetected frame F i is an undetected frame, and the values of w i and h i in the tracking box (x i , y i ) of the undetected frame are w i and h i , and obtain the tracking box (x i-1 , y i-1 , w i , y i , w i-1 , h i-1 ); conversely, if the optical flow tracking position (χ i , y i ) falls outside the image frame size range, it is determined that the hand has left the detection range and no tracking box needs to be calculated.

[0027] Further, in the hand ROI extraction step, for the image frames in which the hand area is detected in the hand position detection step, the detection box in the image frame is extracted as the ROI, and a hand image is formed; for the image frames in which the hand area is not detected in the hand position detection step, the tracking box is extracted as the ROI, and a hand image is formed.

[0028] Further, the effectiveness of actions in the action classification result sequence is judged by a sliding window method, which specifically includes:

[0029] Define a time window, including consecutive Z image frames, where Z > 2;

[0030] Within the time window, count the action classification detection results of each image frame, calculate the number of occurrences of each action within the time window, and record the action A with the highest number of occurrences j ;

[0031] Action A j The average confidence level wei avg，j The calculation formula is:

[0032]

[0033] In the formula, Z is the number of frames within the window, wei l is the confidence level of the classified action obtained from the l-th frame, count(A j ) is the number of occurrences of action A within the time window j δ jl is the Kronecker function, A l is the hand action corresponding to the l-th frame;

[0034] If the average confidence level wei j of action A avg，j is higher than the set threshold, and the cumulative number of occurrences within the time window exceeds the preset number, then record action A j and the average confidence level wei avg，j as the time series analysis result A sequenced and the time series analysis confidence level wei 2 , where 0 < wei 2 ≤1;

[0035] Conversely, if the average confidence level of action A j is lower than the set threshold, or the cumulative number of occurrences within the time window is lower than the preset number, then record the action classification result A detected and the corresponding confidence level wei 1 as the time series analysis result A sequenced and the time series analysis confidence level wei 2 .

[0036] Preferably, in the hand action classification step, the image frames in the hand image frame sequence are input into the fine-grained feature extraction model after normalizing the size: the fine-grained feature extraction model is a lightweight convolutional neural network, and the lightweight convolutional neural network realizes lightweight through grouped convolution, structural reparameterization, and dropout.

[0037] Specifically, the lightweight convolutional neural network includes:

[0038] Input stage, which is constructed as: performing downsampling using 3×3 convolution to extract the spatial features of the input image and obtain the input feature map;

[0039] Fine-grained feature extraction stage, which is constructed as: configured with at least one first intermediate module and at least one second intermediate module to extract the fine-grained features of hand actions; the first intermediate module and the second intermediate module adopt the structural reparameterization strategy;

[0040] Input stage, which is constructed as: enhancing the information interaction between channels through a 1×1 convolutional layer, and then successively passing through a global average pooling layer and two fully connected layers to output the action classification result and the action classification confidence.

[0041] In the lightweight convolutional neural network, the network backbone for extracting the fine-grained features of hand actions is configured with at least one first intermediate module and at least one second intermediate module;

[0042] The first intermediate module upsamples the input feature map through 3×3 convolution, and then respectively passes through a 1×1 grouped convolution branch, a 3×3 grouped convolution branch, and a branch that only performs batch normalization. Then, the outputs of the three branches are summed, and after dimensionality reduction through 1×1 convolution, regularization processing is performed through a dropout layer. Finally, a residual connection is made with the input feature map to obtain the first output feature map;

[0043] The second intermediate module first passes the input feature map through a 1×1 grouped convolution branch, a 3×3 grouped convolution branch, and a branch that only performs batch normalization. Then, the outputs of the three branches are summed. Subsequently, a squeeze-and-excitation attention module is introduced to obtain an intermediate feature map. The intermediate feature map is successively passed through 1×1 convolutional modules for dimensionality increase and decrease, and then regularization processing is performed through a dropout layer. Finally, a residual connection is made with the intermediate feature map to obtain the second output feature map;

[0044] The first intermediate module and the second intermediate module adopt the structural reparameterization strategy, combining the 1×1 grouped convolution branch, the 3×3 grouped convolution branch, and the branch that only performs batch normalization in the first intermediate module and the second intermediate module into a single 3×3 grouped convolution channel.

[0045] The identity recognition network includes:

[0046] Feature extraction module, which is constructed to have a gait feature extraction branch, a posture feature extraction branch, and a hand action feature extraction branch;

[0047] The feature fusion and encoding module is configured to fuse the gait feature map, the pose feature map, and the hand action feature map output by the feature extraction module to form an individual feature encoding;

[0048] The encoding comparison module is configured to compare the individual feature encoding with the template feature encoding in the pre-constructed feature encoding template library, and perform user identity recognition according to the comparison result.

[0049] The gait feature extraction branch uses the GaitSet model backbone to extract features from the image frames in the pre-processed gait feature image frame sequence, and outputs the first gait feature map and the second gait feature map through the convolutional neural network part and the multi-layer global pipeline part included therein;

[0050] The pose feature extraction branch uses the OpenPose convolutional neural network model backbone to extract features from the image frames in the pose feature image frame sequence, and outputs a pose feature map;

[0051] The hand action feature extraction branch uses the OpenPose convolutional neural network model backbone to extract features from the hand images in the hand image frame sequence after the standardized size adjustment, and outputs a hand action feature map.

[0052] Furthermore, the OpenPose convolutional neural network model backbone has a part affinity field module and a key point heat map module. Through the iterative optimization of the part affinity field module and the key point heat map module, a feature map for two-dimensionally representing the position and correlation of the pose or hand action key points is obtained;

[0053] In the iterative process, the L2 norm loss function is used to evaluate the iterative effect of the part affinity field module and the key point heat map module;

[0054] Among them, the loss function fL of the part affinity field module is expressed as:

[0055]

[0056] In the formula, C is the number of affinity fields, c is the affinity field index number, p is each pixel position in the image, W(p) is the spatial weighting mask function. If there is no annotation at pixel p, then W(p)=0, otherwise W(p)=1, L c (p) is the predicted value of the c-class affinity field at pixel p, is the true value of the c-class affinity field at pixel p, ||·|| 2 represents the L2 norm;

[0057] Among them, the loss function f of the key point heat map s is expressed as:

[0058]

[0059] In the formula, D is the number of key points, d is the key point index number, and S d (p) is the predicted value of the d-type key point at pixel p, is the true value of the d-type key point at pixel p.

[0060] Among them, the feature fusion and encoding module is configured to: fuse the first gait feature map, the second gait feature map, the pose feature map, and the hand action feature map, optimize the feature expression by using horizontal pyramid mapping, map the fused features to a low-dimensional space, and form an individual feature encoding.

[0061] During the training process of the feature fusion and encoding module, by optimizing the value of the triplet loss function, learn the association between gait, pose, and hand action features, and adjust the model parameters; the triplet loss function f TripletLoss The calculation formula of is:

[0062]

[0063] In the formula, is the feature encoding of the anchor sample, is the positive sample feature encoding belonging to the same identity as the anchor, is the negative sample feature encoding belonging to a different identity from the anchor, γ is the boundary value used to ensure that there is a sufficient interval between positive and negative samples, and max represents taking the maximum value.

[0064] The encoding comparison module is configured to: calculate the cosine similarity between the individual feature encoding and the template feature encoding in the pre-constructed feature encoding template library, and output the template identity corresponding to the template feature encoding with the highest similarity as the recognized identity;

[0065] The calculation formula of the cosine similarity is:

[0066]

[0067] In the formula, and respectively represent the individual feature encoding and the template feature encoding.

[0068] The second aspect of the present invention also provides an electronic device, including a processor and a memory. The processor is used to call the computer program stored in the memory and execute the method for identifying the subject of qualified hand hygiene behavior based on video stream provided in the first aspect of the present invention.

[0069] The third aspect of the present invention also provides a computer-readable storage medium for storing program codes, and the program codes are used to execute the method for identifying the subject of qualified hand hygiene behavior based on video stream provided in the first aspect of the present invention.

[0070] 3. Beneficial effects

[0071] Compared with the prior art, the present invention has the following advantages:

[0072] (1) Most of the existing hand hygiene behavior detection methods rely on video processing and need to analyze the entire formed video, resulting in a large amount of calculation, often leading to processing delays and insufficient real-time feedback. When the object detection algorithm identifies similar gestures, especially in complex environments such as occlusion, insufficient light, or difficult distinction between the foreground and background of handwashing, false detection or missed detection is likely to occur. The present invention first processes the video stream frame by frame through an object detection model, then combines the optical flow method to track the missed detection frames, and finally applies time series analysis technology to further analyze and confirm the classification results, achieving real-time and accurate detection of hand movements at a small time cost and realizing real-time and accurate classification of hand hygiene behaviors.

[0073] Among them, the present invention uses the optical flow method to accurately track and estimate the hand position in the suspected missed detection frames, which can effectively make up for the detection loss of the object detection model in complex backgrounds, as well as temporary occlusion and detection failures, greatly improving the performance and adaptability in various application scenarios. In addition, the optical flow method also has the advantages of real-time and high computational efficiency, can adapt to diverse hand movements, reduce the dependence on single-frame detection algorithms, and ensure efficient and reliable hand tracking in complex dynamic environments.

[0074] Among them, the present invention also introduces time series analysis. By combining the detection results of multiple frames through the sliding window method, it can effectively judge the persistence and consistency of actions, significantly reducing the impact caused by single-frame detection errors. Moreover, this continuous-frame analysis method improves the classification accuracy of similar gestures and enhances the reliability of action recognition.

[0075] Furthermore, during the hand movement classification process of the present invention, a lightweight convolutional neural network model adopts lightweight strategies such as grouped convolution, structural reparameterization, and dropout. In particular, the adopted structural reparameterization strategy enhances the feature extraction and expression capabilities while ensuring computational efficiency, and is especially suitable for the detailed recognition of hand movements. Moreover, this model design can also improve the generalization ability of the network, accelerate the inference process, enable it to process hand images in large-scale datasets in real time, effectively reduce the computational complexity and memory consumption, and realize the synchronous completion of hand movement detection and classification during the hand hygiene process.

[0076] (2) The identity recognition of the hand hygiene behavior subject based on the video stream needs to address the challenges of the complex actual conditions and environmental factors in the medical environment. On the one hand, the sink is located indoors, and the surrounding area belongs to narrow passages or small spaces. The duration of the gait and posture videos collected is calculated in seconds, and the information is limited. Moreover, the images usually have a certain oblique angle. On the other hand, the wrapped wear of medical staff blurs the body contour, most of the body area is blocked, and different individuals in the same environment have the same height of wear, making it extremely difficult to identify the user's identity. The identity recognition method with a single feature is not applicable.

[0077] Therefore, the present invention constructs an identity recognition network through a multi-dimensional feature fusion technology, comprehensively utilizes three identity features of gait, posture and hand movement, can effectively reduce the error caused by environmental interference, ensure the accuracy of identity recognition in a complex environment, and greatly improve the robustness and recognition accuracy.

[0078] Furthermore, the present invention adopts the GaitSet model backbone including a convolutional neural network part and a multi-layer global pipeline part in the gait feature extraction branch of the identity recognition network to extract two sets of gait features. On the one hand, it can enrich the hierarchy and diversity of feature representation. On the other hand, it can also make the gait feature account for a higher proportion in the entire identity recognition network, improving the influence of the gait feature on the identity recognition result.

[0079] Furthermore, the present invention uses the OpenPose convolutional neural network model backbone including a part affinity field module and a key point heat map module in the posture feature extraction branch and the hand movement feature extraction branch of the identity recognition network, which can effectively capture and understand the spatial and directional relationships between the detected parts of the human body, realize the accurate modeling of the human body posture / hand movement, and the precise positioning of the key points of the human body posture / hand movement, and then obtain the human body posture / hand movement features.

[0080] Furthermore, the present invention adopts a horizontal pyramid mapping in the identity feature fusion and encoding module of the identity recognition network to further optimize the feature expression, map the fusion feature to a low-dimensional space, and the formed individual feature encoding is compact and has the ability to identify.

[0081] Furthermore, the encoding and comparison module in the identity recognition network uses cosine similarity for feature comparison. Cosine similarity can measure the direction consistency of two feature vectors in a high-dimensional space, can avoid errors caused by differences in the size of feature vectors, and enable the network to focus on optimizing the semantic matching of features rather than just the numerical proximity, thereby improving the accuracy of comparison.

[0082] (3) The present invention can detect hand movements in real time and complete user identity recognition, associating hand hygiene behaviors with their identity information. This can not only accurately track and evaluate the hand hygiene behaviors of each user, screening out medical staff with qualified hand hygiene behaviors, but also provide detailed behavioral data and liability traceability for medical institutions to ensure the implementation of hand hygiene compliance. Moreover, the combination based on identity information can identify the behavioral patterns of specific individuals, timely detect problems and conduct targeted interventions, thereby improving the hand hygiene compliance rate, optimizing the behavioral management of medical institutions, and providing reliable technical support for public health management. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 is a schematic flow chart of the method for identifying the main body of qualified hand hygiene behavior based on video stream of the present invention;

[0084] Figure 2 is a schematic framework diagram of the lightweight convolutional neural network in the embodiment;

[0085] Figure 3 is a schematic structural diagram of the first intermediate module in the embodiment;

[0086] Figure 4 is a schematic structural diagram of the second intermediate module in the embodiment;

[0087] Figure 5 is a schematic diagram of the actions of hand hygiene behavior in the embodiment;

[0088] Figure 6 is a schematic structural diagram of the feature extraction network of the GaitSet model in the embodiment;

[0089] Figure 7 is a schematic structural diagram of the feature extraction network of the OpenPose model in the embodiment;

[0090] Figure 8 is a schematic structural diagram of the feature fusion and encoding module in the embodiment;

[0091] Figure 9 is a schematic flow chart of the identity recognition algorithm in the embodiment;

[0092] Figure 10 is a schematic structural diagram of the specific implementation network of the method for identifying the main body of qualified hand hygiene behavior based on video stream in the embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0093] The technical solutions of the present invention will be described in detail below in conjunction with the accompanying drawings of the specification and specific embodiments.

[0094] The explanations of the relevant technical terms involved in the present invention are as follows:

[0095] Camera: A device used to capture, record, and process visual images, typically consisting of an image sensor, a lens assembly, an image processing unit, a data storage and transmission interface, and a power management module.

[0096] Hand hygiene: The general term for handwashing, antiseptic handrub, and surgical hand antisepsis performed by medical staff during their professional activities.

[0097] Hand hygiene behavior: Hand hygiene behavior specifically includes the following 16 hand movements: 1. Wet the hands and wrists under running water; 2. Take soap solution or hand sanitizer; 3. Rub the palms together with fingers interlaced; 4. Rub the back of the right hand with the palm of the left hand along the finger seams; 5. Rub the back of the left hand with the palm of the right hand along the finger seams; 6. Rub the palms together with fingers crossed; 7. Rub the finger joints with the knuckles of both hands overlapping; 8. Rotate and rub the bent finger joints of the left hand in the palm of the right hand; 9. Rotate and rub the bent finger joints of the right hand in the palm of the left hand; 10. Rotate and rub the left thumb in the palm of the right hand; 11. Rotate and rub the right thumb in the palm of the left hand; 12. Rotate and rub the fingertips of the left hand in the palm of the right hand with fingers together; 13. Rotate and rub the fingertips of the right hand in the palm of the left hand with fingers together; 14. Rub the wrist of the right hand with the left hand; 15. Rub the wrist of the left hand with the right hand; 16. Rinse the soap solution or hand sanitizer foam from the fingertips towards the wrists.

[0098] Feature map: A multi-dimensional structure representing the activation or response of data in different dimensions in the feature space. The feature map maps the original data to a high-dimensional feature space through various feature extraction methods to capture and describe the inherent features and patterns of the data. The multi-dimensional representation form of the feature map can express multiple feature information simultaneously, retain the spatial or structural information of the data, and achieve hierarchical feature abstraction in a multi-layer model.

[0099] Feature vector: An ordered vector representing the numerical description of an object or data instance in each feature dimension in the feature space.

[0100] An embodiment of the present invention proposes a method for identifying the subject of qualified hand hygiene behavior based on video streams, aiming to effectively identify the implementation of hand hygiene and associate it with the identity information of the actor. Through a multi-stage processing flow combining hand position detection, optical flow tracking, and fine-grained feature extraction, real-time and accurate hand hygiene monitoring can be achieved in a complex medical environment. At the same time, the identity information of the actor is identified based on gait, posture, and handwashing actions. The present invention corresponds the hand hygiene monitoring data with the identity information, which is helpful for the daily behavior management of medical institutions and provides important technical support for improving the hand hygiene compliance rate and public health management.

[0101] As Figure 1 shown, the method for identifying the subject of qualified hand hygiene behavior based on video streams proposed by the embodiment of the present invention mainly includes the following steps:

[0102] S100, feature image frame acquisition: a camera installed in front of the wash basin is used to capture the user's gait when approaching the wash basin and the user's posture during hand washing to obtain video stream 1; a camera installed directly above the wash basin is used to capture the user's hand movements during hand washing to obtain video stream 2. Different video segments or image frame sequences of video stream 1 and video stream 2 are intercepted to form corresponding feature image frame sequences.

[0103] The hand hygiene process starts when the image of video stream 2 starts to change, and the hand hygiene process ends when the image of video stream 2 remains unchanged for P seconds (P>0). Specifically, the method for extracting different video segments or image frame sequences of video stream 1 and video stream 2 is as follows: intercept the video segment Q seconds (Q≥5) before the start of the hand hygiene process in video stream 1 as the gait feature video segment, intercept the video of the time period from the start to the end of the hand hygiene process in video stream 1 as the posture feature video segment, and intercept the video of the time period from the start to the end of the hand hygiene process in video stream 2 as the hand movement feature video segment.

[0104] Output each frame of image (i.e., hand motion feature image frame) according to the hand motion feature video segment to form a hand motion feature image frame sequence {F 1 , F 2 ,......,F i ,......,F m}, the image frame sequence is sent to the subsequent action detection process for processing and used for hand action recognition. Each frame of the gait feature video segment, posture feature video segment and hand action feature video segment continuously output (gait feature image frame, posture feature image frame and hand action feature image frame) forms a gait feature image frame sequence, a posture feature image frame sequence and a hand action feature image frame sequence, and then the image frame sequences are respectively input into the subsequent identity recognition process for processing and used for identity recognition.

[0105] Preferably, image frames in each video stream are extracted according to the frame interval L to obtain key image frames, and the key image frames are arranged in chronological order to form a key image frame sequence. The key image frame sequence is used to replace the continuous image frame sequence frame by frame as the gait feature image frame sequence, the posture feature image frame sequence and the hand movement feature image frame sequence for subsequent hand hygiene behavior and identity recognition analysis. Preferably, L is 3. The key image frame sequence is used to replace the continuous image frame sequence frame by frame to further improve the real-time performance of hand movement detection and identity recognition.

[0106] Subsequent hand movement recognition is a process of extracting and analyzing these features through a sequence of hand movement images during the handwashing process, combined with object detection and optical flow tracking technologies, to accurately identify and confirm the user's handwashing behavior actions in real time. Identity recognition is a system based on multi-dimensional feature fusion technology. Through the extraction and integration of three identity features: gait, posture, and hand movement, combined with a feature extraction branch, a feature fusion module, and an encoding module, it realizes high-robustness and high-precision individual identity recognition in complex environments.

[0107] S200, Hand position detection: During hand hygiene, the hand movement feature image frames obtained by S100 are analyzed in real time for hand position detection.

[0108] Among them, the hand position refers to the hands and forearms of the actor in the hand movement feature image frame. Before hand position detection, an initial object detection model should first be constructed and trained for hand shape detection. After the model training is completed, it can be used to identify the hand position in each image frame. The object detection model can be constructed based on models such as YOLO and SSD. Preferably, the YOLO model is used as the object detection model.

[0109] The object detection model is used to preliminarily detect the hand position in each frame or key frame. The specific method is: input the hand movement feature image frame sequence into the object detection model, and through forward propagation, output the detection box (x, y, w, h) of the hand area in each image frame, where x and y are the coordinates of the center point of the detection box, and w and h are the width and height of the detection box.

[0110] S300, Hand position tracking: According to the detection box of the hand obtained by S200, combined with the optical flow method, the change of the hand position is tracked in real time, and the hand position in the missed detection frames where the hand is not detected is characterized by a tracking box.

[0111] In the hand position detection stage of S200, for a continuous sequence of image frames {F 1 , F 2 ,......, F i ,......, F m}, there are usually some image frames in which the hand information cannot be detected, which are called suspected missed detection frames. In this embodiment, the frame number of the suspected missed detection frame is set as i, and F i is the suspected missed detection frame. There are two main reasons for missed detection. One is that due to complex situations such as light changes, blurred hand boundaries, or occlusion, the object detection model fails to detect. Another situation is that the hand leaves the detection range, resulting in the hand position not being detected. The frames F i where the object detection model may miss detecting the hand are tracked to achieve continuous detection of the hand position and movement.

[0112] Among them, the specific method for hand position tracking includes the following sub-steps:

[0113] S301, calculate the suspected undetected frame F i The temporal gradient and spatial gradient of the images of the previous K frames (K≥1) are calculated, and the average motion vector of the detection box (x i-1 , y i-1 , w i-1 , h i-1 ) of the hand is estimated by the optical flow method to generate the optical flow tracking position (x i ) of the center point of the detection box of the suspected undetected frame F i , y i ), and the tracking box (x i , y i , w i , h i ) for characterizing the hand position in the suspected undetected frame is obtained.

[0114] S302, if the optical flow tracking position (x i , y i ) falls within the image frame size range, it is determined that the current suspected undetected frame is an undetected frame, then the values of w i and h i in the tracking box (x i , y i ) of the undetected frame are w i and h i , and the tracking box (x i-1 , y i-1 , w i , h i , w i-1 , h i-1 ) is obtained, and the subsequent processing is continued according to the tracking box of the undetected frame. Conversely, if the optical flow tracking position falls outside the image frame size range, it is determined that the hand has left the detection range and no detection is made, and no tracking box is calculated and no subsequent processing is performed.

[0115] Among them, the detailed process of the optical flow tracking algorithm in sub-step S301 is as follows:

[0116] S3011, perform grayscale conversion and Gaussian blur processing on the previous K frames of images {F i-K , F i-K+1 ,......, F i-1} and the current frame image F i to reduce the influence of noise.

[0117] S3012, calculate the spatial gradient of two adjacent frames of images F s-1 and F s (s = i - K + 1, i - K + 2, ……, i) to identify the edges and hand contours in the image, and the calculation method is:

[0118] E s，x = F s-1 * G x ,

[0119] E s，y = F s-1 * G y ,

[0120] In the formula, E s，x and E s,y are the spatial gradients in the x and y directions respectively, G x and G y are the convolution kernels in the x and y directions respectively, and * is the convolution calculation.

[0121] Optionally,

[0122] S3013, calculate the temporal gradient between the current frame and the previous frame to represent the intensity change of the hand between consecutive frames. The calculation method is:

[0123] E s，t = F s-1 - F s ,

[0124] In the formula, E s，t is the temporal gradient between the previous frame and the current frame image.

[0125] Assume that within a small neighborhood, the optical flow of all points is the same. According to the optical flow constraint equation, an optical flow constraint equation is established for each pair of adjacent frames:

[0126] E s,x u + E s,y v + E s,t = 0,

[0127] In the formula, u and v are the optical flow components of each pixel in the x and y directions of the image respectively.

[0128] S3014, stack the optical flow constraint equations of the first K pairs of adjacent frames to form an overdetermined linear equation system:

[0129]

[0130] In the formula,

[0131] S3015, use the weighted least squares method to solve the optimal optical flow vector (u, v) of the overdetermined equation system. The calculation method is:

[0132]

[0133] Wherein, W is a diagonal weight matrix used to assign different weights to the gradients between different frames.

[0134] Preferably, W = diag(1, 2, …, K), where diag represents a diagonal matrix.

[0135] S3016. Extract the optical flow vectors of all pixels within the detection box, and calculate the average motion speed of the pixels within the detection box:

[0136]

[0137] In the formula, k is the index of the pixels within the detection box, u avg and v avg are respectively the average motion speeds of all pixels within the detection box in the x and y directions, M is the number of pixels within the detection box, u k and v k are the motion speeds of each pixel index k.

[0138] S3017. Update the hand position of the current frame using the calculated average motion vector:

[0139] x i = x i-1 + u avg ,

[0140] y i = y i-1 + v avg ,

[0141] In the formula, x i-1 and y i-1 are respectively the central point coordinates of the hand position in the previous frame.

[0142] Furthermore, the hand position tracking box for the missed detection frame F i without a generated detection box is (x i , y i , w i-1 , h i-1 ), where w i-1 and h i-1 are respectively the width and height of the hand position detection box in the previous frame.

[0143] It should be noted that the present invention uses the optical flow method to track the hand position, which can ensure accurate tracking of the hand position in the missed detection frame F i , not only improving the stability of tracking, but also effectively handling temporary occlusion and detection failures. In addition, the optical flow method has the advantages of real-time and high computational efficiency, can adapt to diverse hand movements, reduce the dependence on single-frame detection algorithms, and ensure efficient and reliable hand tracking in complex dynamic environments, greatly improving the performance of this method in various application scenarios.

[0144] S400, Region of Interest (ROI) extraction: Based on the detection boxes obtained from S200 and the tracking boxes obtained from S300, the hand ROI in each image frame is extracted, and after sorting, a sequence of hand image frames is obtained.

[0145] Specifically, for the image frames in which S200 detects the hand region, the ROI is the detection box region generated by the object detection model, i.e., the detection box; for the image frames in which S200 does not detect the hand region, the position of the hand is tracked by the optical flow method of S300. If the position tracked by the optical flow is within the image region, the tracking box is regarded as the ROI, and then the image frame is cropped according to the ROI to obtain a hand image, focusing on the local region related to the handwashing action; the hand images are arranged in chronological order to obtain a sequence of hand image frames.

[0146] It should be noted that the main function of these ROIs is to ensure that subsequent analysis is only targeted at relevant regions, reducing background interference. In consecutive image frames, the ROIs are dynamically updated to ensure continuous tracking of the hand position. Through the tracking of the hand position by the optical flow method in the present invention, the system can adjust the ROIs in real time and maintain continuous attention to hand hygiene behavior.

[0147] The present invention extracts the region of interest, i.e., the hand region, in the image frame through the ROI extraction technology. In the subsequent process of hand action classification and user identity information recognition, due to reducing background interference, the image processing efficiency and the accuracy of action detection can be effectively improved.

[0148] S500, Hand action classification: Deeply analyze the hand images extracted by S400 through a fine-grained feature extraction model, identify the specific details of the hand actions, obtain the hand action being performed in the current image frame, and obtain the action classification result A detected and the corresponding action classification confidence level wei 1 (0 < wei 1 < 1), arrange the action classification results corresponding to each image frame in chronological order to obtain an action classification result sequence.

[0149] It can be understood that before extracting features from the hand images, since the size of the object detection box is different for each frame, it is usually necessary to adjust the standardized size of the hand images so that the image sizes are consistent at the model input stage to obtain images suitable for input into the model.

[0150] In the embodiment, the fine-grained feature extraction model specifically selects a lightweight convolutional neural network and uses lightweight strategies such as grouped convolution, structural reparameterization, and dropout. Its structure is as Figure 2As shown in the figure, it specifically includes the following: In the input stage of the fine-grained feature extraction model, 3×3 convolution is used for downsampling to initially extract the spatial features of the input image and obtain the input feature map; the backbone of the fine-grained feature extraction model consists of a first intermediate module and a second intermediate module to extract the fine-grained features of the hand action; in the output stage, 1×1 convolution layer is used to enhance the information interaction between channels, and then it sequentially passes through the global average pooling layer and two fully connected layers, and finally outputs the action classification result A detected and the action classification confidence wei 1 . Among them, the backbone is composed of N 1 first intermediate modules and N 2 second intermediate modules connected in sequence, N 1 , N 2 ≥1. The two intermediate modules adopt the structural reparameterization strategy, which can effectively reduce the computational complexity in the inference stage.

[0151] As Figure 3 shown, in the model training stage, the first intermediate module uses 3×3 convolution to increase the dimension of the input feature map, and then passes through the 1×1 grouped convolution branch, 3×3 grouped convolution branch and the branch only for batch normalization respectively, and then sums the outputs of the three branches, reduces the dimension through 1×1 convolution, and then performs regularization processing through the dropout layer, and then makes a residual connection with the input feature map to obtain the first output feature map. As Figure 4 shown, the second intermediate module first makes the input feature map pass through the 1×1 grouped convolution branch, 3×3 grouped convolution branch and the branch only for batch normalization, then sums the outputs of the three branches, then introduces the squeeze and excitation (SE) attention module to obtain the intermediate feature map, passes the intermediate feature map through the 1×1 convolution module for dimension increase and dimension reduction in sequence, and then performs regularization processing through the dropout layer, and finally makes a residual connection with the intermediate feature map to obtain the second output feature map.

[0152] As Figure 3 and Figure 4 shown, the inference stage (i.e., during application) is different from the model training stage. In order to improve the hand action recognition efficiency, the present invention specifically uses the structural reparameterization strategy to combine the 1×1 grouped convolution branch, 3×3 grouped convolution branch and the branch only for batch normalization in the first intermediate module and the second intermediate module into a single 3×3 grouped convolution channel. For the g-th feature group I g of the grouped convolution, the structural reparameterization process can be expressed as:

[0153]

[0154] In the formula, G o g , B og are the convolution kernel and bias after structural reparameterization, G 3×3 g is the convolution kernel of the 3×3 convolution branch before structural reparameterization, G 1×1，padding g is the convolution kernel of the 1×1 convolution branch after padding before structural reparameterization, G BN，structured g is the normalized convolution kernel constructed from the branch that only performs batch normalization before structural reparameterization, B 3×3 g , B 1×1 g , B BN g are the biases of the three branches before structural reparameterization respectively.

[0155] It should be noted that structural reparameterization plays a role in optimizing the network structure and improving the inference efficiency in this model. By adopting a complex multi-branch structure in the training stage to enhance the feature extraction ability, the structural reparameterization strategy can, when the model is converted to the inference stage, equivalent these multi-branches to a single-branch convolution operation. This process effectively reduces the computational amount and memory occupation in the inference stage, while maintaining the feature expression ability learned by the model in the training stage. In other words, structural reparameterization ensures that no additional computational burden is added during the inference process of the model, thus making the recognition more real-time and efficient, and meeting the purpose of synchronously completing hand action detection and classification during the hand hygiene process.

[0156] The hand image is input into the fine-grained feature extraction model, and the action classification result A detected and the action classification confidence wei 1 are output. The classification results include 16 hand action classifications in the term explanation part, as can be seen in Figure 5 shown. The lightweight convolutional neural network used in the present invention, combined with the structural reparameterization process, while ensuring the computational efficiency, enhances the feature extraction and expression ability, and is particularly suitable for the detailed recognition of hand actions. The model design based on this invention not only improves the generalization ability of the network, but also accelerates the inference process, enabling it to process hand images in large-scale datasets in real time, effectively reducing the computational complexity and memory consumption, avoiding the overfitting problem, and greatly improving the performance of the model in the fine-grained image classification task. Especially in devices with limited computational resources or application scenarios that require fast convergence, this model performs particularly well, ensuring the high accuracy and efficiency of hand action detection, and greatly enhancing the overall reliability and practicality of hand hygiene behavior evaluation.

[0157] S600, time series analysis operation: Determine the effectiveness of the hand action classification results in the action classification result sequence through the sliding window method, and obtain the time series analysis result Asequenced , output the final action classification sequence after arranging them in chronological order.

[0158] The specific method for judging the effectiveness of actions by the sliding window method is as follows: After the hand action classification is completed, define a time window, which usually includes consecutive Z (Z>2) image frames. Within the time window, count the action classification detection results of each image frame, calculate the number of occurrences of each hand action within this window, and record the action A with the highest number of occurrences j , where j is the index of the hand action with the highest number of occurrences within the window. By accumulating the detection results, the action A j The average confidence level wei avg，j :

[0159]

[0160] In the formula, Z is the number of frames within the window, wei l is the confidence level of the classified action obtained from the l-th frame, count(A j ) is the number of occurrences of action A within the time window j , δ jl is the Kronecker function, A l is the hand action corresponding to the l-th frame.

[0161] If the average confidence level wei j of action A avg，j is higher than the set threshold, and the cumulative number of occurrences within the window exceeds the preset number, then record action A j and the average confidence level wei avg，j as the time series analysis result A sequenced and the time series analysis confidence level wei 2 (0 < wei 2 ≤1). Conversely, if the average confidence level of action A j is lower than the set threshold, or the cumulative number of occurrences within the window is lower than the preset number, then record the hand action classification result A detected and the corresponding confidence level wei 1 as the time series analysis result A sequenced and the time series analysis confidence level wei 2 . Arrange the time series analysis results A sequenced of each image frame in the hand hygiene process in chronological order, and output the final action classification sequence. Through this step, the failure of overall recognition caused by single-frame errors can be avoided, thereby improving the recognition accuracy and the stability of hand action detection

[0162] S700, Time Statistics: After the hand hygiene process ends, record the total duration of the hand hygiene process and the duration of each hand action.

[0163] Statistically calculate the total duration of the hand hygiene process, that is, calculate the duration of the video segment of the hand movement features extracted in S100, and thus obtain the total duration of the hand hygiene process.

[0164] Statistically calculate the duration of each hand movement in the final action classification sequence to obtain the duration of each hand movement. The statistical method for the duration of each hand movement is as follows: According to the frame capture frame rate fps of the image frames, calculate the time t represented by each image frame fra =(1 / fps), and statistically calculate the number of occurrences count(A n ) of each action image frame in the final action classification sequence, where n represents the hand movement index, and then calculate the duration of each hand movement as t A，n =t fra ·count(A n ).

[0165] S800, user identity recognition: Input the gait feature image frame sequence, posture feature image frame sequence obtained by S100, and the hand movement feature image frame sequence formed by S400 into the identity recognition network for user identity recognition to obtain user identity information.

[0166] In practical applications, identity recognition needs to cope with various complex actual conditions and environmental factors, which may seriously affect the accuracy of traditional identity verification methods. Especially in narrow spaces, the installation position and viewing angle of the camera are usually restricted, resulting in problems such as image tilt, blur, and light changes. Moreover, the unified protective equipment (such as masks, gloves, goggles, etc.) worn by medical staff often obscures facial features, making face-based identity recognition difficult.

[0167] The input of gait recognition is a sequence of video images of walking, and the main features extracted are the movements of each joint of the human body. Therefore, its data collection is non-invasive and acceptable. However, due to the large amount of data in the sequence images, the computational complexity is relatively high and it is also difficult to process. Although a large amount of research work has been done on gait in biomechanics, gait-based identity discrimination is still not mature. In addition, in the medical environment applicable to the hand hygiene detection process, there are often many people with similar dressing styles appearing at the same time, and factors such as the dynamic behavior of medical staff, clothing occlusion, and environmental interference may also cause detection errors in features such as posture and gait. Specifically for the collection during the handwashing process, the walking gait video segment is only a few seconds, and the information volume is limited.

[0168] In view of the above practical difficulties, the identification method based on any single feature of gait, posture, or hand movement cannot meet the actual needs. To meet this challenge, the present invention uses multi-dimensional feature fusion technology to comprehensively utilize the three identity features of gait, posture and hand movement to construct a more robust and efficient identity recognition network, which can effectively reduce the errors caused by environmental interference and ensure that the accuracy of identity recognition can be maintained in complex environments.

[0169] The identity recognition network provided by the present invention is a multi-dimensional feature fusion network constructed based on gait, posture and hand movement features. The features of gait, posture and hand movement are extracted respectively through three different branches through feature extraction branches of different frameworks, and then these features are fused, and the feature expression is further optimized by using deep separable convolution and fully connected layers to obtain feature coding representing individual identity.

[0170] At the same time, by collecting individual gait, posture and hand movement videos during registration, template feature coding is formed as the basis for recognition and comparison. The limited nature of samples is used to further improve the accuracy of identity authentication among a limited population.

[0171] It should be noted that there is no strict time sequence between this step and the aforementioned hand motion detection steps S500 to S600 and the time statistics step S700, and they can usually be performed simultaneously.

[0172] like Figure 9 As shown in the figure, the identity recognition network mainly includes feature extraction module, feature fusion encoding module and encoding comparison module. The feature extraction module includes three branches: gait feature extraction branch, posture feature extraction branch and hand movement feature extraction branch. Each branch extracts the features of gait, posture and hand movement respectively through different frameworks (GaitSet and OpenPose).

[0173] Among them, the gait feature extraction branch uses the GaitSet model trunk, and its structure is as follows Figure 6 As shown in the figure, it mainly includes a convolutional neural network part and a multi-layer global pipeline part. Before inputting the model, the gait feature image frame sequence in video stream one is first preprocessed, including alignment, normalization and binarization, to obtain an image that meets the network input requirements. The preprocessed image is input into the convolutional neural network, and high-level frame-level features are extracted through multi-layer convolution and nonlinear activation functions to generate feature maps with channel, height and width dimensions; then, these frame-level features are aggregated into gait set-level features using the Set Pooling (SP) operation to achieve invariance to the input order and adaptability to different numbers of inputs. The convolutional neural network part generates gait features. Figure 1Meanwhile, the Multilayer GlobalPipeline (MGP) part extracts and fuses set-level features under different receptive fields at multiple convolutional levels, further enriching the hierarchy and diversity of feature representations, and the MGP part generates gait features. Figure 2 It should be noted that through the above design, the gait feature extraction branch outputs two feature maps, enabling gait features to account for a higher proportion in the entire identity recognition network, so as to enhance the impact of gait features on the identity recognition result.

[0174] Among them, the pose feature extraction branch uses the backbone of the OpenPose convolutional neural network model to extract the pose features of each image frame, and its structure is as Figure 7 shown, mainly including the VGG-19 network, the Part Affinity Fields (PAFs) module, and the Confidence Maps module. Among them, the main function of the PAFs module is to capture and understand the spatial and directional relationships between various detected parts of the human body, so as to accurately model the human pose; the main task of the Confidence Maps module is to locate the human key points in the input feature image. First, the pose feature image in Video Stream 1 is adjusted to a predetermined size, and then input into the convolutional neural network for feature extraction. Specifically, the model uses the first ten layers of the pre-trained VGG-19 network as the initial feature extractor to generate high-dimensional feature maps; the generated feature maps are then input into the PAFs module, where the model parameters are optimized through multiple iterations to improve the prediction accuracy of the part affinity fields; subsequently, the intermediate feature maps generated by the PAFs module are input into the Confidence Maps module, and the model parameters are optimized through multiple iterations to accurately locate the human key points therein.

[0175] In multiple iterative stages of the entire network, it is responsible for gradually refining the prediction results of the PAFs and the Confidence Maps. In each iterative stage, the prediction result of the current stage is fused with the initial feature map and processed through consecutive convolutional modules (3×3 convolutional layers). The outputs of each convolutional module are concatenated to enhance the network's ability to capture multi-scale features.

[0176] In addition, at the end of each iterative stage, loss functions are applied to the PAFs and the Confidence Maps respectively. Specifically, the L2 norm loss function is used to evaluate the iterative effects of the PAFs and the Confidence Maps to ensure the accuracy of the inter-part association and the precision of the key point localization. Among them, the loss function f of the PAFs module L is expressed as:

[0177]

[0178] Wherein, C is the number of Affinity Fields, c is the index number of the Affinity Field, p is each pixel position in the image, W(p) is the spatial weighting mask function. If there is no annotation at pixel p, then W(p) = 0; otherwise, W(p) = 1, and L c (p) is the predicted value of the c-th Affinity Field at pixel p, is the ground truth value of the c-th Affinity Field at pixel p, and ||·|| 2 represents the L2 norm.

[0179] Among them, the loss function f of the key point heat map module s is expressed as:

[0180]

[0181] Wherein, D is the number of key points, d is the index number of the key point, S d (p) is the predicted value of the d-th key point at pixel p, is the ground truth value of the d-th key point at pixel p.

[0182] Finally, after being processed through all iterative stages, a feature map representing the positions and correlations of two-dimensional pose key points for each frame is generated. These frame-level features are aggregated into set-level features using the set pooling operation, and the output is the pose feature map.

[0183] Among them, the hand action feature extraction branch also uses the backbone of the OpenPose convolutional neural network model to extract the hand action features of each image frame. The specific network structure and characteristics will not be elaborated here. It should be noted that before extracting the hand action features based on the hand images extracted by S400, the standardized size adjustment of the hand images should be carried out first. By using the method of padding with all zeros for the redundant parts, the ratio of the hand images is ensured to remain unchanged, and an image suitable for inputting into the model is obtained. Subsequently, the adjusted images are input into the hand action feature extraction branch. After being processed through the OpenPose iterative stage for each image, a feature map representing the positions and correlations of two-dimensional hand action key points for each frame is generated. These frame-level features are aggregated into hand action set-level features using the set pooling operation to output the hand action feature map.

[0184] Identity feature fusion and encoding module: Fuse the extracted gait features Figure 1 and gait features Figure 2 , the pose feature map and the hand action feature map, and further optimize the feature expression using the horizontal pyramid mapping, mapping the fused features to a low-dimensional space to form a compact and discriminative individual feature encoding.

[0185] Specifically, feature fusion is to divide each feature map into several strips through Horizontal Pyramid Mapping (HPM). The feature fusion process is as follows Figure 8 shown. Specifically, for the horizontal pyramid mapping with a scaling factor of S, first, Horizontal Pyramid Pooling (HPP) is performed, that is, on scales s ∈ 1, 2, …, S, the feature map is divided into 2 s-1 strips in the height dimension, that is, a total of strips for each feature map; then, global pooling is applied to the three-dimensional strips to obtain one-dimensional features, and global pooling is the sum of global max pooling and global average pooling; then, since the strips of different scales depict features of different receptive fields, and the different strips of each scale depict features of different spatial positions, independent fully connected layers are used to map the features to the discriminant space; finally, an identity feature vector, that is, an individual feature encoding, is generated through feature flattening and a fully connected layer.

[0186] It should be noted that considering that gait features have a strong correlation with the recognition of user identity information, the identity feature vector obtained by the present invention fuses two gait feature maps, one pose feature map, and one hand feature map, containing comprehensive information on gait, pose, and hand, and the proportion of gait information is greater than that of pose and hand information, indicating that gait is the main detection basis, and pose and hand information play an auxiliary detection role. The fused identity feature vector is used to encode individual identity features, forming a feature fusion encoding, that is, an individual feature encoding.

[0187] Coding comparison module: Match the obtained individual feature encoding with the template feature encoding stored in the database. Specifically, the cosine similarity function can be used to measure the similarity between feature vectors (that is, between the fused individual feature encoding and the template feature encoding in the feature encoding template library). The template identity corresponding to the template feature encoding with the highest cosine similarity to the individual feature encoding is output as the recognized identity, thereby determining the identity information of the action execution individual.

[0188] It should be noted that cosine similarity can measure the direction consistency of two feature vectors in a high-dimensional space, and can avoid errors caused by differences in the magnitudes of feature vectors, so that the network focuses on optimizing the semantic matching of features rather than just the numerical proximity. The calculation formula of cosine similarity is:

[0189]

[0190] In the formula, and represent the individual feature encoding and the template feature encoding respectively.

[0191] It is understandable that during the model training process, the model parameters of the gait, posture, and hand movement feature extraction branches are frozen to maintain the feature extraction capabilities of the corresponding features of each branch, and only the model parameters in the feature fusion and encoding stage are trained. The training process combines the Triplet Loss function to ensure that samples of the same identity in the feature space have a high degree of similarity, while samples of different identities have a low degree of similarity. By optimizing the value of the Triplet Loss function, the network can learn the associations between gait, posture, and hand movement features and gradually adjust the model parameters to improve the accuracy and robustness of feature extraction. The Triplet Loss function f TripletLoss is calculated as follows:

[0192]

[0193] where, is the feature encoding of the anchor sample, is the feature encoding of the positive sample belonging to the same identity as the anchor, is the feature encoding of the negative sample belonging to a different identity from the anchor, γ is the margin value used to ensure there is a sufficient gap between positive and negative samples, and max represents taking the maximum value.

[0194] It is understandable that the template feature encoding is formed during the user registration stage. During registration, the system collects video data of the user walking through the passage to the designated sink and completing the handwashing process via the camera installed in front of the sink, and obtains individual series of gait and posture information from it; through the camera installed above the sink, it collects video data of the user's handwashing process and obtains individual handwashing process information from it. The above process information undergoes forward propagation through the trained identity recognition network to obtain the template feature encoding of each user. The template feature encoding is stored in the database to form a feature encoding template library, which is used as the feature data uniquely identifying their identity during the identity verification process.

[0195] It is worth noting that these template feature encodings can be obtained by fusing the measurement results of the same user multiple times, that is, by collecting gait, posture, and handwashing action data of the same individual multiple times, repeating sampling at different times, locations, and environments, and performing feature fusion on the multiple collection results through weighted averaging, so as to achieve the purpose of obtaining a diverse and stable set of gait features. The template feature encoding is a feature vector with a unique value in the database. This vector contains the numerical vectors of all identity features extracted from the user and the relationships between each identity feature, and can distinguish different users.

[0196] S900, comprehensive evaluation and association with identity information: comprehensively evaluate the hand hygiene process based on the final action classification sequence, the total duration of the hand hygiene process and the duration of each hand hygiene process, and then associate the comprehensive evaluation data with the identified user identity information to screen out user information with qualified hand hygiene behavior.

[0197] After the hand hygiene process is completed, the hand hygiene process is scored through a pre-set comprehensive scoring system for the hand hygiene process, combined with the final action classification sequence output by S600, and the total duration of the hand hygiene process and the duration of each hand hygiene process obtained by S700. Then, combined with the user identification information obtained by S800, the comprehensive scoring data of the hand hygiene of the corresponding user is obtained, which can be counted as a hand hygiene data and updated to the database in time. At the same time, the user information with qualified hand hygiene behavior can be screened out through threshold comparison.

[0198] Furthermore, the present invention can also simultaneously record the score of each action in the hand hygiene data of this time, compare the comprehensive score with the preset threshold, and determine the user whose hand hygiene behavior is unqualified if it is lower than the specific threshold, and remind the specific user to perform hand hygiene again through the electronic screen and the sound speaker. If the comprehensive score shows that the time of a specific hand action is low and lower than the threshold corresponding to the action, the specific medical staff will be prompted to focus on training and strengthening during the next hand hygiene process. If the comprehensive score is higher than the passing score and lower than the specified score, an early warning reminder will be fed back when uploading the information. If the number of early warning reminders exceeds a certain threshold, the corresponding user will be prompted to relearn the hand hygiene method. It can be understood that in other embodiments, other evaluation methods besides the comprehensive score can also be used.

[0199] In summary, the method for identifying hand hygiene qualified behavior subjects based on video stream provided by the present invention can Figure 10 To realize the network diagram, deep learning and optical flow tracking technology are combined to ensure accuracy and real-time performance in complex environments. First, the video stream of medical staff's hand hygiene behavior is collected through multi-angle cameras, key frames are extracted to reduce the amount of calculation, and the hand position is detected and tracked in combination with a deep learning model, which effectively avoids the problem of missed target detection; secondly, the fine-grained feature extraction network enhances the feature expression ability and improves the recognition accuracy while ensuring the computational efficiency; in addition, the stability of motion detection is improved through time series analysis, avoiding overall failure due to single-frame misjudgment; finally, identity recognition integrates gait, posture and hand motion features to achieve efficient and accurate identity authentication. This method has the advantages of high efficiency, accuracy, and strong robustness, and is suitable for practical application scenarios such as medical staff hand motion detection and user identity recognition.

[0200] The present invention combines hand movement classification data with user identity information, which can effectively improve hospital management efficiency, individual performance, and postoperative infection control. For example, first, identity recognition can be linked with the operating room access control system to ensure that the identity of each person entering the operating room is clear, preventing unauthorized personnel from entering. At the same time, the handwashing behavior of each medical staff before and after surgery is automatically recorded and associated with their identity information, thus ensuring that medical staff strictly follow hand hygiene norms during each surgery and reducing the risk of cross-infection. Second, the combination of identity recognition and hand hygiene data can be used to improve individual performance. Customized improvement suggestions or training programs can be provided based on the hand hygiene operation of each medical staff to help them improve their handwashing habits, thereby improving the standardization and safety of medical operations. In addition, by associating each handwashing behavior with specific surgical operations and postoperative patient infection data, more accurate comprehensive analysis can be provided for the hospital. This data-driven analysis not only helps to discover potential infection risks but also optimizes the relationship between handwashing operations and operating rooms, thus providing a strong basis for formulating infection control strategies and reducing the incidence of postoperative infections in patients. Generally speaking, the combination of hand movement detection and identity recognition can bring significant benefits in terms of individual behavior supervision, hospital management, and infection control, improving the quality of medical services and patient safety.

[0201] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the program codes in the above-described electronic devices and computer-readable storage media can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0202] Finally, it should be noted that the above has schematically described the present invention and its implementation manners. This description is not restrictive. Without departing from the spirit or basic characteristics of the present invention, the present invention can be implemented in other specific forms. What is shown in the drawings is only one of the implementation manners of the present invention, and the actual structure is not limited thereto. Any reference signs in the claims should not limit the claimed rights. Therefore, if those of ordinary skill in the art are inspired by it and design similar structural manners and embodiments without creative efforts without departing from the purpose of this creation, they should all fall within the protection scope of this application. In addition, the term "including" does not exclude other elements or steps, and the term "a" before an element does not exclude including "a plurality" of such elements. The multiple elements stated in the product claims can also be implemented by one element through software or hardware. The terms such as "first" and "second" are used to indicate names and do not represent any specific order.

[0203] Although the embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields. The above specific embodiments are merely illustrative and guiding, rather than restrictive. Under the inspiration of this specification, those of ordinary skill in the art can also make many forms without departing from the scope protected by the claims of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A method for identifying qualified hand hygiene behavior subjects based on video streams, characterized in that: The following steps are involved: Acquire a video stream that can reflect the user's gait information, hand washing posture information, and hand movement information, and form a gait feature image frame sequence, a posture feature image frame sequence, and a hand movement feature image frame sequence; Input the hand action feature image frame sequence into the pre-built target detection model to detect the hand position frame by frame, and obtain a detection frame for representing the hand position information in the image frame; Based on the detection frame, the optical flow method is combined to track the hand position changes in real time, and the hand position in the missed frames in the hand position detection step is represented by the tracking frame; Based on the detection frame and the tracking frame, the hand ROI in the image frame is extracted to form a hand image, and the hand image is arranged in time sequence to obtain a hand image frame sequence: Inputting the hand image frame sequence into the fine-grained feature extraction model to perform fine-grained feature extraction, obtaining the action classification results and action classification confidence of the hand actions in the image frames, arranging the action classification results in time sequence, and obtaining an action classification result sequence; The validity of the actions in the action classification result sequence is determined by a sliding window method to obtain a time series analysis result, the time series analysis result is arranged in time sequence, and a final action classification sequence is output; After the hand hygiene process is completed, the total duration of the hand hygiene process and the duration of each hand movement are counted; Inputting the gait feature image frame sequence, the posture feature image frame sequence and the hand image frame sequence into an identity recognition network to perform user identity recognition to obtain user identity information; the identity recognition network is a multi-dimensional feature fusion network constructed based on gait, posture and hand movement features; The user's hand hygiene behavior is evaluated based on the final action classification sequence, the total duration of the hand hygiene behavior, and the duration of each hand action. The evaluation result is associated with the user's identity information to screen out the hand hygiene behavior subjects whose evaluation results are qualified.

2. The method for identifying hand hygiene qualified behavior subjects based on video stream according to claim 1, characterized in that: In the hand position detection step, a hand area detection frame (x, y, w, h) is output, where x and y are the coordinates of the center point of the detection frame, and w and h are the width and height of the detection frame; an image frame in which the hand is not detected is a suspected missed detection frame; The steps of tracking hand position changes include: S301, calculating the suspected missed detection frame F in the hand position detection step i The temporal gradient and spatial gradient of the previous K frames, K ≥ 1; the hand detection frame (x i-1 ,y i-1 , w i-1 ,h i-1 ) to generate the suspected missed frame F i The optical flow tracking position of the center point coordinate of the detection box (x i ,y i ), and obtain the tracking frame (x) used to represent the hand position in the suspected missed frame i ,y i , w i ,h i ); S302, if the optical flow tracking position (x i ,y i ) falls within the image frame size range, and the current suspected missed frame F is determined i is a missed frame, then the tracking frame of the missed frame (x i ,y i , w i ,h i ) i and h i The value of w i-1 and h i-1 , get the tracking box (x i ,y i , w i-1 ,h i-1 ); On the contrary, if the optical flow tracking position (x i ,y i ) falls outside the image frame size range, it is judged that the hand has left the detection range and is not detected, so there is no need to calculate the tracking frame.

3. The method for identifying hand hygiene qualified behavior subjects based on video stream according to claim 1, characterized in that: The determining the validity of the actions in the action classification result sequence by the sliding window method specifically includes: Define a time window, including Z consecutive image frames, Z>2; In the time window, the action classification detection results of each image frame are counted, the number of occurrences of each action in the time window is calculated, and the action with the highest number of occurrences is recorded. j ; Action A j The average confidence level wei avg，j The calculation formula is: Where Z is the number of frames in the window, wei1 is the confidence of the classified action obtained in the lth frame, count(A j ) is the action A in the time window j The number of occurrences of jl is the Kronecker function, A l is the hand movement corresponding to the,th frame; If action A j The average confidence level wei avg，j If the value is higher than the set threshold and the cumulative number of occurrences within the time window exceeds the preset number, then action A is recorded separately. j and the average confidence wei avg，j A is the result of time series analysis sequenced and time series analysis confidence level wei2,0 <wei2≤1; On the contrary, if action A j If the average confidence of is lower than the set threshold, or the cumulative number of occurrences in the time window is lower than the preset number, the action classification results A and B are recorded respectively. detected And the corresponding confidence wei1 is the time series analysis result A sequenced and time series analysis confidence level wei2.

4. The method for identifying hand hygiene qualified behavior subjects based on video stream according to claim 1, characterized in that: In the hand movement classification step, the image frames in the hand image frame sequence are standardized and resized before being input into the fine-grained feature extraction model; the fine-grained feature extraction model is a lightweight convolutional neural network, and the lightweight convolutional neural network is lightweight through grouped convolution, structural re-parameterization and random inactivation.

5. The method for identifying hand hygiene qualified behavior subjects based on video stream according to any one of claims 1 to 4, characterized in that: The identity network includes: A feature extraction module is constructed to have a gait feature extraction branch, a posture feature extraction branch, and a hand movement feature extraction branch; The feature fusion coding module is constructed to fuse the gait feature graph, the posture feature graph and the hand movement feature graph output by the feature extraction module to form an individual feature code; The code comparison module is constructed to compare the individual feature code with the template feature code in the pre-constructed feature code template library, and perform user identity recognition based on the comparison result.

6. The method for identifying hand hygiene qualified behavior subjects based on video stream according to claim 5, characterized in that: The gait feature extraction branch uses the GaitSet model trunk to extract features from the image frames in the preprocessed gait feature image frame sequence, and outputs the first gait feature map and the second gait feature map respectively through the convolutional neural network part and the multi-layer global pipeline part contained therein; The posture feature extraction branch uses the OpenPose convolutional neural network model trunk to extract features from image frames in the posture feature image frame sequence and outputs a posture feature map; The hand motion feature extraction branch uses the OpenPose convolutional neural network model backbone to extract features from the hand images in the hand image frame sequence after standardized size adjustment, and outputs a hand motion feature map.

7. The method for identifying hand hygiene qualified behavior subjects based on video stream according to claim 6, characterized in that: The OpenPose convolutional neural network model backbone has a partial affinity field module and a key point heat map module, and a feature map for two-dimensionally representing the position and correlation of key points of posture or hand movements is obtained through iterative optimization of the partial affinity field module and the key point heat map module; In the iterative process, the L2 norm loss function is used to evaluate the iterative effects of the partial affinity field module and the key point heat map module; Among them, the loss function f of some affinity modules L It is expressed as: Where C is the number of affinity fields, c is the affinity field index, p is the position of each pixel in the image, W(p) is the spatial weighted mask function, if there is no label at pixel p, then W(p) = 0, otherwise W(p) = 1, L c (p) is the predicted value of the affinity field of type c at pixel p, is the true value of the affinity field of type c at pixel p, and ||·||2 represents the L2 norm; Among them, the loss function f of the key point heat map S It is expressed as: In the formula, D is the number of key points, d is the key point index number, S d (p) is the predicted value of the key point of category d at pixel p, is the true value of the key point of category d at pixel p.

8. The method for identifying hand hygiene qualified behavior subjects based on video stream according to claim 7, characterized in that: The feature fusion coding module is constructed as follows: the first gait feature map, the second gait feature map, the posture feature map and the hand movement feature map are fused, the feature expression is optimized by using horizontal pyramid mapping, the fused features are mapped to a low-dimensional space, and individual feature coding is formed.

9. The method for identifying hand hygiene qualified behavior subjects based on video stream according to claim 8, characterized in that: During the training process, the feature fusion coding module learns the association between gait, posture and hand movement features and adjusts the model parameters by optimizing the triplet loss function value; the triplet loss function f TripletLoss The calculation formula is: In the formula, is the feature encoding of the anchor sample, Encode the features of positive samples that belong to the same identity as the anchor point, It is the feature encoding of negative samples that belong to different identities from the anchor point, y is the boundary value used to ensure that there is enough interval between positive and negative samples, and max means taking the maximum value.

10. The method for identifying hand hygiene qualified behavior subjects based on video stream according to claim 9, characterized in that: The coding comparison module is constructed to: calculate the cosine similarity between the individual feature code and the template feature code in the pre-built feature code template library, and output the template identity corresponding to the template feature code with the highest similarity as the identification identity; The calculation formula of the cosine similarity is: In the formula, and They represent individual feature coding and template feature coding respectively.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor is used to call a computer program stored in the memory to execute the method for identifying a subject of hand hygiene qualified behavior based on a video stream as described in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the method for identifying hand hygiene qualified behavior subjects based on video streams as described in any one of claims 1 to 10.

Images