Online hand hygiene assessment method and system based on temporal attention

Abstract

The present invention discloses an online hand hygiene assessment method and system based on temporal attention. The method comprises: obtaining an online hand washing video stream, modeling various temporal features respectively through an encoder and a decoder; using temporal attention to enhance and fuse the various temporal features to obtain fused features; performing action classification on the fused features and segmenting the initial features to obtain temporal features of each hand washing action and its corresponding action category as action features; aggregating the action features through a first global maximum pooling layer to obtain a root feature, linearly mapping the root feature to two sub-features through two different first fully connected layers, processing the features of subsequent layers in the same manner, and finally using MLP to regress all features of the last layer to obtain a score for each hand washing action, and adding the scores of the various hand washing actions in the current video stream to obtain a final prediction score. The present invention has the advantage of improving the accuracy of the assessment results.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to an online hand hygiene assessment method and system based on temporal attention. Background Art

[0002] Hand hygiene assessment is an important preventive measure and a crucial task for reducing the potential risk of disease transmission. A standard hand hygiene video contains six handwashing actions, but each action may be interspersed with background motion, which can easily interfere with handwashing assessment performance. Compared to the short-duration diving action, handwashing actions last longer and contain more details, making hand hygiene assessment more challenging.

[0003] Most of the existing hand hygiene assessment methods directly evaluate the extracted video features. Existing technology 1: Zhong et al., Designing a Computer-Vision Application: A Case Study for Hand-HygieneAssessment in an Open-Room Environment. Journal of Imaging, 2021, 7(9): 170. The iterative engineering process is applied to design a hand hygiene behavior detection system. After feature extraction, the video is directly classified, and the results of action classification are used to evaluate the hand washing action. However, it directly classifies the video after feature extraction, and does not model the temporal relationship of the action well. In addition, the use of action classification results to evaluate the hand washing action ignores the inherent complexity of each hand washing action, so that the entire scheme cannot give an accurate evaluation result.

[0004] Prior Art 2: Chinese Patent Publication No. CN115205972A discloses a six-step handwashing assessment method and device based on deep learning. This method designs a key action scorer that focuses on the key action information contained in each handwashing action. The key action scorer consists of a global average pooling layer and two fully connected layers, which are used to extract key actions from action features. The two branches correspond to two different key actions. A learnable Sigmoid layer is also designed to adaptively adjust the score weights of different key actions. However, the key action scorer is only a shallow regression module, making it difficult to fully explore the details of handwashing actions and unable to provide accurate evaluation results. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that the existing hand hygiene assessment method cannot provide accurate assessment results.

[0006] The present invention solves the above technical problems through the following technical means: an online hand hygiene assessment method based on temporal attention, the method comprising:

[0007] Step a: Obtain an online handwashing video stream and model various temporal features through the encoder and decoder.

[0008] Step b: Use temporal attention to enhance and fuse various temporal features to obtain fused features;

[0009] Step c: Classify the fused features and segment the initial features to obtain the temporal features of each hand washing action and its corresponding action category as the action features;

[0010] Step d: The action features are pooled through the first global maximum pooling layer to obtain the root feature, which is linearly mapped to two sub-features by two different first fully connected layers. The processing method of the features of subsequent layers is the same. Finally, all features of the last layer are regressed using MLP to obtain the score of each hand washing action. The scores of each hand washing action in the current video stream are added together to obtain the final prediction score.

[0011] The present invention performs action classification on the fused features and segments the initial features, which can not only perform independent and sufficient fine-grained evaluation on each type of hand-washing action, reducing the mutual interference between unrelated actions, but also remove the background noise between different types of hand-washing actions, thereby improving the accuracy of the evaluation results. The action features are aggregated through the first global maximum pooling layer to obtain the root feature, which is linearly mapped to two sub-features by two different first fully connected layers. The processing methods of the subsequent layers of features are the same, and finally the weighted and summed results are obtained to obtain the predicted score of the hand-washing action, thereby evaluating each hand-washing action from coarse to fine in a tree structure, fully exploring the detailed information of the hand-washing action, and further improving the accuracy of the evaluation results.

[0012] Furthermore, before step a, the method further includes:

[0013] 224×224 visible light video frames and 224×224 optical flow frames are extracted from the online hand washing video stream and then input into the pre-trained I3D network. 1024-dimensional visible light features and optical flow features are obtained through feature extraction. These two modal features are then concatenated in the channel dimension to obtain 2048-dimensional initial features.

[0014] Furthermore, the step a includes:

[0015] Extract the first preset length of features from the initial features as long-term features F long , extract the second preset length feature from the initial feature as the long-term feature F short, the first preset length is greater than the second preset length, and the formulas for modeling various types of time series features through the encoder and decoder are as follows

[0016] {F0,F2}=Encoder(F long )

[0017] F1=Decoder(F short ,F0,F0)

[0018] F3=Decoder(Q,F0,F0)

[0019] Among them, Encoder represents the encoder, F0 represents the long-term historical features, F2 represents the category historical features, F1 represents the short-term current features, Decoder represents the decoder, F3 represents the short-term future features, and Q represents the learnable features.

[0020] Furthermore, the step b includes:

[0021] The short-term current feature F1, the category history feature F2, and the short-term future feature F3 are all input into the attention module of the same structure. The attention mechanism is used to enhance the features in space and channel, and then the transfer connection is performed to achieve progressive interaction. Finally, the three are connected in the channel dimension to obtain the fusion feature F m .

[0022] Furthermore, the step b further includes:

[0023] Construct the following formula to obtain the fusion feature F m

[0024] F i ′ =Attention(F i )

[0025]

[0026]

[0027] Among them, i∈{1,2,3}, Attention represents the attention module, AP is the first average pooling layer, and Concat represents connection.

[0028] Furthermore, the attention module includes a second global pooling layer, a second average pooling layer, a second fully connected layer to a sixth fully connected layer, and a first activation layer to a third activation layer. The second global pooling layer is connected to the first activation layer through the second fully connected layer, the second average pooling layer is connected to the second activation layer through the third fully connected layer, the output results of the first activation layer and the second activation layer are fused and input into the fourth fully connected layer, the output result of the fifth fully connected layer after the third activation layer and the output result of the first activation layer and the second activation layer are fused and input into the sixth fully connected layer, and the output results of the fourth connection layer and the sixth connection layer are respectively fused and output after passing through an activation function.

[0029] Furthermore, the step c includes:

[0030] The fusion feature F m Input into the classification head to identify the action category of each frame of the current video stream, and then use this category information to segment the initial features extracted by the I3D network to obtain the temporal features α of each hand washing action in the current video stream. i , i corresponds to the category of hand washing action, α i As an action feature.

[0031] Furthermore, the step d comprises:

[0032] By formula i =EST(α i ) Get the score of the i-th hand washing action in the current video stream;

[0033] By formula Add up the scores of each hand washing action in the current video stream to obtain the predicted score of the hand washing action in the current video stream;

[0034] Among them, EST represents the integrated scoring tree, which represents the structure of the first global maximum pooling layer, two different first fully connected layers, and subsequent layers in step d, and n represents the number of categories of hand washing actions in the current video stream.

[0035] Furthermore, the method further includes a pre-training process:

[0036] Adjust the parameters of each module involved in the evaluation process, continuously train the entire model until the loss function is minimized, and stop training to obtain the optimal model. Use this optimal model to perform online hand hygiene evaluation. The loss function includes the first loss function Loss1 = CE(p, g) + μCE(p′, g′)

[0037] Where p is the predicted probability of the current action category and p = CF(F m ), p ′is the predicted probability of the future frame and p′=CF′(F3), CE represents the cross entropy loss function, CF and CF ′ Both are classifiers, g and g ′ are the true categories of current and future actions respectively, μ is the balancing factor;

[0038] The loss function also includes a second loss function Loss2 = MSE (S, S')

[0039] Among them, S and S ′ are the predicted score and the true score respectively, and MSE represents the mean square error.

[0040] The present invention also provides an online hand hygiene assessment system based on temporal attention, the system comprising:

[0041] The temporal feature acquisition module is used to obtain the online handwashing video stream and model various temporal features through the encoder and decoder;

[0042] The fusion feature acquisition module is used to use temporal attention to enhance and fuse various temporal features to obtain fusion features;

[0043] The action feature acquisition module is used to classify the fused features and segment the initial features to obtain the temporal features of each hand washing action and its corresponding action category as the action features;

[0044] The evaluation module is used to aggregate the action features through the first global maximum pooling layer to obtain the root feature, which is linearly mapped to two sub-features by two different first fully connected layers. The processing method of the features of subsequent layers is the same. Finally, MLP is used to regress all the features of the last layer to obtain the score of each hand washing action. The scores of each hand washing action in the current video stream are added together to obtain the final prediction score.

[0045] The advantages of the present invention are:

[0046] (1) The present invention classifies the fused features into actions and segments the initial features, which not only allows for independent and sufficient fine-grained evaluation of each type of hand-washing action, reducing mutual interference between unrelated actions, but also removes background noise between different types of hand-washing actions, thereby improving the accuracy of the evaluation results. The action features are aggregated through the first global maximum pooling layer to obtain a root feature, which is linearly mapped to two sub-features by two different first fully connected layers. The processing methods of the subsequent layers of features are the same, and finally the weighted and summed scores are obtained to obtain the predicted score of the hand-washing action, thereby evaluating each hand-washing action from coarse to fine in a tree structure, fully exploring the detailed information of the hand-washing action, and further improving the accuracy of the evaluation results.

[0047] (2) This paper designs a temporal Transformer (including an encoder, two decoders, and a temporal attention), which fully models long-term historical information, short-term current information, and short-term future information, and uses the attention mechanism to enhance individual features in space and channels, and achieves progressive fusion through transitive connections.

[0048] (3) The regression capability of existing evaluation heads is relatively poor, which easily overlooks the inherent complexity of each hand-washing action and makes it difficult to mine the detailed information of the action. In order to achieve fine-grained evaluation of hand-washing actions, this paper designs an integrated scoring tree (composed of pooling layers, fully connected layers, and multi-layer perceptrons (MLPs)) to capture the fine-grained details of hand-washing actions. Each hand-washing action is evaluated from coarse to fine in a tree structure, fully mining the detailed information of the hand-washing actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a diagram showing the overall architecture of the online hand hygiene assessment method based on temporal attention disclosed in an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the temporal attention part of the online hand hygiene assessment method based on temporal attention disclosed in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of an integrated scoring tree in the online hand hygiene assessment method based on temporal attention disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0053] Example 1

[0054] like Figure 1 As shown, an online hand hygiene assessment method based on temporal attention comprises:

[0055] Step 1: Extract visible light features and optical flow features from the input video stream. The specific process is as follows:

[0056] 224×224 visible light video frames and 224×224 optical flow frames are extracted from the online hand washing video stream and then input into the pre-trained I3D network. 1024-dimensional visible light features and optical flow features are obtained through feature extraction. These two modal features are then concatenated in the channel dimension to obtain 2048-dimensional initial features.

[0057] Step 2: Obtain an online handwashing video stream and model various temporal features using the encoder and decoder. The specific process is as follows:

[0058] Extract the first preset length of features from the initial features as long-term features F long , extract the second preset length feature from the initial feature as the long-term feature F short , the first preset length is greater than the second preset length. Since the long-term historical information covers rich action features and timing information, the encoder is used to model the long-term features of the input F long , obtain the long-term historical features F0 and the category historical features F2. In addition, the short-term current information includes the current frame and some adjacent recent frames, which are crucial for the current action detection and subsequent evaluation, so the short-term features F short As a query, the long-term historical feature F0 is input into the decoder as a key and value to obtain the short-term current feature F1. Regarding the future information, it is invisible but closely connected to the current frame, which makes it provide useful relevant temporal information for action detection when reasonably predicted. Therefore, a custom learnable feature Q is used as a query, and the long-term historical feature F0 is input into another decoder as a key and value to predict the short-term future feature F3. The relevant formula process is as follows

[0059] {F0,F2}=Encoder(F long )

[0060] F1=Decoder(F short ,F0,F0)

[0061] F3=Decoder(Q,F0,F0)

[0062] Among them, Encoder represents the encoder, F0 represents the long-term historical features, F2 represents the category historical features, F1 represents the short-term current features, Decoder represents the decoder, F3 represents the short-term future features, and Q represents the learnable features. long and F short The channel dimension of is 2048, and the dimensions of the other feature channels are all 1024.

[0063] Step 3: Use temporal attention to enhance and fuse various temporal features to obtain fused features. The specific process is as follows:

[0064] like Figure 2 As shown in Figure 2, the short-term current feature F1, the category history feature F2, and the short-term future feature F3 are all input into the attention module of the same structure. The attention mechanism is used to enhance the features in space and channel, and then the transfer connection is performed to achieve progressive interaction. Finally, the three are connected in the channel dimension to obtain the fusion feature F m The relevant formula is as follows.

[0065] F i ′ =Attention(F i )

[0066]

[0067]

[0068] Where i∈{1,2,3}, Attention represents the attention module, AP is the first average pooling layer, and Concat represents the connection. m The channel dimension of is 3072, and the dimensions of the other feature channels are all 1024.

[0069] Continue reading Figure 2 The attention module includes a second global pooling layer 1, a second average pooling layer 4, a second fully connected layer 2 to a sixth fully connected layer 10, and a first activation layer to a third activation layer. The second global pooling layer 1 is connected to the first activation layer 3 through the second fully connected layer 2, and the second average pooling layer 4 is connected to the second activation layer 6 through the third fully connected layer 5. The output results of the first activation layer 3 and the second activation layer 6 are fused and input into the fourth fully connected layer 7. The output results of the fifth fully connected layer 8 after the third activation layer 9 and the output results of the first activation layer 3 and the second activation layer 6 are fused and input into the sixth fully connected layer 10. The output results of the fourth connection layer 7 and the sixth connection layer 10 are respectively fused and output after passing through an activation function.

[0070] Step 4: Classify the fused features and segment the initial features to obtain the temporal features of each hand-washing action and its corresponding action category as the action features. The specific process is as follows:

[0071] First, the fusion feature F output by the temporal Transformer m Input into the classification head to identify the action category of each frame of the current video stream, and then use this category information to segment the initial features extracted by the I3D network to obtain the temporal features α of each hand washing action in the current video stream. i , i corresponds to the category of hand washing action, α iAs action features, this not only allows for independent and sufficient fine-grained evaluation of each type of handwashing action, reducing the mutual interference between unrelated actions, but also removes the background noise between different types of handwashing actions.

[0072] Step 5: The action features are pooled through the first global maximum pooling layer to obtain the root feature, which is linearly mapped to two sub-features by two different first fully connected layers. The processing method of the features of each subsequent layer is the same. Finally, all the features of the last layer are regressed using MLP to obtain the score of each hand washing action. The scores of each hand washing action in the current video stream are added together to obtain the final prediction score. The specific process is as follows:

[0073] Existing evaluation heads have relatively poor regression capabilities, easily overlooking the inherent complexity of each handwashing action and making it difficult to mine detailed information about the action. To achieve fine-grained evaluation of handwashing actions, this paper proposes an integrated scoring tree that can evaluate each handwashing action from a coarse to fine scale, thereby significantly capturing the details of the handwashing action.

[0074] The details of the ensemble scoring tree are as follows Figure 3 As shown, Figure 3 In the example, FC stands for fully connected layer, and the subscript FC represents the number of the fully connected layer. First, the action features are aggregated through global maximum pooling to obtain the root feature, which is then linearly mapped to two sub-features by two different fully connected layers. The processing method of the features of each subsequent layer is the same. Finally, MLP is used to regress all the features of the last layer to obtain the score of each branch, which is then weighted and summed to obtain the predicted score of the hand washing action. The relevant formula is as follows

[0075] By formula i =EST(α i ) Get the score of the i-th hand washing action in the current video stream;

[0076] By formula Add up the scores of each hand washing action in the current video stream to obtain the predicted score of the hand washing action in the current video stream;

[0077] Among them, EST represents the integrated scoring tree, which represents the structure of the first global maximum pooling layer, two different first fully connected layers, and subsequent layers in step d, and n represents the number of categories of hand washing actions in the current video stream. i The channel dimension is 2048.

[0078] It should be noted that the present invention also includes a pre-training process: adjusting the parameters of each module involved in the evaluation process, continuously training the entire model until the loss function is minimized, and then stopping the training to obtain the optimal model, which is then used to perform online hand hygiene evaluation. Specifically, in the action detection part, the fusion feature Fm Input it into the classification head to get the category prediction probability p of the current action. Similarly, input feature F3 into the classification head to get the prediction probability p of the future frame ′ . Use the cross entropy loss function, also known as the first loss function, to supervise the prediction of each frame:

[0079] p=CF(F m )

[0080] p′=CF′(F3)

[0081] Loss1=CE(p,g)+μCE(p′,g′)

[0082] Where p is the category prediction probability of the current action), p ′ is the predicted probability of the future frame, CE represents the cross entropy loss function, CF and CF ′ Both are classifiers, g and g ′ are the true categories of the current and future actions, respectively, and μ is the balancing factor, which is set to 0.7.

[0083] In the action evaluation part, it is regarded as a regression task, and the mean square error loss function, also known as the second loss function, is used to correct the prediction score. The formula is as follows:

[0084] Loss2=MSE(S,S ′ )

[0085] Among them, S and S ′ are the predicted score and the true score respectively, and MSE represents the mean square error.

[0086] The following simulation analysis of the method provided by the present invention is conducted, and the effectiveness of the present invention is illustrated based on the simulation analysis results. The present invention is compared with several advanced related algorithms on the public hand hygiene dataset HHA300 (Table 1). The popular mean average accuracy (mAP) of online motion detection algorithms and the popular Spearman correlation rank coefficient (ρ) and relative L2 distance (RL2) of motion quality assessment algorithms are used as evaluation metrics. The results show that the method proposed in this invention achieves significant improvements across all evaluation metrics when applied to the hand hygiene dataset.

[0087] Table 1 Comparison of simulation analysis results

[0088] method Post information mAP (%)↑ ρ↑ RL2(×100)↓ <![CDATA[OadTR ★ ]]> ICCV2021 84.7 0.866 1.66 <![CDATA[TeSTra ★ ]]> ECCV2022 88.8 0.894 1.23 <![CDATA[LSTR ★ ]]> NeurIPS2021 89.1 0.890 1.25 This method - 90.1 0.927 0.99

[0089] Through the above technical solution, the present invention designs a temporal Transformer module: it includes an encoder, two decoders and a temporal attention to make full use of long-term historical information, short-term current information and short-term future information. Among them, the encoder is used to model long-term historical information, one decoder is used to model short-term current information, and the other decoder is used to predict short-term future information. The temporal attention uses the attention mechanism to enhance the feature information of different periods in a spatial and channel manner, and conducts progressive interaction through transitive connections. An integrated scoring tree module is also designed, which is a tree-like multi-branch multi-level evaluation model composed of a pooling layer, a fully connected layer and a multi-layer perceptron. In particular, the integrated scoring tree can evaluate each hand washing action from coarse to fine, so that it can pay good attention to the detailed information of the hand washing action.

[0090] Example 2

[0091] Based on Example 1, Example 2 of the present invention further provides an online hand hygiene assessment system based on temporal attention, the system comprising:

[0092] The temporal feature acquisition module is used to obtain the online handwashing video stream and model various temporal features through the encoder and decoder;

[0093] The fusion feature acquisition module is used to use temporal attention to enhance and fuse various temporal features to obtain fusion features;

[0094] The action feature acquisition module is used to classify the fused features and segment the initial features to obtain the temporal features of each hand washing action and its corresponding action category as the action features;

[0095] The evaluation module is used to aggregate the action features through the first global maximum pooling layer to obtain the root feature, which is linearly mapped to two sub-features by two different first fully connected layers. The processing method of the features of subsequent layers is the same. Finally, MLP is used to regress all the features of the last layer to obtain the score of each hand washing action. The scores of each hand washing action in the current video stream are added together to obtain the final prediction score.

[0096] Specifically, the time series feature acquisition module further includes:

[0097] 224×224 visible light video frames and 224×224 optical flow frames are extracted from the online hand washing video stream and then input into the pre-trained I3D network. 1024-dimensional visible light features and optical flow features are obtained through feature extraction. These two modal features are then concatenated in the channel dimension to obtain 2048-dimensional initial features.

[0098] Specifically, the time series feature acquisition module is further used to:

[0099] Extract the first preset length of features from the initial features as long-term features F long , extract the second preset length feature from the initial feature as the long-term feature F short , the first preset length is greater than the second preset length, and the formulas for modeling various types of time series features through the encoder and decoder are as follows

[0100] {F0,F2}=Encoder(F long )

[0101] F1=Decoder(F short ,F0,F0)

[0102] F3=Decoder(Q,F0,F0)

[0103] Among them, Encoder represents the encoder, F0 represents the long-term historical features, F2 represents the category historical features, F1 represents the short-term current features, Decoder represents the decoder, F3 represents the short-term future features, and Q represents the learnable features.

[0104] More specifically, the fusion feature acquisition module is further used to:

[0105] The short-term current feature F1, the category history feature F2, and the short-term future feature F3 are all input into the attention module of the same structure. The attention mechanism is used to enhance the features in space and channel, and then the transfer connection is performed to achieve progressive interaction. Finally, the three are connected in the channel dimension to obtain the fusion feature F m .

[0106] More specifically, the fusion feature acquisition module is further used to:

[0107] Construct the following formula to obtain the fusion feature F m

[0108] F′ i =Attention(F i )

[0109]

[0110]

[0111] Among them, i∈{1,2,3}, Attention represents the attention module, AP is the first average pooling layer, and Concat represents connection.

[0112] More specifically, the attention module includes a second global pooling layer, a second average pooling layer, a second fully connected layer to a sixth fully connected layer, and a first activation layer to a third activation layer. The second global pooling layer is connected to the first activation layer through the second fully connected layer, and the second average pooling layer is connected to the second activation layer through the third fully connected layer. The output results of the first activation layer and the second activation layer are fused and input into the fourth fully connected layer. The output results of the fifth fully connected layer after the third activation layer and the output results of the first activation layer and the second activation layer are fused and input into the sixth fully connected layer. The output results of the fourth connection layer and the sixth connection layer are respectively fused and output after passing through an activation function.

[0113] More specifically, the action feature acquisition module is further used to:

[0114] The fusion feature F m Input into the classification head to identify the action category of each frame of the current video stream, and then use this category information to segment the initial features extracted by the I3D network to obtain the temporal features α of each hand washing action in the current video stream. i , i corresponds to the category of hand washing action, α i As an action feature.

[0115] More specifically, the evaluation module is further configured to:

[0116] By formula i =EST(α i ) Get the score of the i-th hand washing action in the current video stream;

[0117] By formula Add up the scores of each hand washing action in the current video stream to obtain the predicted score of the hand washing action in the current video stream;

[0118] Among them, EST represents the integrated scoring tree, which represents the structure of the first global maximum pooling layer, two different first fully connected layers, and subsequent layers in step d, and n represents the number of categories of hand washing actions in the current video stream.

[0119] More specifically, the system further includes a pre-training module, which is used to:

[0120] Adjust the parameters of each module involved in the evaluation process, continuously train the entire model until the loss function is minimized, and stop training to obtain the optimal model. Use this optimal model to perform online hand hygiene evaluation. The loss function includes the first loss function Loss1 = CE(p, g) + μCE(p′, g′)

[0121] Where p is the predicted probability of the current action category and p = CF(F m), p′ is the predicted probability of the future frame and p′=CF′(F3), CE represents the cross entropy loss function, CF and CF′ are both classifiers, g and g′ are the true categories of the current and future actions respectively, and μ is the balancing factor;

[0122] The loss function also includes a second loss function Loss2 = MSE (S, S')

[0123] Among them, S and S′ are the predicted score and the true score respectively, and MSE represents the mean square error.

[0124] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An online hand hygiene assessment method based on temporal attention, characterized by: The method comprises: Step a: Obtain an online handwashing video stream and model various temporal features through the encoder and decoder respectively; Step b: Use temporal attention to enhance and fuse various temporal features to obtain fused features; short-term current features , category historical features and short-term future characteristics All are input into the attention module of the same structure, and the attention mechanism is used to enhance the features in space and channel. Then the transfer connection is performed to achieve progressive interaction. Finally, the three are connected in the channel dimension to obtain the fusion feature. The attention module includes a second global pooling layer, a second average pooling layer, a second fully connected layer to a sixth fully connected layer, and a first activation layer to a third activation layer. The second global pooling layer is connected to the first activation layer through the second fully connected layer, and the second average pooling layer is connected to the second activation layer through the third fully connected layer. The output results of the first activation layer and the second activation layer are fused and input into the fourth fully connected layer. The output result of the fifth fully connected layer after passing through the third activation layer and the output result of the first activation layer and the second activation layer are fused and input into the sixth fully connected layer. The output results of the fourth connection layer and the sixth connection layer are respectively fused and output through an activation function. Step c: Classify the fused features and segment the initial features to obtain the temporal features of each hand washing action and its corresponding action category as the action features; Step d: The action features are pooled through the first global maximum pooling layer to obtain the root feature, which is linearly mapped to two sub-features by two different first fully connected layers. The processing method of the features of subsequent layers is the same. Finally, all features of the last layer are regressed using MLP to obtain the score of each hand washing action. The scores of each hand washing action in the current video stream are added together to obtain the final prediction score.

2. The online hand hygiene assessment method based on temporal attention according to claim 1 is characterized in that: The step a also includes: Extracting 224 online handwashing video streams 224 visible light video frames and 224 224 optical flow frames are then input into the pre-trained I3D network to obtain 1024-dimensional visible light features and optical flow features through feature extraction. These two modal features are then connected in the channel dimension to obtain 2048-dimensional initial features.

3. The online hand hygiene assessment method based on temporal attention according to claim 1, characterized in that The step a comprises: Extract the first preset length of features from the initial features as long-term features , extract the second preset length feature from the initial feature as the long-term feature F short , the first preset length is greater than the second preset length, and the formulas for modeling various types of time series features through the encoder and decoder are as follows in, represents the encoder, Represents long-term historical characteristics, Represents the historical characteristics of the category, Represents short-term current characteristics, Describes the decoder, Represents short-term future characteristics, Represents learnable features.

4. The online hand hygiene assessment method based on temporal attention according to claim 1 is characterized in that The step b further comprises: Construct the following formula to obtain the fusion feature in, 1,2,3}, represents the attention module, is the first average pooling layer, Indicates a connection.

5. The online hand hygiene assessment method based on temporal attention according to claim 1, characterized in that The step c comprises: The fusion features Input into the classification head to identify the action category of each frame of the current video stream, and then use this category information to segment the initial features extracted by the I3D network to obtain the temporal features of each hand washing action in the current video stream. , i corresponds to the category of hand washing action, As an action feature.

6. The online hand hygiene assessment method based on temporal attention according to claim 5 is characterized in that: The step d comprises: By formula Get the score of the i-th hand washing action in the current video stream; By formula Add up the scores of each hand washing action in the current video stream to obtain the predicted score of the hand washing action in the current video stream; in, Represents the integrated scoring tree, which represents the structure of the first global maximum pooling layer, two different first fully connected layers, and subsequent layers in step d. Indicates the number of categories of hand washing actions in the current video stream.

7. The online hand hygiene assessment method based on temporal attention according to claim 6 is characterized in that: The method also includes a pre-training process: Adjust the parameters of each module involved in the evaluation process, continuously train the entire model until the loss function is minimized, stop training, obtain the optimal model, and use the optimal model to perform online hand hygiene evaluation. The loss function includes the first loss function in, Predict the probability of the category of the current action and , is the predicted probability of the future frame and , represents the cross entropy loss function, and are all classifiers, and are the true categories of current and future actions, is the balance factor; The loss function also includes a second loss function in, and are the predicted score and the true score respectively, represents the mean square error.

8. An online hand hygiene assessment system based on temporal attention, characterized by: The system comprises: The temporal feature acquisition module is used to obtain the online handwashing video stream and model various temporal features through the encoder and decoder; The fusion feature acquisition module is used to use temporal attention to enhance and fuse various temporal features to obtain fusion features; short-term current features , category historical features and short-term future characteristics All are input into the attention module of the same structure, and the attention mechanism is used to enhance the features in space and channel. Then the transfer connection is performed to achieve progressive interaction. Finally, the three are connected in the channel dimension to obtain the fusion feature. The attention module includes a second global pooling layer, a second average pooling layer, a second fully connected layer to a sixth fully connected layer, and a first activation layer to a third activation layer. The second global pooling layer is connected to the first activation layer through the second fully connected layer, and the second average pooling layer is connected to the second activation layer through the third fully connected layer. The output results of the first activation layer and the second activation layer are fused and input into the fourth fully connected layer. The output result of the fifth fully connected layer after passing through the third activation layer and the output result of the first activation layer and the second activation layer are fused and input into the sixth fully connected layer. The output results of the fourth connection layer and the sixth connection layer are respectively fused and output through an activation function. The action feature acquisition module is used to classify the fused features and segment the initial features to obtain the temporal features of each hand washing action and its corresponding action category as the action features; The evaluation module is used to aggregate the action features through the first global maximum pooling layer to obtain the root feature, which is linearly mapped to two sub-features by two different first fully connected layers. The processing method of the features of subsequent layers is the same. Finally, MLP is used to regress all the features of the last layer to obtain the score of each hand washing action. The scores of each hand washing action in the current video stream are added together to obtain the final prediction score.

Images