A six-step hand washing evaluation method and device based on deep learning

Abstract

The application discloses a six-step hand washing evaluation method and device based on deep learning, and the method comprises the following steps: performing frame decomposition and optical flow extraction on a hand washing video, and then performing feature coding to obtain a feature map; feeding the feature map subjected to the feature coding into a first prediction network to obtain a first prediction result; inputting the first prediction result into a plurality of cascaded second prediction networks to obtain a hand washing step segmentation result; combining the obtained hand washing step segmentation result with original features to obtain features belonging to each step, feeding the features of each step into corresponding key action scorers to perform scoring; after obtaining the scores of each step, adding the scores of each step to obtain a total score, comparing the total score with a preset standard score to obtain an evaluation result; and the application has the advantage that the evaluation result is relatively accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, more particularly to a six-step hand washing evaluation method and device based on deep learning. BACKGROUND

[0002] In 2005, the World Health Organization designated October 15 as "World Hand Washing Day". Unfortunately, not many people in life can wash their hands well, and a considerable number of people have not developed good hand washing habits. The droplets sprayed by patients with infectious viruses not only spread into the air but also stick to their hands and survive for a long time. If they touch others or other things again with their hands, they will have a high probability of infecting others. Therefore, hands are an important medium for the spread of these viruses. Studies have shown that scientific hand washing can reduce the risk of illness by 20%. Therefore, it is very important to have a correct evaluation of the hand washing process. Through evaluation, medical staff can correct their hand washing behavior to reduce the risk of disease transmission as much as possible.

[0003] Current six-step hand washing method evaluation methods mostly use additional sensors to evaluate the accuracy of the six-step hand washing method, which results in additional costs. In recent years, action evaluation, which evaluates the quality of actions, has attracted widespread attention. It has important applications in many fields of the real world, such as sports and medicine. Past action evaluation models often directly evaluate a video and calculate a score, such as Chinese patent publication CN112906453A, which discloses an intelligent identification and real-time feedback hand washing monitoring system. However, these methods ignore many details in long actions, which usually involve several short steps, thus reducing the performance of long action evaluation. The six-step hand washing method prescribed by the World Health Organization (WHO) is a standard long action that includes six steps. There are two main problems in the six-step hand washing method evaluation task. First, each video contains some or all of the six steps, so existing action evaluation models cannot know which frames belong to which step, which poses a great challenge to accurate evaluation of the entire video. Second, in each step, there are several key actions that determine the quality, so not all actions are useful for six-step hand washing method evaluation. Unnecessary actions can easily interfere with the evaluation result, leading to inaccurate evaluation. SUMMARY

[0004] The technical problem to be solved by the present application is the low accuracy of the existing six-step hand washing evaluation method.

[0005] The present application solves the above technical problems by the following technical means: a six-step hand washing evaluation method based on deep learning, the method comprising:

[0006] Step one: after frame decomposition and optical flow extraction of the hand washing video, feature encoding is performed to obtain a feature map;

[0007] Step two: send the feature map coded with features into the first prediction network to obtain the first prediction result, the first stage prediction network includes a dimension reduction convolution layer, a hollow residual layer and a prediction convolution layer connected in sequence;

[0008] Step three: input the first prediction result into multiple cascaded second prediction networks to obtain more accurate hand washing step segmentation results, the second prediction network adds a linear transformer before its dimension reduction convolution layer compared with the first prediction network;

[0009] Step four: combine the obtained hand washing step segmentation results with the original features to obtain the features belonging to each step, and send the features of each step into the corresponding key action scorer for scoring;

[0010] Step five: after obtaining the score of each step, add the score of each step to obtain the total score, and compare the total score with the preset standard score to obtain the evaluation result.

[0011] The application is not to process the whole video, but to perform frame decomposition and optical flow extraction on the hand washing video, and then to code the features to obtain the feature map, and then to further process the feature map, which will not ignore many details in long actions, improve the performance of hand washing action evaluation, and the way of extracting the feature map only focuses on several key actions, not all actions in the video, avoiding useless actions that easily interfere with the evaluation result, improving the accuracy of evaluation. Secondly, the feature map coded with features is sent into the first prediction network to obtain a preliminary first prediction result, and then the first prediction result is input into multiple cascaded second prediction networks to obtain the hand washing step segmentation result, so as to know which frame belongs to which step, which is beneficial to accurately evaluate the whole video.

[0012] Further, the step one comprises:

[0013] The hand washing video is frame decomposed and optical flow extracted and sent into the feature extraction module in the pre-trained I3D network for feature extraction to obtain 1024-dimensional RGB data and optical flow data, and then the two data features are connected to obtain a 2048-dimensional feature map.

[0014] Further, the step two comprises:

[0015] The dimension reduction convolution layer is used to reduce the input 2048-dimensional feature map to 256-dimensional feature, the hollow residual layer includes hollow convolution and residual link, which is used to establish the relationship between frames on the feature, and the prediction convolution layer is used to predict the final classification result.

[0016] Further, the step three comprises:

[0017] The first prediction result is input into the second prediction network, further modeling of inter-frame relationship is performed through a linear Transformer, global relationship between frames is modeled by using a self-attention module of the Transformer, then further correction is performed through a dimension reduction convolution layer, a hollow residual layer and a prediction convolution layer, and finally the final hand washing step segmentation result is obtained.

[0018] Further, the step four comprises:

[0019] The key action scorer comprises a global average pooling layer, two fully connected layers and two Sigmoid layers, the global average pooling layer is connected with one of the two fully connected layers, the outputs of the two fully connected layers are connected with the two Sigmoid layers respectively, the features of each step are input into the global average pooling layer, the global average pooling layer and the two fully connected layers are used for dimension reduction of the features of each step, and the Sigmoid layers are used for scoring the features of each step after dimension reduction.

[0020] The application further provides a six-step hand washing method evaluation device based on deep learning, comprising:

[0021] The video acquisition unit is configured to acquire the hand washing process video of the hand washing personnel.

[0022] The memory is configured to store the hand washing process video captured by the video acquisition unit.

[0023] The data reading unit is configured to read the hand washing process video of the hand washing personnel from the memory and send the hand washing process video into the processor when it is detected that there is a hand washing process video that has not been evaluated in the memory.

[0024] The processor is configured to execute the six-step hand washing method evaluation method.

[0025] Further, the video acquisition unit is arranged on the hand washing pool and is further configured to capture the hand washing process video of the hand washing personnel when the hand washing personnel is washing hands on the hand washing pool, and transmit the captured hand washing process video to the memory for storage when the hand washing personnel finishes the hand washing process.

[0026] Further, the evaluation device further comprises a display unit configured to display the evaluation result returned by the processor.

[0027] The method has the advantages that the method is not for processing the whole video, but for frame decomposition and light flow extraction on the hand washing video, feature coding to obtain a feature map, and further processing of the feature map, so that many details in long actions are not ignored, the performance of hand washing action evaluation is improved, and the feature map extraction method only focuses on several key actions, and does not focus on all actions in the video, so that useless actions are avoided to interfere with the evaluation result, and the accuracy of evaluation is improved. Secondly, the feature map subjected to feature coding is input into a first prediction network to obtain a preliminary first prediction result, and then the first prediction result is input into a plurality of cascaded second prediction networks to obtain a hand washing step segmentation result, so that it can be known which frame belongs to which step, and the whole video can be accurately evaluated. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 A network model schematic diagram of the six-step hand washing evaluation method based on deep learning provided by the embodiment of the application;

[0029] Figure 2 A hand washing step decomposition schematic diagram of the six-step hand washing evaluation method based on deep learning provided by the embodiment of the application;

[0030] Figure 3 A video acquisition unit, a hand washing station and a hand washing personnel position schematic diagram in the six-step hand washing evaluation method based on deep learning provided by the embodiment of the application;

[0031] Figure 4 A comparison of the evaluation results of the six-step hand washing evaluation method based on deep learning provided by the embodiment of the application and the prior art method;

[0032] Figure 5 A six-step hand washing evaluation device schematic diagram provided by the embodiment of the application. DETAILED DESCRIPTION

[0033] To make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely in combination with the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0034] Embodiment 1

[0035] As shown in Figure 1 and Figure 2 , a six-step hand washing evaluation method based on deep learning, the method comprises:

[0036] Step one: video feature coding refers to compressing the original video to obtain a feature representation that can reflect the video information. Existing action segmentation networks usually only use RGB features for subsequent prediction. On this basis, the application further extracts the optical flow data of the video. Optical flow method is an important method of motion analysis. Therefore, the hand washing video is frame decomposed and the optical flow is extracted and sent to the feature extraction module in the pre-trained I3D network for feature extraction to obtain 1024-dimensional RGB data and optical flow data. Then the two data features are connected to obtain a 2048-dimensional feature map; wherein the hand washing video is collected by a video acquisition unit with video recording function, as shown in Figure 3 The video acquisition unit is installed on the wash basin and is required to be able to shoot the hand washing video of the hand washing personnel without obstruction. The hand washing personnel is required to face the wash basin directly in order to obtain a six-step hand washing method video of the hand washing personnel; when a hand washing personnel needs to be evaluated by the six-step hand washing method, the video acquisition unit is started to begin recording. When the hand washing personnel finishes the hand washing process, the video acquisition unit will transmit the shot hand washing process video to the storage for storage for subsequent reading and browsing.

[0037] Step two: the feature map after feature coding is input into the first prediction network to obtain the first prediction result. The first stage prediction network includes a dimension reduction convolution layer with a size of 2048*256 and a kernel size of 1, four empty residual layers with a size of 256*256, an empty rate of 2, 4, 8, 16, and a kernel size of 3, and a prediction convolution layer with a size of 256*7. The dimension reduction convolution layer is used to reduce the input 2048-dimensional feature map to 256-dimensional feature to reduce the calculation amount and compress the feature. The empty residual layer includes empty convolution and residual link, which is used to establish the relationship between frames on the feature. The prediction convolution layer is used to predict the final classification result. The overall process is first to compress the feature information by using the empty convolution, and then to obtain the rough prediction result of the first stage through the compressed feature.

[0038] Step three: input the first prediction result into multiple cascaded second prediction networks to obtain the hand washing step segmentation result. The second prediction network adds a linear Transformer before its dimension reduction convolution layer compared with the first prediction network. The specific process is that the compressed first prediction result is further modeled by the linear Transformer for the inter-frame relationship, and the self-attention module of the Transformer is used to model the global relationship between frames. The attention mechanism formula is:

[0039] Attention(Q,K,V)=φ(Q)φ(K) T V

[0040] Wherein, Attention() is an attention function, the input vectors of the attention layer are Query vector (Q), Key vector (K) and Value vector (V), and phi(·) = elu(·) + 1. The attention operation selects relevant information by measuring the similarity between Q and each K, and the output vector is obtained by the weighted sum of the value of the similarity score. Wherein, in the self-attention, Q, K and V are vectors corresponding to the same feature. After the linear transformation, the same is further modified by the dimension reduction convolution layer, the hollow residual layer and the prediction convolution layer, and the same modification is performed 4 times, and finally the final step segmentation result is obtained.

[0041] Step four: combine the obtained hand washing step segmentation result with the original feature to obtain the feature belonging to each step, and input the feature of each step into the corresponding key action scorer for scoring; the key action scorer is composed of a plurality of branches with the same structure and unshared parameters, each branch mainly includes two first fully connected layers with a size of 2048*256 and 256*1, a second fully connected layer and a Sigmoid layer with learnable parameters, the global average pooling layer output is input into a plurality of branches, and each branch is sequentially connected with the first fully connected layer, the second fully connected layer and the learnable Sigmoid layer. The feature of each step is input into the global average pooling layer in the corresponding key action scorer, and the global average pooling layer and the two fully connected layers (the first fully connected layer and the second fully connected layer) are used for key action extraction of the obtained corresponding step feature. Each branch corresponds to different key actions, and the learnable Sigmoid layer is used for scoring the key action feature of each step. Since the completion degree of the key action determines the completion quality of each step, the evaluation of each hand washing step through the key action is more accurate.

[0042] Step five: after obtaining the score of each step, the scores of each step are added to obtain the total score of the six-step hand washing method video, and the total score is compared with the preset standard score to obtain the evaluation result. Since the sum is obtained after the evaluation of each short step, the accuracy is much higher than that of directly evaluating the whole video.

[0043] As shown in Figure 4 The effectiveness of the method provided by the application is verified on the six-step hand washing method data set, and compared with the existing method, the method of the application achieves the best effect on the evaluation indexes of step segmentation: accuracy, edit distance, F1 score, and the action evaluation indexes of Spearman rank correlation coefficient and related L2 distance.

[0044] By the above technical solution, the application does not process the whole video, but performs frame decomposition and light flow extraction on the hand washing video, and then performs feature coding to obtain a feature map, and further processes the feature map, so that many details in long actions are not ignored, the performance of hand washing action evaluation is improved, and the feature map extraction method only focuses on several key actions, and does not focus on all actions in the video, so as to avoid that useless actions easily interfere with the evaluation result, and improve the accuracy of evaluation. Secondly, the feature map subjected to feature coding is input into a first prediction network to obtain a preliminary first prediction result, and then the first prediction result is input into a plurality of cascaded second prediction networks to obtain a hand washing step segmentation result, so that it can be known which frame belongs to which step, and the whole video can be accurately evaluated.

[0045] Embodiment 2

[0046] As Figure 5 shown, corresponding to the embodiment 1 of the application, the embodiment 2 of the application also provides a six-step hand washing method evaluation device based on deep learning, comprising:

[0047] a video acquisition unit, configured to acquire a hand washing process video of a hand washing person;

[0048] a memory, configured to store the hand washing process video photographed by the video acquisition unit;

[0049] a data reading unit, configured to read the hand washing process video of the hand washing person from the memory and input into the processor when it is detected that there is a hand washing process video which has not been evaluated in the memory;

[0050] a processor, configured to execute the six-step hand washing method evaluation method of the embodiment 1.

[0051] Specifically, the video acquisition unit is arranged on a hand washing pool, and is further configured to photograph the hand washing process video of the hand washing person when the hand washing person washes hands on the hand washing pool, and transmit the photographed hand washing process video to the memory for storage when the hand washing person finishes the hand washing process.

[0052] Specifically, the evaluation device further comprises a display unit, configured to display the evaluation result returned by the processor.

[0053] The above embodiments are only used to illustrate the technical solutions of the application, but not to limit the same; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features thereof; and these modifications or replacements do not make the essence of the corresponding technical solution deviate from the spirit and scope of the technical solutions of the embodiments of the application.

Claims

1. A six-step handwashing assessment method based on deep learning, characterized in that, The method includes: Step 1: After performing frame decomposition and optical flow extraction on the handwashing video, feature encoding is performed to obtain a feature map; the handwashing video is then decomposed into frames and optical flow extracted and fed into the feature extraction module of the pre-trained I3D network to extract 1024-dimensional RGB data and optical flow data. These two data features are then concatenated to obtain a 2048-dimensional feature map. Step 2: The feature map after feature encoding is fed into the first prediction network to obtain the first prediction result. The prediction network in the first stage includes a dimensionality reduction convolutional layer, a dilated residual layer, and a prediction convolutional layer connected in sequence. The dimensionality reduction convolutional layer is used to reduce the input 2048-dimensional feature map to 256-dimensional features. The dilated residual layer includes dilated convolution and residual connections, which are used to establish the relationship between frames on the features. The prediction convolutional layer is used to predict the final classification result. Step 3: Input the first prediction result into multiple cascaded second prediction networks to obtain the handwashing step segmentation result. Compared with the first prediction network, the second prediction network adds a linear Transformer before its dimensionality reduction convolutional layer. Input the first prediction result into the second prediction network, and further model the inter-frame relationship through the linear Transformer. The self-attention module of the Transformer is used to model the global relationship between frames. Then, it is further corrected through dimensionality reduction convolutional layers, dilated residual layers and prediction convolutional layers to finally obtain the final handwashing step segmentation result. Step 4: Combine the obtained handwashing step segmentation results with the original features to obtain the features belonging to each step, and send the features of each step into its corresponding key action scorer for scoring. Step 5: After obtaining the score for each step, add up the scores for each step to get the total score. Compare the total score with the preset standard score to obtain the evaluation result.

2. The six-step handwashing assessment method based on deep learning according to claim 1, characterized in that, Step four includes: The key action scorer consists of a global average pooling layer, two fully connected layers, and two sigmoid layers. The output of the global average pooling layer is connected to a fully connected layer, and the outputs of the two fully connected layers are each connected to a sigmoid layer. The features of each step are input to the global average pooling layer. The global average pooling layer and the two fully connected layers are used to reduce the dimensionality of the features obtained at each step. The sigmoid layers are used to score the dimensionality-reduced features at each step.

3. A deep learning-based six-step handwashing technique evaluation device, characterized in that, include; The video capture unit is used to capture video of the handwashing process of people washing their hands. The memory is used to store the handwashing process video captured by the video acquisition unit; The data reading unit is used to read the handwashing process video of the person washing their hands from the memory and send it to the processor when it detects that there is an unevaluated handwashing process video in the memory; A processor for executing the six-step handwashing technique evaluation method according to any one of claims 1-2.

4. The six-step handwashing technique evaluation device based on deep learning according to claim 3, characterized in that, The video acquisition unit is installed on the handwashing sink and is also used to record the handwashing process of the person washing their hands when they are doing so. After the person finishes washing their hands, the video of the handwashing process is transmitted to the memory for storage.

5. The six-step handwashing technique evaluation device based on deep learning according to claim 3, characterized in that, It also includes a display unit for displaying the evaluation results returned by the processor.

Images