Standardized hand washing automated assessment method and system
By setting a standard matrix template and a 3D interactive hand posture evaluation model, combined with CNN and Transformer technologies, the problem of lack of unified standards in the handwashing process is solved, achieving highly accurate automatic evaluation and providing standardized handwashing guidance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ANHUI UNIV
- Filing Date
- 2023-06-15
- Publication Date
- 2026-06-05
AI Technical Summary
In existing technologies, there is a lack of unified standards for the standardized assessment of the handwashing process, which leads to strong subjectivity in human judgment and makes it difficult to effectively guide standardized handwashing.
By adopting a standard matrix template and evaluation model for the handwashing process, the system scores the 3D interactive hand postures and standard matrix templates obtained during continuous handwashing. It utilizes CNN feature extraction and Transformer encoding and decoding techniques, combined with interactive hand feature enhancement, to achieve automatic evaluation of the handwashing process.
It enables accurate assessment of the handwashing process, improves the accuracy of automatic assessment of handwashing standardization, provides a unified assessment standard, and reduces subjective errors.
Smart Images
Figure CN116758634B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of hand posture assessment technology, and in particular relates to a standardized automatic handwashing assessment method and system. Background Technology
[0002] Frequent and proper handwashing is crucial to individual and collective health. However, many people do not wash their hands properly, and hand hygiene issues still have a high rate of causing illness. Assessing the posture during handwashing can help determine whether handwashing is done correctly; however, human judgment on whether handwashing is done correctly is highly subjective, lacks a unified standard, and is difficult to provide guidance on proper handwashing. Summary of the Invention
[0003] To address the problems in the prior art, the present invention proposes the following technical solution:
[0004] One objective of this invention is to provide a standardized automated handwashing assessment method, wherein the method is as follows:
[0005] Establish a standard matrix template and evaluation model for the handwashing process;
[0006] Input the 3D interactive hand poses and standard matrix templates obtained during continuous handwashing into the evaluation model to score the continuous handwashing process.
[0007] Preferably, the standard matrix template is formed as follows: according to the standardized handwashing steps, N standard action RGB images of the start, peak and process of each step are taken for real three-dimensional joint point annotation, and the annotation results are normalized and integrated into a matrix template. There are N standard matrix templates for each step.
[0008] Preferably, the evaluation model scores the continuous handwashing process as follows:
[0009] Input a 3D interactive hand gesture obtained during any handwashing step, and determine which step in the handwashing process the 3D interactive hand gesture belongs to;
[0010] Retrieve the standard matrix template corresponding to the handwashing step after the 3D interactive hand posture judgment, and compare and score it with the 3D interactive hand posture.
[0011] The total score is the sum of all the handwashing steps.
[0012] Preferably, the method for comparing and scoring the standard matrix template with the 3D interactive hand pose is as follows:
[0013] Compare each 3D interactive hand gesture with all sequence frames in the standard matrix template. If the sequence frames in the 3D interactive hand gesture are the same as those in the standard matrix template, the difference is recorded as 0. If they are different, the difference is recorded as X. Sum the differences of all sequence frames and determine whether the difference exceeds the standard value Y. If it does not exceed the standard value, the handwashing step is determined to meet the standardized handwashing requirements, and the score is increased by P.
[0014] The preferred method for determining which step in the handwashing process a 3D interactive hand gesture belongs to is:
[0015] Arrange the three-dimensional coordinates of the 3D interactive hand gesture into a matrix according to the joint point order, and then normalize the matrix.
[0016] Traverse the sequence frames of each standard matrix template, calculate the cosine distance and Euclidean distance between the normalized matrix and the standard matrix template respectively, and calculate the total distance by weighting the cosine distance and Euclidean distance with a 1:1 ratio. Select the handwashing step corresponding to the standard matrix template with the smallest total distance as the handwashing step to which the 3D interactive hand gesture belongs.
[0017] The preferred method for obtaining 3D interactive hand poses is as follows:
[0018] Acquire RGB images of the handwashing process and perform preprocessing;
[0019] The preprocessed RGB image is then subjected to CNN feature extraction. The CNN network consists of four parts. The first part includes a standardized 2D convolutional layer (Standardization Conv2d) with a kernel size of 7×7 and a stride of 2 to transform the input data into 128×128×64 pixels. This is followed by a group normalization layer (Group). The input data consists of a Normalization layer and a ReLU activation layer, followed by MaxPool to obtain a 64×64×64 data stream. The second, third, and fourth parts are obtained by stacking three types of convolutional blocks 3, 6, and 9 times, respectively. Each convolutional block consists of three 2D convolutional layers with kernel sizes of 1×1, 3×3, and 1×1. The difference lies in the number of channels between each block: the first convolutional block has 64, 64, and 256 channels, the second block has 128, 128, and 512 channels, and the third block has 256, 256, and 1024 channels. After CNN feature extraction, the shape of the input data is changed from 256×256×3 to 16×16×1024.
[0020] The data after CNN feature extraction is encoded using Transformer. A 2D convolutional layer with a 1×1 kernel and a stride of 1 is used to transform the data shape from 16×16×1024 to 16×16×768. The 16×16 features are stretched into a one-dimensional sequence, resulting in a 256×768 data stream. Tokens are added to each sequence and concatenated to obtain a 257×768 data stream, which is then fed into a Transformer block stacked 12 times for encoding, while maintaining the original shape.
[0021] The Transformer-encoded data is decoded, and the output data is divided into three paths. One path is connected to a linear layer to predict the relative depth of the hands to the root node. The other two paths are connected to two layers of deconvolution, upsampled to obtain left-hand and right-hand features with a shape of 64×64×256. The left-hand and right-hand features are then connected to the Interactive Hand Enhancement (IHE) module to obtain the enhanced left-hand features. A 1×1 2D convolution is used to change the number of channels to 1344, i.e., 64×64×(64×21). The first two 64s represent the prediction results on the X and Y axes, the third 64 represents the prediction results of the relative depth Z within the hand, and the 21 dimensions represent the 21 joints of each hand. This data is then connected to a classifier to obtain a 1×1×1×21 data stream, i.e., the 2.5D coordinates (X-axis, Y-axis, and relative depth Z within the hand) of the 21 joints of each hand. The 2.5D coordinates are added to the relative depth between the hands in the third dimension to obtain the 3D coordinates of each joint.
[0022] Preferably, the method for enhancing the interactive hand features is as follows:
[0023] The two upsampled outputs are represented as left-hand and right-hand features. The left-hand features are used as the query, and the right-hand object features are used as the key. The pairwise relationship between the left and right-hand features is modeled. As shown in the formula below, FL i + This indicates the enhanced object feature, where the subscript i represents the given position i of the feature.
[0024]
[0025] Where Ω represents the set of all positions on the key. V is a value transformation function parameterized by Wv, and w(i, j) is the pairwise spatial similarity score between query position i and key position j.
[0026] The formula for calculating W(i, j) is:
[0027]
[0028] in , Let W represent the query and key at positions i and j, respectively. q W k W V All are 1×1 convolutions. This module allows for further differentiation of hand features, enhancing the feature set.
[0029] Another object of the present invention is to provide a standardized automated handwashing assessment system, the assessment system comprising:
[0030] A local storage module, wherein a standard matrix template is stored.
[0031] The system includes an intelligent assessment module, which acquires 3D interactive hand gestures and standard matrix templates into the assessment model to score the continuous handwashing process.
[0032] Preferred options also include:
[0033] The hand acquisition module is used to acquire RGB images of the handwashing process;
[0034] Central processing unit, wherein the central processing unit has an embedded intelligent evaluation module;
[0035] The GPU image processing module is electrically connected to the central processing unit and is used to extract 3D interactive hand gestures from the acquired RGB images.
[0036] Preferably, it also includes a display module for displaying the score obtained by the intelligent evaluation module.
[0037] The beneficial effects of the present invention are as follows: The evaluation method of the present invention can obtain RGB images through ordinary image acquisition devices, utilize CNN feature extraction, Transformer encoding and custom decoding, and combine interactive hand feature enhancement to obtain more accurate interactive hand 3D pose, and further evaluate and score the handwashing process based on normative standards through intelligent evaluation methods, thereby helping to determine the standardization of handwashing. Attached Figure Description
[0038] Figure 1 The diagram shown is a schematic of a standardized automated handwashing assessment method and system;
[0039] Figure 2 This is a schematic diagram of the GPU image processing module;
[0040] Figure 3 The diagram shown is a schematic of the intelligent evaluation module;
[0041] Figure 4 The diagram shown is a schematic of CNN feature extraction.
[0042] Figure 5(a) shows a schematic diagram of the Transformer encoder;
[0043] Figure 5(b) shows a schematic diagram of Transformer block 13 inside the Transformer encoder;
[0044] Figure 6 This is a schematic diagram of the decoder;
[0045] Figure 7 The diagram shown is a schematic of the interactive hand feature enhancement module;
[0046] Figure 8 The image shown is an RGB image of hands facing each other with open arms;
[0047] Figure 9 What is shown is Figure 8 A schematic diagram of the 3D joint coordinates output by the decoding unit;
[0048] Figure 10 The image shown is an RGB image of hands crossed and facing each other;
[0049] Figure 11 What is shown is Figure 10 A schematic diagram of the 3D joint coordinates output by the decoding unit;
[0050] Figure 12 This diagram illustrates the process of an RGB image being processed by the GPU image processing module. Detailed Implementation
[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments.
[0052] Example
[0053] Figure 1 The diagram shown is a schematic of the standardized automatic handwashing assessment method and system in this embodiment. The standardized automatic handwashing assessment method and system includes a hand acquisition module 1, a central processing unit 2, a GPU image processing module 3 (GPU Graphics Processing Unit), an intelligent assessment module 4, a display module 5, and a local storage module 6. The central processing unit 2 is electrically connected to the hand acquisition module 1, the display module 5, and the local storage module 6, and the intelligent assessment module 4 is embedded in the central processing unit 2. The GPU image processing module 3 is electrically connected to the central processing unit 2, and the GPU image processing module 3 accelerates the image processing of the central processing unit 2.
[0054] In this embodiment, the hand acquisition module 1 is a component with camera function, such as a CCD camera.
[0055] The standardized automatic handwashing assessment method and system works as follows:
[0056] First, the hand image captured during the handwashing process is transmitted from the hand image acquisition module 1 to the central processing unit 2;
[0057] The central processing unit transmits the image to the GPU image processing module 3, which outputs the interactive hand 3D pose 12 and transmits the interactive hand 3D pose 12 to the central processing unit 2.
[0058] The central processing unit 2 inputs the interactive hand 3D pose 12 into the intelligent evaluation module 4. The intelligent evaluation module 4 inputs the pose estimation action recognition sequence and score into the local storage module 6, and at the same time sends the image obtained by the hand acquisition module 1 and the final score into the display module 5 for display.
[0059] Existing hand posture estimation can be divided into three stages:
[0060] (1) Non-visual hand pose estimation is based on wearable external sensing devices, most commonly data gloves and color, but it has not been well developed due to the complexity of the devices and scenarios.
[0061] (2) The traditional machine learning method uses the random forest algorithm for deep images. With the development of deep learning technology, the shortcomings of the algorithm have been gradually exposed and it has been eliminated.
[0062] (3) The estimation algorithm using advanced deep learning algorithms has higher accuracy and has unparalleled advantages over the other two methods. It is the mainstream trend of current research.
[0063] The estimation algorithms that utilize advanced deep learning algorithms mainly fall into the following research directions:
[0064] (1) 3D hand pose estimation based on depth images or multi-view images. This method usually involves converting depth images or multi-view images into 3D space and performing pose estimation through 3D convolution, graph convolution, etc. However, this method has high equipment requirements, requiring a depth camera or multiple cameras to capture images simultaneously, and its applicable scenarios are limited.
[0065] (2) 3D hand pose estimation based on a single RGB image. This method usually uses CNN for feature extraction and obtains 3D coordinates by regression heatmap or direct regression. Due to the high dexterity of the hands and the problem of mutual occlusion between the hands, the accuracy is generally poor.
[0066] In this embodiment, Figure 2This is a schematic diagram of GPU image processing module 3. Figure 12 The diagram shows the process of RGB image processing by GPU image processing module 3. GPU image processing module 3 includes image preprocessing unit 7, CNN feature extraction unit 8, Transformer encoding unit 9 and decoding unit 10. Decoding unit 10 includes interactive hand enhancement subunit 11, which finally outputs interactive hand 3D pose 12.
[0067] The image preprocessing unit 7 is used to perform enhancement operations such as cropping and flipping on the image to obtain an image that meets the standard. For example, the image is cropped to obtain a 256×256 RGB image of a hand. The purpose of this operation is to make the subsequent processing network pay more attention to the hand area and filter out invalid background information.
[0068] In order to improve the processing efficiency of the image preprocessing unit 7, it is usually necessary to set the hand acquisition module 1 to a fixed state and set an effective acquisition area within the acquisition range of the hand acquisition module 1. The user needs to place their hands in the effective acquisition area to perform the hand washing operation.
[0069] Figure 4 This diagram illustrates the CNN feature extraction process 8. The image processed by the image preprocessing unit 7 is input to the CNN feature extraction unit 8 (CNN stands for Convolutional Neural Network). The CNN network consists of four parts. The first part includes a standardized 2D convolutional layer (Standardization Conv2d) with a kernel size of 7×7 and a stride of 2, which transforms the input data into 128×128×64 pixels. This is followed by a group normalization layer (Group Normalization Layer). The input data consists of a Normalization layer and a ReLU activation function layer, followed by MaxPool to obtain a 64×64×64 data stream. The second, third, and fourth parts are obtained by stacking three types of convolutional blocks 3, 6, and 9 times, respectively. Each convolutional block consists of three 2D convolutional layers with kernel sizes of 1×1, 3×3, and 1×1. The difference lies in the number of channels between each block: the first convolutional block has 64, 64, and 256 channels, the second block has 128, 128, and 512 channels, and the third block has 256, 256, and 1024 channels. After CNN feature extraction, the shape of the input data is changed from 256×256×3 to 16×16×1024.
[0070] Figure 5 shows a schematic diagram of the Transformer encoder 9. The data processed by the CNN feature extraction unit 8 is input into the Transformer encoding unit 9. A two-dimensional convolutional layer with a kernel of 1×1 and a stride of 1 is used to transform the data shape from 16×16×1024 to 16×16×768. The 16×16 features are stretched into a one-dimensional sequence to obtain a data stream of 256×768. Tokens are added to each sequence and concatenated to obtain a data stream of 257×768, which is then fed into a Transformer block stacked 12 times for encoding, without changing the shape.
[0071] Transformers, applied in the vision domain, can capture high-level global dependencies. However, their patch operations can destroy the internal structural information between image patches, and long-range attention mechanisms can ignore local properties of the image. For interactive hand pose estimation, the local features between the hands are a crucial issue, containing contextual information about joints and occluded joints. Addressing the unique challenges of interactive hand pose estimation, this application combines the advantages of CNNs in local image feature extraction with the advantages of Transformers in globally capturing long-distance spatial interactions between feature vectors across locations. A CNN network is used for local feature extraction, which is then fed into Transformer encoding unit 9 for encoding, thus solving the occlusion problem caused by the close interaction of hands during handwashing.
[0072] Figure 6 The diagram shows the decoder 10. The data processed by the Transformer encoding unit 9 is input into the decoding unit 10. The data output from the Transformer encoding is divided into three paths. One path is connected to a linear layer to predict the relative depth of the two hands to the root node. The other two paths are connected to two layers of deconvolution, upsampled to obtain left-hand and right-hand features with a shape of 64×64×256. The left-hand and right-hand features are then connected to the Interactive Hand Enhancement (IHE) module. After hand enhancement, the enhanced left-hand features are obtained. A 1×1 2D convolution is used to change the number of channels to 1344, i.e., 64×64×(64×21). The first two 64s are the predicted correlation results on the X and Y axes, the third 64 represents the predicted correlation result of the relative depth Z in the hand, and the 21 dimensions represent the 21 joints of each hand. This data is fed into a classifier to obtain a 1×1×1×21 data stream, i.e., the 2.5D (X-axis, Y-axis and relative depth Z in the hand) coordinates of the 21 joints of each hand. The 2.5D coordinates are added to the relative depth between the hands in the third dimension to obtain the 3D coordinates of each joint.
[0073] The interactive hand enhancement subunit 11 addresses the problem that when both hands are interacting closely, some key points are occluded by key points of the other hand, leading to significant ambiguity due to the self-similarity of the two hands. This application proposes an interactive hand enhancement subunit 11. Figure 7 The diagram shows the interactive hand feature enhancement module 11. It represents the two upsampled outputs as left-hand and right-hand features, using the left-hand features as the query and the right-hand object features as the key, to model the pairwise relationship between the left and right-hand features. As shown in the following formula, FL... i + This indicates the enhanced object feature, where the subscript i represents the given position i of the feature.
[0074]
[0075] Where Ω represents the set of all positions on the key. V is a value transformation function parameterized by Wv, and w(i, j) is the pairwise spatial similarity score between query position i and key position j.
[0076] The formula for calculating W(i, j) is:
[0077]
[0078] in , Let W represent the query and key at positions i and j, respectively. q W k W V All are 1×1 convolutions. This module allows for further differentiation of hand features, enhancing the feature set.
[0079] The decoding unit 10 and its embedded interactive hand enhancement subunit 11 jointly resolve the ambiguity caused by hand self-similarity, further enhancing hand features and determining which hand the features belong to. RGB images are 2D; the main challenge in estimating 3D pose from them is the ambiguity in depth coordinate estimation. Compared to directly estimating depth components, this application uses two relative depth components for prediction, thereby reducing prediction errors. This application estimates intra-hand relative depth and inter-hand relative depth to calculate global joint depth. Intra-hand relative depth refers to the depth of each joint in the palm relative to the root joint, while inter-hand relative depth refers to the depth of the right hand relative to the root joint of the left hand. Combining these two components yields the depth coordinate Z, thus completing the upgrade from 2D to 3D coordinates.
[0080] After decoding unit 10, the 3D joint coordinates of a single RGB image can be obtained, such as... Figures 8-11 As shown, where Figure 8 The image shown is an RGB image of hands facing each other with open arms. Figure 9yes Figure 8 A schematic diagram of the 3D joint coordinates output by the decoding unit 10; Figure 10 The image shown is an RGB image of hands crossed and facing each other. Figure 11 yes Figure 10 A schematic diagram of the 3D joint coordinates output by the decoding unit 10.
[0081] The model provided in this application achieves the best performance on the public dataset Interhand2.6M. On the Interhand2.6M-IH branch, compared with the baseline network internet's MPJPE index of 16.02, the model in this application achieves a score of 11.18, and the error is reduced by 30.2%.
[0082] Figure 3 The diagram shown is of the intelligent evaluation module 4. In the intelligent evaluation module 4, the 3D joint coordinates output by the decoding unit 10 are scored based on the standard matrix template.
[0083] The standard matrix template is stored in the local storage module 6, and the intelligent evaluation module 4 can call the standard matrix template. Since the entire handwashing action is dynamic, this application takes 5 standard action RGB images of the start, peak and process of each step to perform real three-dimensional joint point annotation, and integrates the annotation results into a matrix template after normalization by calculations such as cosine similarity and Euclidean distance. There are 5 standard matrix templates for each step, which are numbered such as 11~15 (first step of handwashing), 21~25 (second step of handwashing), etc.
[0084] The input to the intelligent evaluation module 4 is the 3D joint coordinates, that is, the 3D joint coordinates of the captured RGB image, which are used for scoring in this application.
[0085] According to the "Guidelines for Handwashing Techniques for Medical Personnel," each step of the standard seven-step handwashing method takes 10-15 seconds. This application pre-sets the handwashing time for each step to 15 seconds within the system. Timing begins at the start of handwashing, and within the 15 seconds of each step, the system sets the standard sequence to the current step number. Depending on the camera frame rate, this application uses sequences of different lengths as the standard sequence. The camera used in this application has a frame rate of 30 frames per second (M frames), and the standard sequence length is 20 (2 / 3M). For example, if the current step is the first step, the standard sequence consists of 20 ones. When the system starts running, it performs step recognition on the input 3D interactive hand gesture, determines which step the current action is, and stores the recognition results of the acquired image frames in the local sequence. If the current image does not show any hand (i.e., the hand coordinates in the current image are recognized as 0), the current frame image will be discarded and not saved in the local sequence. The system checks if the current sequence is 20 frames; if not, it waits until 20 frames are reached; if it exceeds 20, it divides the sequence into 20 frames, and the excess is accumulated to the next local sequence. Each template sequence is compared with a standard sequence. If they are the same as the standard sequence, the difference is recorded as 0; if they are different, the difference is recorded as 2. After calculating all 20 sequences, the sum of the differences is saved. If the difference is less than 10, the user is considered to have completed the target handwashing action within this time period, and 2 points are added to the score. The maximum score for each step is 20 points, and no further points are added after reaching the maximum score. This process is repeated to complete the entire handwashing process, and the score is then output to the display module.
[0086] The method for step recognition of the input 3D interactive hand pose is as follows: the 3D interactive hand pose is calculated in the same way as the standard matrix template, and a matrix is obtained. This matrix is compared with the standard matrix template and determined to be the matrix type with the highest similarity score. For example, if the similarity score with the standard matrix template 25 is the highest, then this application determines the result of this frame image as the second step of hand washing, and the stored local sequence number is 2.
[0087] More specifically, the 21 3D coordinates of the two hands are arranged into a matrix of shape [42,3] according to the keypoint order. After normalization, the cosine distance and Euclidean distance between the matrix and the standard template matrix are calculated. There are a total of 35 templates, and each template is traversed, requiring 35 calculations. The Euclidean distance reflects the absolute difference in value, while the cosine distance reflects the relative difference in direction. A total score is calculated by weighting the two distances with a 1:1 ratio, and the template category with the smallest calculated distance among the 35 templates is selected as the recognition result for that frame.
[0088] Calculate cosine distance .
[0089] Euclidean distance This is a three-dimensional space, i.e., N=3.
[0090] For example, if the distance to the standard matrix template 25 is the shortest, that is, the similarity score is the highest, then this application determines the result of this frame image as the second step of handwashing, and the stored local serial number is 2.
[0091] The results of this invention can be widely applied in scenarios such as hospital infection monitoring and standardized training, as well as hand-foot-mouth disease screening. For example, it can be used for monitoring standardized handwashing of medical staff in medical institutions, training and assessment of standardized handwashing in medical colleges, guiding children to wash their hands properly in primary and secondary schools, and hygiene supervision of catering establishments and food production enterprises.
[0092] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it.
Claims
1. A standardized automated handwashing assessment method, characterized in that, The method is as follows: Establish a standard matrix template and evaluation model for the handwashing process; Input the 3D interactive hand poses and standard matrix templates obtained during continuous handwashing into the evaluation model to score the continuous handwashing process; The method for obtaining 3D interactive hand poses is as follows: Acquire RGB images of the handwashing process and perform preprocessing; CNN feature extraction is performed on the preprocessed RGB image; The data after CNN feature extraction is then encoded using Transformer encoding; The Transformer-encoded data is decoded, and the output data is divided into three paths. One path is connected to a linear layer to predict the relative depth of the two hands to the root node. The other two paths are connected to two layers of deconvolution, and upsampled to obtain left-hand and right-hand features with a shape of 64×64×256. After the left-hand and right-hand features are connected to the interactive hand feature enhancement module, the enhanced left-hand features are obtained. A 2D convolution with a 1×1 kernel is used to change the number of channels to 1344, i.e., 64×64×64×21. The first two 64s are the prediction results on the X and Y axes, the third 64 represents the prediction results of the relative depth Z within the hand, and the 21 dimensions represent the 21 joints of each hand. This data is connected to a classifier to obtain a 1×1×1×21 data stream, i.e., the 2.5D coordinates of the 21 joints of each hand. The 2.5D coordinates are added to the relative depth between the hands in the third dimension to obtain the 3D coordinates of each joint. The feature enhancement method of the interactive hand feature enhancement module is as follows: The two upsampled outputs are represented as left-hand and right-hand features. The left-hand features are used as the query, and the right-hand object features are used as the key. The pairwise relationship between the left and right-hand features is modeled; the following formula is FL. i + Indicates the enhanced object characteristics in the output: Where Ω represents the set of all positions on the key, V is the value transformation function parameterized by Wv, and w(i, j) is the pairwise spatial similarity score between query position i and key position j; The formula for calculating W(i, j) is: in , W represents the query and key at positions i and j, respectively. q W k W V They are all 1×1 convolutions.
2. The standardized automatic handwashing assessment method according to claim 1, characterized in that, The standard matrix template is formed as follows: based on the standardized handwashing steps, take N standard action RGB images of the start, peak and process of each step to perform real three-dimensional joint point annotation, and integrate the annotation results into a matrix template after normalization. There are N standard matrix templates for each step.
3. The standardized automated handwashing assessment method according to claim 1, characterized in that, The evaluation model scores the continuous handwashing process as follows: Input a 3D interactive hand gesture obtained during any handwashing step, and determine which step in the handwashing process the 3D interactive hand gesture belongs to; Retrieve the standard matrix template corresponding to the handwashing step after the 3D interactive hand posture judgment, and compare and score it with the 3D interactive hand posture. The total score is the sum of all the handwashing steps.
4. The standardized automatic handwashing assessment method according to claim 3, characterized in that, The method for comparing and scoring the standard matrix template with the 3D interactive hand pose is as follows: Compare each 3D interactive hand gesture with all sequence frames in the standard matrix template. If the sequence frames in the 3D interactive hand gesture are the same as those in the standard matrix template, the difference is recorded as 0. If they are different, the difference is recorded as X. Sum the differences of all sequence frames and determine whether the difference exceeds the standard value Y. If it does not exceed the standard value, the handwashing step is determined to meet the standardized handwashing requirements, and the score is increased by P.
5. The standardized automated handwashing assessment method according to claim 3, characterized in that, The method for determining which step in the handwashing process a 3D interactive hand gesture belongs to is as follows: Arrange the three-dimensional coordinates of the 3D interactive hand gesture into a matrix according to the joint point order, and then normalize the matrix. Traverse the sequence frames of each standard matrix template, calculate the cosine distance and Euclidean distance between the normalized matrix and the standard matrix template respectively, and calculate the total distance by weighting the cosine distance and Euclidean distance with a 1:1 ratio. Select the handwashing step corresponding to the standard matrix template with the smallest total distance as the handwashing step to which the 3D interactive hand gesture belongs.
6. The standardized automated handwashing assessment method according to claim 1, characterized in that, The CNN network used for feature extraction is divided into four parts. The first part contains a normalized two-dimensional convolutional layer with a kernel size of 7×7 and a stride of 2 to transform the input data into 128×128×64. Then, it is connected to a group normalization layer and an activation function layer. After max pooling, a 64×64×64 data stream is obtained. The second, third, and fourth parts are obtained by stacking three types of convolutional blocks 3, 6, and 9 times, respectively. Each convolutional block consists of three two-dimensional convolutional layers with kernel sizes of 1×1, 3×3, and 1×1. The difference is that the number of channels between each block is different. The number of channels in the first convolutional block is 64, 64, and 256, the number of channels in the second block is 128, 128, and 512, and the number of channels in the third block is 256, 256, and 1024. After CNN feature extraction, the shape of the input data is transformed from 256×256×3 to 16×16×1024. The Transformer encoding of the data after CNN feature extraction includes: using a 2D convolutional layer with a kernel of 1×1 and a stride of 1 to transform the data shape from 16×16×1024 to 16×16×768; stretching the 16×16 features into a one-dimensional sequence to obtain a 256×768 data stream; adding a token to each sequence and concatenating them to obtain a 257×768 data stream; and feeding it into a Transformer block stacked 12 times for encoding, while maintaining the original shape.
7. A standardized automatic handwashing assessment system, characterized in that, The evaluation system is used to execute the standardized automated handwashing evaluation method according to any one of claims 1 to 6, and the evaluation system includes: A local storage module, wherein a standard matrix template is stored. The system includes an intelligent assessment module, which acquires 3D interactive hand gestures and standard matrix templates into the assessment model to score the continuous handwashing process.
8. The standardized automatic handwashing assessment system according to claim 7, characterized in that, Also includes: The hand acquisition module is used to acquire RGB images of the handwashing process; Central processing unit, wherein the central processing unit has an embedded intelligent evaluation module; The GPU image processing module is electrically connected to the central processing unit and is used to extract 3D interactive hand gestures from the acquired RGB images.
9. The standardized automatic handwashing assessment system according to claim 7, characterized in that, It also includes a display module for showing the scores generated by the intelligent evaluation module.