Apparatus, method and storage medium for monitoring hand-washing processes

Abstract

To provide a hand-washing process monitoring device, a method, and a storage medium.SOLUTION: The hand-washing process monitoring device includes a memory in which instructions are stored, and a processor that can execute instructions acquired from the memory so that hand-washing processes can be monitored by running multiple modules in parallel. The multiple modules include a classification module that determines motion type attributes of selected two-handed images in a two-handed image sequence generated based on the hand-washing processes, with reference to a set of reference hand-washing motions, and a counting module that determines the number of times a selected reference hand-washing motion has been performed, based on the selected reference hand-washing motion, the two-handed image sequence, and the determined motion type attributes of the two-handed images in the two-handed image sequence. According to the method, device, and storage medium of the present disclosure, at least the execution quality of the hand-washing processes of a hand-washing person can be comprehensively and accurately evaluated.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to image processing, and more particularly to an apparatus, method and storage medium for monitoring a hand-washing process. [Background technology]

[0002] Accurate completion of the standard hand-washing process plays an important role in food safety, hygiene, and health management. Traditionally, inspecting the hand-washing process is done manually. This method is time-consuming and labor-intensive, as inspectors must monitor the hand-washer's entire hand-washing process. Summary of the Invention [Problem to be solved by the invention]

[0003] The following presents a simplified summary of the disclosure in order to provide a basic understanding of aspects of the disclosure. However, this summary is not an exhaustive overview of the disclosure, and it is not intended to identify key or important portions of the disclosure or to limit the scope of the disclosure. Rather, it is intended to merely introduce concepts in a simplified form as a prelude to the more detailed description that is presented later.

[0004] With the development of computer technology, especially image processing technology, it has become possible to monitor handwashing using a computer. For example, by detecting and recording the start and end times of handwashing, it is possible to determine whether the handwashing process is adequate. However, it is desirable to monitor, evaluate, and guide the handwashing process by taking more factors into consideration. For this reason, the inventor of the present invention devised the present invention. The inventor discovered at least the following: A standard handwashing process includes various standard actions. Each action needs to be performed at least a predetermined number of times. It is desirable to comprehensively, accurately, and real-timely evaluate the handwashing quality of a handwasher. It is desirable to provide real-time feedback and / or guidance to the handwasher. [Means for solving the problem]

[0005] One aspect of the present disclosure provides an apparatus for monitoring a hand washing process, the apparatus including: a memory having instructions stored therein; and a processor capable of executing the instructions retrieved from the memory to monitor the hand washing process by executing a plurality of modules in parallel, the plurality of modules including: a classification module that determines, based on a set of reference hand washing actions, action type attributes of selected hand-washing images in a sequence of hand-washing images generated based on the hand washing process; and a counting module that determines the number of times the selected reference hand washing action has been performed, based on the selected reference hand washing action, the sequence of hand-washing images, and the determined action type attributes of the hand-washing images in the sequence of hand-washing images.

[0006] Another aspect of the present disclosure provides a method for monitoring a hand washing process by executing a plurality of modules in parallel, the plurality of modules including: a classification module that determines, based on a set of reference hand washing actions, action-type attributes of selected hand-washing images in a sequence of hand-washing images generated based on the hand washing process; and a counting module that determines a number of times the selected reference hand-washing action has been performed, based on the selected reference hand-washing action, the sequence of hand-washing images, and the determined action-type attributes of the hand-washing images in the sequence of hand-washing images.

[0007] Another aspect of the present disclosure provides a computer-readable storage medium having a program stored thereon, the program, when executed by a processor, monitoring a hand washing process by executing multiple modules in parallel, the multiple modules including: a classification module that determines, based on a set of reference hand washing actions, action-type attributes of selected hand-washing images in a sequence of hand-washing images generated based on the hand washing process; and a counting module that determines a number of times the selected reference hand-washing action has been performed, based on the selected reference hand-washing action, the sequence of hand-washing images, and the determined action-type attributes of the hand-washing images in the sequence of hand-washing images.

[0008] The method, device, and storage medium of the present disclosure can achieve at least one of the following effects: The execution quality of the hand-washing process of a hand-washing user can be comprehensively and accurately evaluated. [Brief explanation of the drawings]

[0009] In order to make the above and other objects, features, and advantages of the present disclosure more easily understandable, the following describes embodiments of the present disclosure with reference to the drawings. It should be noted that the drawings are merely for illustrating the principles of the present disclosure. The drawings do not necessarily depict the size and relative positions of each part according to scale. The same reference numerals may represent the same features. [Figure 1] FIG. 1 illustrates an example of a standard hand washing process. [Figure 2] FIG. 1 illustrates a device for monitoring a hand washing process according to one embodiment of the present disclosure. [Figure 3] FIG. 1 illustrates multiple modules for monitoring a hand washing process according to one embodiment of the present disclosure. [Figure 4] FIG. 2 illustrates an exemplary graphical user interface according to one embodiment of the present disclosure. [Figure 5] FIG. 1 illustrates a device for monitoring a hand washing process according to one embodiment of the present disclosure. [Figure 6] FIG. 1 illustrates a device for monitoring a hand washing process according to one embodiment of the present disclosure. [Figure 7] 1 is an exemplary flow chart illustrating a method for monitoring a hand-washing process according to the present disclosure. [Figure 8] 1 is an exemplary flow chart illustrating a method for monitoring a hand-washing process according to the present disclosure. [Figure 9] 10 is an exemplary flowchart illustrating processing performed by a classification module according to the present disclosure. [Figure 10] 1 is an exemplary block diagram illustrating an information processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] The following describes exemplary embodiments of the present disclosure with reference to the drawings. For the sake of convenience, the specification does not show all features of the actual embodiments. However, when implementing the embodiments, those skilled in the art may make certain decisions to implement the embodiments, and these decisions may vary depending on the embodiment.

[0011] It should be noted that, for clarity of the present disclosure, the drawings only show device components and / or process steps closely related to the present disclosure, and omit details unrelated to the present disclosure.

[0012] It should be noted that the present disclosure is not limited to the described embodiments, as will be described below with reference to the accompanying drawings. In this specification, where feasible, the embodiments may be combined with each other, features of different embodiments may be substituted or utilized, or one or more features may be omitted in one embodiment.

[0013] The hand-washing process monitoring method or device of the present disclosure can evaluate and guide the actual hand-washing process based on the reference hand-washing process, classify the action types of the images of both hands, and count the hand-washing actions.

[0014] To monitor the hand-washing process, the inventor analyzed a reference hand-washing process. The following describes the reference hand-washing process.

[0015] A standard handwashing process (which may be referred to as a "reference handwashing process") may be divided into sub-processes for performing several handwashing actions. For example, a "seven-step handwashing method" includes rubbing palms with palms, rubbing backs of hands with palms, clasping hands together, rubbing backs of fingers, rubbing thumbs, rubbing fingertips, and rubbing wrists. Each of the above steps needs to be repeated several times, and some steps (e.g., rubbing thumbs) need to be repeated several times for both the left and right hands. The handwashing process may also include additional steps, such as applying soap. In the present disclosure, the reference handwashing process is divided into kmax reference sub-processes. For example, a "seven-step handwashing method" with "applying soap" added is divided into 14 sub-processes, and similar handwashing actions performed on the left and right hands are divided into two sub-processes. Each reference sub-process Pk includes a corresponding reference hand-washing action Ak (e.g., a standard action for rubbing the thumb of the left hand), where k=1, 2, ..., kmax, where k is an index for distinguishing each reference hand-washing action. Each reference hand-washing action Ak includes multiple reference hand-washing action states Ak.S[n] (e.g., states corresponding to each video frame of a video clip within the period of the corresponding standard action for rubbing the thumb of the left hand). Execution of the reference hand-washing sub-process Pk includes executing the reference hand-washing action Ak at least a predetermined number of times Ak.PreCount (i.e., a desired number of executions), for example, once, five times, etc.

[0016] FIG. 1 illustrates an example of a standard hand-washing process. This exemplary standard hand-washing process includes 14 standard subprocesses P1 through P14. The total time required to execute the standard subprocesses P1 through P14 is T. Each standard subprocess Pk includes a corresponding standard hand-washing action Ak, where k = 1, 2, ..., 14. As shown in this figure, in the second standard subprocess P2, the standard hand-washing action A2 is executed five times (i.e., P2 has five execution cycles, and the same standard hand-washing action A2 is executed in each cycle). The five standard hand-washing actions A2 may be referred to as the first action P2.A2[1], the second action P2.A2[2], the third action P2.A2[3], the fourth action P2.A2[4], and the fifth action P2.A2[5]. A2.PreCount = 5. 1 further exemplarily illustrates nmax reference handwashing action states A2.S[n] included in the second action P2.A2[1], where the nmax reference handwashing action states correspond to different hand configurations. Note that, according to a predetermined rule, in an actual handwashing process, the handwashing actions may not be performed in the order of the sub-processes shown in the reference handwashing process. To ensure the quality of handwashing, preferably, in an actual handwashing process, each reference handwashing action Ak is performed at least a predetermined number of times Ak.PreCount.

[0017] The inventors discovered that when using image processing techniques to classify (i.e., recognize) video images (also referred to as "video frames" or "frames," corresponding to handwashing action states Ak.S[n]) in a reference handwashing process based on a set of reference handwashing actions, for example, to determine the action type attribute of an image corresponding to a handwashing action state as Ak, the recognition level of each handwashing action state varies during the reference handwashing action period. That is, the recognition level of video frame positions (also referred to as "frame positions" for short, e.g., time t corresponding to state A2.S[n] in period t0) corresponding to different time points may differ, and there may be video frame positions with low recognition levels and video frame positions with high recognition levels. When monitoring the handwashing process, it is desirable to accurately determine the time point at which a handwashing action is switched (e.g., from reference handwashing action Ak to Ak') for accurate counting. If a new handwashing action is determined to have occurred at a video frame position corresponding to a low recognition level, the determination result may be inaccurate, and therefore the classification result may be adjusted.

[0018] In one aspect of the present disclosure, a hand-washing process monitoring device is provided. The device may determine an action type corresponding to an image related to hand-washing, determine the number of times each predetermined hand-washing action has been performed, and determine notification information to notify a user of the hand-washing process. The following is an exemplary description of the hand-washing process monitoring device of the present disclosure with reference to FIG. 2.

[0019] FIG. 2 illustrates a hand-washing process monitoring device 200 according to one embodiment of the present disclosure. The device 200 includes a memory 201 and a processor 203. The memory 201 stores instructions. The processor 203 may be a multi-core processor and thus may execute multiple processes in parallel. The processor 203 may execute instructions retrieved from the memory 201 to monitor the hand-washing process by executing multiple modules in parallel, preferably in real time or near real time. The multiple modules may include a classification module and a counting module. The classification module determines an action type attribute Is[j].Cx (for convenience of explanation, sometimes referred to as the "action type attribute Cx") ​​of a selected hand-washing image Is[j] in a hand-washing image sequence Sh generated based on a hand-washing process based on a reference hand-washing action set Sa (Sa={A1, A2, ..., Ak, ..., Akmax}). The counting module determines a number Ak.Count of times the selected reference handwashing action Ak has been performed based on the selected reference handwashing action Ak, the sequence of hand images Sh, and the determined action type attributes of the hand images in the sequence of hand images.

[0020] FIG. 3 illustrates multiple modules for monitoring a handwashing process according to one embodiment of the present disclosure, namely, an update module 301, a classification module 303, a counting module 305, and a presentation module 307. Here, the update module 301 and the presentation module 307 are optional modules. In the absence of an update module, the classification module and the counting module may read a data storage location of a photographing device to obtain necessary image data. For example, the sampled image sequence of the photographing device may be a sequence of images of both hands. The classification module and the counting module may read a buffer unit storing the sampled image sequence and perform predetermined processing based on the sampled image sequence. If the classification and / or counting results do not need to be presented to the handwashing user, the classification and / or counting results may be stored for later review. Because at least these modules have different processing speeds, at least the classification module and the counting module are executed in parallel to ensure the quality of monitoring (e.g., classification accuracy, counting accuracy, and real-time processing results) and reasonable hardware costs.

[0021] The following describes the update module of the present disclosure.

[0022] The update module 301 updates the sequence of images of both hands Sh based on images of the handwashing process of the person washing their hands captured in real time by a camera. The update module 301 receives images from the camera. The received images are referred to as images Iri, where i is an index for distinguishing images. The camera is aimed at a predetermined position, such as a sink. The lens of the camera may be fixed or rotatable to track, align, and focus the moving hands of the person washing their hands. Each captured image Iri has a time attribute Iri.Time indicating the time of capture. The fixed position of the camera and the operating parameters of the camera may be selected to ensure that the captured images are sufficiently clear for subsequent classification and counting. The device 200 may include a camera. The camera may detect a trigger for capturing an image, such as a specific gesture, a specific action, a click of a command button, or a specific voice from the person washing their hands. The handwashing process monitoring method of the present disclosure may also start after detecting a specific gesture, a specific action, a click of a command button, or a specific voice from the person washing their hands.

[0023] The update module 301 determines whether the image Iri is a two-hand image. If the determination result is "YES," the two-hand image sequence Sh is updated. For example, if both hands are detected or recognized in the image Iri, the image Iri is determined to be a two-hand image. Alternatively, for simplicity, if the image Iri is different from a background image without both hands, the image Iri is determined to be a two-hand image. Whether the image Iri is a two-hand image may be determined using a neural network image recognition model or other image recognition technology. Preferably, if the determination result is "NO," the update module 301 may determine whether to terminate the process based on a predetermined condition. For example, the process is terminated only if the number of consecutive occurrences of a "NO" determination result reaches a threshold. The predetermined condition may be a specific gesture, a specific movement, the click of a command button, a specific voice, etc., of the person washing the hands. Preferably, the update module 301 may set a hand-washing process completion flag (abbreviated as "completion flag") based on the received information about the hand-washing process. For example, if the information regarding the hand washing process indicates that the time that both hands have been out of the imaging area has reached a predetermined time, the hand washing process completion flag is set to "true."

[0024] Before updating, the two hand image sequence Sh={Is[1], Is[2], ..., Is[je-1]}, where Is[je-1] is the latest two hand image in the image sequence Sh before updating. Preferably, during updating, the number of images included in the image sequence Sh becomes je after adding 1. Preferably, the newly received two hand images may be directly added to the image sequence Sh without changing the newly received two hand images (i.e., Iri is set to Is[je]). Preferably, the received two hand images may be cropped and added to the image sequence Sh (i.e., the cropped Iri is set to Is[je]). Preferably, a segmentation process is performed on the received two hand images, so that the image portion including the hands of the person washing their hands is in the foreground of the two hand image Is[je]. Furthermore, the image sequence Sh may be updated periodically, or the cycle for updating the image sequence Sh may be determined based on the currently determined movement type.

[0025] Generally, the hand-washing process only requires monitoring the hand movements. For example, a segmentation process may be performed on the received hand image to improve the speed and / or accuracy of subsequent classification and counting. The segmentation process may be used to update the hand image sequence. An image portion (i.e., an image portion having a non-rectangular, irregular shape) containing the hand of the person washing their hands is extracted from the image as the foreground of an intermediate (middle) hand image, the background of the intermediate hand image is set to a single color (e.g., black), and the intermediate hand image is added to the hand image sequence to update the hand image sequence. For example, the image portion containing the hand of the person washing their hands is extracted from the received image based on skin color. For example, a deep learning model may be used to extract the image portion containing the hand of the person washing their hands from the received image.

[0026] Preferably, the images in the sequence of two hands images are ordered by time of capture.

[0027] The capture time of the received image may be set to the capture time of the corresponding image in the sequence of two-hand images.

[0028] To periodically update the image sequence Sh regardless of the currently determined motion type, for example, the image sequence Sh is updated based on the image Iri only when Iri.Time - Is[je-1].Time is greater than a predetermined value. Here, Iri is a newly received image determined to be a two-hand image, Is[je-1] is the most recent two-hand image in the image sequence Sh, and Is[je-1].Time is the capture time attribute of the two-hand image Is[je-1]. Is[je-1].Time may be determined based on the corresponding received image. For example, for every m received images, one image from the image sequence of received images is selected as the two-hand image Is[je] to be added to the image sequence Sh, where m is 1 or greater.

[0029] In one example, the interval for updating the image sequence Sh may be determined based on the currently determined action type Is[j].Cx (e.g., Is[j].Cx = Ak), and the interval may be used to determine whether to update the image sequence next time. That is, different intervals for updating the image sequence may be adopted depending on different hand-washing action type attributes. Preferably, when a new type of hand-washing action type attribute is determined (i.e., when switching between different hand-washing actions), the interval for updating the image sequence may be shortened to improve the accuracy of action classification. For example, when the action type attribute of the previous hand-washing image is determined to be Ak' and the action type attribute of the current hand-washing image is determined to be Ak different from Ak', the interval for updating the image sequence may be shortened, and the image sequence may be updated at the shortened interval for a predetermined period until the classification result "Ak" is repeated a predetermined number of times, and then the default update interval may be restored. That is, the update module and the classification module may operate cooperatively, and the classification result of the classification module may affect the processing parameters of the update module.

[0030] In one example, if the background of the received image is simple and the size is appropriate, the newly received two-hand image may be added directly to the image sequence Sh without correction, and the image sequence Sh may be updated.

[0031] In one example, to improve the speed and / or accuracy of subsequent classification and counting, the received hand image may be cropped and added to the image sequence Sh. Preferably, the cropped image may have less background. For example, a rectangular frame that surrounds the hand of the person washing their hands in the received image may be used to crop the image portion containing the hand of the person washing their hands from the image, and the image portion may be used as the hand image Is[je] in the image sequence Sh to update the image sequence Sh.

[0032] If hardware cost and processing speed requirements are met, the two hand image sequences may preferably be updated more frequently, i.e., so that the number of images per unit time in the two hand image sequences is greater than a predetermined threshold.

[0033] The following describes the classification module of the present disclosure.

[0034] If the cost and processing speed conditions are met, the classification module may determine an action type attribute for each hand image in the sequence of hand images Sh based on the reference hand washing action set, for example, including a non-reference hand washing action and a reference hand washing action in the reference hand washing action set Sa.

[0035] In one example, the classification module determines the action type attribute of the selected two hand images in the two hand image sequence generated based on the hand washing process based on the reference hand washing action set Sa. In one example, the classification module 303 selects one image from the two hand image sequence Sh as a selected two hand image for every d two hand images, where d is equal to or greater than 1. In other words, the two hand image Is[j] may be selected and classified only if the index j of the newly added two hand image Is[j] in the two hand image sequence Sh satisfies the condition that j+1 is an integer multiple of (d+1). (That is, the action type attribute Is[j].Cx of only some two hand images Is[j] in the two hand image sequence Sh is not 0, and the default value of Cx is 0, meaning that no classification has been performed on the two hand image Is[j]. For example, different numerical values ​​may be used to identify different action types.) Alternatively, if a new two hand image is added to the two hand image sequence Sh, classification is performed on the newly added two hand images. d may be related to the determined action type attribute of the hand washing action performed by the current hand washer. For example, if a longer reference time is required to perform one action of the action type attribute, d may be a large value. Possible example values ​​of the action type attribute Cx include -1 (indicating that the action type is a non-reference hand washing action), 0 (default value (value set at initialization), indicating that the image has not been classified or that a classification decision process has not been performed on the image), 1 (the corresponding action type is the reference hand washing action A1), 2 (the corresponding action type is the reference hand washing action A2), ..., k (the corresponding action type is the reference hand washing action Ak), ... kmax (the corresponding action type is the reference hand washing action Akmax).

[0036] In one embodiment, the classification module may determine the action type attribute Is[j].Cx of the selected hand image Is[j] in the hand image sequence Sh using a deep learning model of a frame-based image (e.g., the selected hand image Is[j]) or a video clip (e.g., the selected hand image Is[j] and several images before Is[j] in the hand image sequence Sh).

[0037] Preferably, the classification module further determines a classification confidence Is[j].Conf of the determined action type attribute Is[j].Cx of the selected two-hand image Is[j]. Different predetermined thresholds may be set according to different action type attributes. Considering representative state images corresponding to multiple handwashing action states Ak.S[n] included in the reference handwashing action Ak, when classification is performed using the classification module, the classification confidence varies (i.e., the classification of some state images can be determined relatively accurately, but the classification of other state images may be erroneously determined). Therefore, when the classification module determines that the determined action type attribute of the selected two-hand image Is[j] is different from the action type attribute determined in the previous classification determination step, if the classification confidence Is[j].Conf is equal to or greater than the predetermined threshold, the classification module maintains the determined action type attribute; otherwise, the classification module adjusts the action type attribute of the selected two-hand image Is[j] from the current action type attribute to the action type attribute determined in the previous action type determination step.

[0038] In one example, when the classification module determines that the frame position of the current hand image, whose action type attribute is the new action type, corresponds to a video frame position with low recognition level, the classification module adjusts the action type attribute of the current hand image to the action type attribute determined by the previous action type determination process, where the video frame position with low recognition level corresponds to a video frame position with a recognition level lower than a predetermined recognition threshold in the execution cycle of the reference hand washing action of the previous action type attribute.

[0039] If the classification module determines that there is an abnormal hand image with abnormal action type attributes in the hand image sequence, the classification module may adjust the action type attributes of the abnormal hand image to the action type determined by the current action type determination process. For example, if the action type attribute sequence determined by the classification module in time series includes Ak, Ak, Ak, Ak', Ak, and Ak, and the image corresponding to Ak' is an abnormal hand image, the classification module may adjust the action type attributes corresponding to the abnormal image to Ak. Furthermore, the counting module may be triggered to count the number of times the reference action type Ak has been performed based on the adjusted action type attributes of the hand images in the hand image sequence. That is, the classification module and the counting module may be configured to operate cooperatively, and the intermediate results of one module may be configured to enable the other module to achieve better performance. For example, the intermediate results of one module may be used to determine the action parameters of the other module.

[0040] The classification module 303 may be further configured to determine whether the action type attribute Is[j].Cx of the selected two-hand image Is[j] belongs to the reference hand-washing action set Sa. If the determination result is "NO", the classification module 303 transmits first information (message) to the presentation module. The first information may be information indicating that the action performed by the hand-washing user is a non-reference hand-washing action.

[0041] The classification module 303 may further be configured to determine whether to adjust the action type attributes of the two hand images in the two hand image sequence, and if the result of the determination is "NO", return to the step of selecting the two hand images, and if the result of the determination is "YES", adjust the action type attributes.

[0042] The classification module may be configured to determine the selected hand images based on the period of the current action type determined by the counting module. For example, if the counting module determines that the currently performed hand washing action is a reference hand washing action Ak and determines that the reference hand washing action Ak will continue to be performed based on the number of times it is performed, the classification module may estimate the time point of a high-recognition frame based on the period of the current hand washing action estimated by the counting module, and select an image from the sequence of hand washing images whose shooting time attribute is close to the estimated time point as the selected hand images. Preferably, if the classification module detects a new reference hand washing action at a position (time point) of a high-recognition image where a previously determined reference hand washing action is predicted to appear, the classification module may maintain the action type attribute corresponding to the new reference hand washing action (i.e., not adjust the classification result). On the other hand, if the classification module detects a new reference hand washing action at a position (time point) of a low-recognition image where a previously determined reference hand washing action is predicted to appear, the classification module may ignore the action type attribute corresponding to the new reference hand washing action (i.e., adjust the classification result).

[0043] The following describes the counting module of the present disclosure.

[0044] The counting module determines the number of times the selected reference hand washing action has been performed based on the selected image sequence segment. Specifically, based on the selected image sequence segment in the two-hand image sequence Sh, it determines the number of times Ak.Count, where Is[j].Cx=Ak, the corresponding reference hand washing action Ak has been performed. The selected image sequence segment may be determined in the following manner: The selected image sequence segment is determined based on the two-hand image that first appears in the two-hand image sequence Sh and whose action type attribute is the target reference hand washing action. Here, the action type attribute of the selected two-hand image is the target reference hand washing action. Furthermore, the counting module may determine the selected image sequence segment based on a recommendation from the classification module (e.g., the classification module may recommend the selected image sequence segment based on the action type attribute and classification confidence, etc.). For example, in a hand image sequence Sh, if the first hand image with an action type attribute Ak appears as Is[j'], images Is[j'] through Is[j] may be selected as the selected image sequence segment, or images Is[j'-δ] through Is[j] may be selected as the selected image sequence segment, where δ is a predetermined integer. If j'-δ is less than 1, the starting image of the selected image sequence segment may be set to Is[j']. For example, the number of times the corresponding hand washing action Ak has been performed may be determined based on the similarity of the images. Furthermore, the number of times the corresponding hand washing action has been performed may be determined based on the number of substantially identical non-adjacent images in the selected image sequence segment. Here, if the absolute value of the difference in capture time between non-adjacent images is greater than a predetermined time threshold and the similarity between the two images is both greater than a predetermined threshold, the two images are determined to be substantially identical. Alternatively, if the similarity between the two images and the reference hand washing image is both greater than a predetermined threshold, the two images are determined to be substantially identical. Here, preferably, the reference hand-washing image is an image corresponding to the hand-washing action state Ak.S[n'] that is easiest to recognize among the hand-washing action states of the reference hand-washing action Ak. In one modified example, the number of times Ak.Count of the corresponding hand-washing action Ak has been performed may be determined based on the cycle of performing the corresponding reference hand-washing action once.The counting module may further record and update the start time and end time of the hand washing sub-process corresponding to the corresponding hand washing action Ak. Here, the start time means the start time of starting to perform the first hand washing action Ak, and the end time means the completion time of performing the last hand washing action Ak. The completion time may be the time when a hand washing image of a new hand washing action different from the hand washing action Ak appears. The counting module may count the time of the entire hand washing process to evaluate the quality of the hand washing process or to provide information to the user.

[0045] The counting module 305 may transmit second information (message) to the presentation module 307. The second information indicates the number of times the determined standard hand-washing action has been performed.

[0046] The counting module 305 may determine whether the completion flag is "true". If the determination result is "YES", it transmits third information (message) to the presentation module. The third information indicates that the hand washing process is completed. This allows the presentation module to display overall evaluation information of the hand washing process according to the information. If the determination result is "NO", it returns to the step of selecting an image sequence segment. The counting module may set the completion flag to "true" if the number of times the last reference hand washing action in the hand washing process has been performed is equal to or greater than the expected value.

[0047] To improve the counting speed, the counting module may select an image sequence segment that has not been counted before as the selected image sequence segment. The counting module may determine the number of times the selected reference hand-washing action has been performed based on the selected image sequence fragment. That is, for sub-processes performing the same reference hand-washing action, counting may be performed for each segment, and the cumulative value may be determined as the number of times the reference hand-washing action has been performed.

[0048] The following describes the presentation module of the present disclosure.

[0049] The presentation module 307 presents notification information to the handwashing user based at least on the number of times the handwashing action has been performed. Specifically, the presentation module 307 determines notification information Noti to notify the handwashing user based at least on the number of times Ak.Count has been performed, and causes the notification unit to present the notification information Noti. For example, the notification information includes the number of times Ak.Count the handwashing action Ak has been performed. In one embodiment, the notification information Noti includes an evaluation of the handwashing action of the handwashing user. The evaluation may include at least one of a determination of whether the handwashing action performed by the handwashing user is correct or not and the accuracy of the handwashing action performed by the handwashing user. The accuracy and precision may be determined by a classification module or a counting module, where the accuracy and precision of the handwashing action performed by the handwashing user may be determined based on a reference handwashing process. The notification information Noti may include the expected number of times Ak.PreCount the corresponding handwashing action has been performed. If it is determined that the action type of the selected hand-washing image does not belong to the set of reference handwashing action types, the notification information includes display information indicating that the handwashing action of the handwashing user is incorrect. Furthermore, the notification information further includes instruction information regarding the correct handwashing action. The presentation module may be further configured to determine overall evaluation information for evaluating the overall handwashing quality to be notified to the handwasher when it is determined that the handwasher has completed the handwashing process. For example, the presentation module may determine that the handwasher has completed the handwashing process when the handwasher has completed all sub-processes of the standard handwashing process. Here, for each sub-process of the standard handwashing process, the number of standard handwashing actions corresponding to the sub-process performed by the handwasher is equal to or greater than a predetermined number. Preferably, when the handwasher has completed all sub-processes of the standard handwashing process, information regarding the total time required for the handwashing process may be added to the notification message. The total time required for the handwashing process may be provided by the update module. The presentation module may cause the presentation unit to present the notification information in the form of light, sound, a still image, a video, or a combination thereof.

[0050] FIG. 4 is a diagram illustrating an exemplary graphical user interface according to one embodiment of the present disclosure, showing various information related to the handwashing process of a handwashing user. Ak.Des is an attribute describing the handwashing action Ak. "Score" indicates the execution quality of the handwashing action Ak, and "Status" indicates whether the handwashing action is "Completed," "In Progress," or "Incomplete." "Overall Evaluation" is overall evaluation information indicating the overall handwashing quality. This may be displayed when the handwashing user's handwashing process is determined to be complete, and may preferably be updated according to the progress of the handwashing process. FIG. 4 does not show representative images of the reference actions, but only shows labels (Im1 to Im14) of the representative images. In an actual display, each label is replaced with a corresponding image. To make it easier for the handwashing user to view the notification information, a portion of the graphical user interface may be displayed within the screen range of the display, displaying information about at least the currently executed action. When switching between handwashing actions, the content displayed in the graphical user interface scrolls. To improve the user experience, the graphical user interface may also be configured to have a function for interaction with the handwashing user.

[0051] FIG. 5 is a diagram illustrating a hand-washing process monitoring device 500 according to one embodiment of the present disclosure. The device 500 in FIG. 5 includes components similar to those of the device 200 in FIG. 2, and a description of these components will be omitted here. The device 500 also includes a presentation unit 505. The presentation unit 505 receives a presentation command sent by the processor 203 and presents notification information (Noti) to the person washing their hands. The presentation unit 505 may present the notification information in the form of light, sound, a still image, a video, or a combination thereof. The presentation unit may include a display. The display may present a graphical user interface (GUI) according to the present disclosure.

[0052] The hand-washing process monitoring device of the present disclosure may include a photographing device. FIG. 6 is a diagram illustrating a hand-washing process monitoring device 600 according to one embodiment of the present disclosure. The device 600 includes the components shown in FIG. 5 and further includes a photographing device 606. The photographing device 606 is used to capture images. In operation, the photographing device 606 is aimed at a hand-washing position, thereby capturing an image of the person washing their hands. The captured image may be processed by the processor 203 and then stored in the memory 201 as an element of a hand-washing image sequence Sh for subsequent use.

[0053] Alternatively, the image capture device may not be connected to the processor. The data sampled by the image capture device is stored in a predetermined buffer, and the update module, classification module, and / or counting module read the buffer to perform predetermined processing. The connection between the image capture device and the buffer may be a remote connection, a wireless connection, etc.

[0054] One aspect of the present disclosure provides a method for monitoring a hand-washing process. The following description will be given with reference to FIG. 7. FIG. 7 is an exemplary flowchart illustrating a method 70 for monitoring a hand-washing process according to the present disclosure. The method 70 includes monitoring a hand-washing process by executing multiple modules in parallel. FIG. 7 illustrates the parallel execution of a classification module and a counting module (shown as steps S701 and S703, respectively). The method 70 may be configured to start the parallel execution of the classification module and the counting module in response to a voice start command from a hand-washing user. The classification module determines, based on a set of reference hand-washing actions, action-type attributes of selected hand-hand images in a hand-hand image sequence generated based on the hand-washing process. The counting module determines the number of times the selected reference hand-washing action has been performed based on the selected reference hand-washing action, the hand-hand image sequence, and the determined action-type attributes of the hand-hand images in the hand-hand image sequence. Starting the operation of the counting module and the classification module includes an operation of initializing execution parameters. The method 70 may be configured to receive a status of the parallel execution of the classification module and the counting module in response to a user's voice end command or a determination that the completion flag is "true." The method 70 may include more steps. The method 70 and the apparatus 200 have a corresponding relationship. For detailed descriptions of other steps, the counting module, and the classification module, reference may be made to the above description of the apparatus 200.

[0055] 8 is an exemplary flowchart illustrating a method 80 for monitoring a hand-washing process according to the present disclosure. More processing steps are shown compared to FIG. 7. The method 80 may be configured to initiate the execution of an update module, a classification module, a counting module, and a presentation module in parallel in response to a voice command from a hand-washing user. The start of execution of each module includes initializing execution parameters (see steps S100, S300, S500, and S700).

[0056] An exemplary operation flow of the update module includes steps S100 to S107. In step S101, an image Iri is received from the image capture device. In step S103, it is determined whether the image Iri is a two-hand image. If the determination result is "YES," the two-hand image sequence Sh is updated (step S105). If the determination result is "NO," it is determined whether to end the method 80 (step S107). In step S107, it may be determined whether to end the method 80 based on a predetermined condition. For example, the method 80 is ended only if the number of consecutive occurrences of the determination result "NO" in step S103 reaches a threshold. The predetermined condition may be that a specific gesture or a specific movement of the person washing their hands is detected. The operation flow of the update module may include a waiting step, and the waiting step may be ended depending on a specific condition.

[0057] An exemplary operation flow of the classification module includes steps S300 to S311. In step S301, a pair of hand images is selected from a pair of hand image sequence, for example, the last pair of hand images in the pair of hand image sequence is always selected. In step S303, a motion type attribute Cx of the selected pair of hand images is determined based on a reference hand washing motion set. In step S305, it is determined whether the motion type attribute Cx belongs to the reference hand washing motion set. If the determination result is "YES," it is determined whether to adjust Cx (step S307). If the determination result is "NO," first information (message) is transmitted to the presentation module (step S311). The first information may be information indicating that the motion performed by the hand washing user is a non-reference hand washing motion. If the determination result in step S307 is "YES," step S309 is executed, i.e., the motion type attribute is adjusted, and Cx may be adjusted based on the current video frame position, the current classification confidence, and / or the previous classification confidence. If the determination result in step S307 is "NO," the process returns to step S301. The operational flow of the classification module may include a waiting step, and may terminate the waiting step depending on a specific condition.

[0058] An exemplary operation flow of the counting module includes steps S500 to S509. In step S501, an image sequence segment is selected based on a reference hand-washing action. In step S503, the number of times the selected reference hand-washing action has been performed is determined based on the selected image sequence segment. In step S505, second information is transmitted to the presentation module. The second information indicates the number of times the determined reference hand-washing action has been performed. In step S507, it is determined whether the completion flag is "true." If the determination result is "yes," third information (message) is transmitted to the presentation module (step S509). The third information indicates that the hand-washing process has been completed. This allows the presentation module to display overall evaluation information about the hand-washing process in response to the information. Preferably, a "wait" step is set between step S509 and "end," so that the presentation module has enough time to update the presented notification information. If the determination result in step S507 is "no," the process returns to step S501. The counting module may set the completion flag to "true" if it determines that the number of times the final reference hand-washing action of the hand-washing process has been performed is equal to or greater than the expected value. The operation flow of the counting module may include a waiting step, and may terminate the waiting step depending on a specific condition.

[0059] An exemplary operation flow of the presentation module includes steps S700 to S703. In step S701, notification information is determined, and the notification information is generated based on, for example, the processing results of the classification module and / or the counting module. In step S703, the presentation module is caused to present the notification information. The operation flow of the presentation module may include a waiting step, and the waiting step may be terminated according to a specific condition.

[0060] Since the method 80 and the apparatus 200 have a corresponding relationship, reference may be made to the above description of the apparatus 200 for details of each module.

[0061] FIG. 9 is an exemplary flowchart illustrating a process 900 executed by the classification module according to the present disclosure. Steps similar to those in FIG. 8 are not described here. The process 900 is suitable for guiding a user to perform a handwashing process according to the order of reference handwashing actions indicated by the reference handwashing process. If a non-reference handwashing action is detected after the last reference handwashing action is performed, the handwashing process monitoring method is terminated. In step S317, initialization is performed, setting k to 1. As described above, −1 to kmax are used to represent the action type attribute Cx of the handwashing action. In step S305′, it is determined whether Cx is k, i.e., it is determined whether the action type attribute of the selected two-hand image is Ak. If the determination result is “YES,” the process returns to step S301. If the determination result is “NO,” the process proceeds to step S313. In step S313, it is determined whether k is kmax. If the determination result is “YES,” the process proceeds to step S321. If the determination result is “NO,” the process proceeds to step S319. In step S319, it is determined whether the action type attribute Cx is k+1. If the determination result is "YES", k is incremented by 1 (step S315). If the determination result is "NO", first information indicating that the hand washing action is incorrect is sent to the presentation module (step S311). In step S321, third information indicating that the hand washing process is completed is sent to the presentation module. After step S321, the hand washing process monitoring method may be terminated. Preferably, after step S321, a waiting step may be set to prevent the presentation module from having enough time to update the presented notification information.

[0062] One aspect of the present disclosure provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, monitors a hand-washing process by executing multiple modules in parallel, including a classification module that determines, based on a set of reference hand-washing actions, action-type attributes of selected hand-washing images in a hand-washing image sequence generated based on the hand-washing process, and a counting module that determines a number of times the selected reference hand-washing action has been performed, based on the selected reference hand-washing action, the hand-washing image sequence, and the determined action-type attributes of the hand-washing images in the hand-washing image sequence.

[0063] One aspect of the present disclosure further provides an information processing device.

[0064] 10 is an exemplary block diagram illustrating an information processing device 1000 according to an embodiment of the present disclosure. In FIG. 10, a central processing unit (CPU) 1001 executes various processes according to programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage unit 1008 to a random access memory (RAM) 1003. The RAM 1003 stores data necessary for the CPU 1001 to execute various processes as needed.

[0065] The CPU 1001, the ROM 1002, and the RAM 1003 are connected to one another via a bus 1004. An input / output interface 1005 is also connected to the bus 1004.

[0066] An input unit 1006 (including a keyboard, a mouse, etc.), an output unit 1007 (including a display, such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.), a storage unit 1008 (including, for example, a hard disk, etc.), and a communication unit 1009 (including a network interface card, such as a LAN card, a modem, etc.) are connected to the input / output interface 1005. The communication unit 1009 executes communication processing via a network, for example, the Internet.

[0067] If necessary, the driver 1010 may be connected to the input / output interface 1005. A removable medium 1011 is set up in the driver 1010 if necessary, and a computer program read from the removable medium 1011 is installed in the storage unit 1008 if necessary.

[0068] The CPU 1001 may execute a program for monitoring a hand-washing process according to the present disclosure.

[0069] According to aspects of the present disclosure, hand-washing actions can be recognized and the number of times each hand-washing action is performed can be determined. Each module operates cooperatively and in parallel, allowing for accurate and comprehensive evaluation of the execution quality of the hand-washing process of the user, maximizing hardware efficiency. The method, device, information processing device, and storage medium of the present disclosure can achieve at least one of the following effects: Accurately evaluate the execution quality of the hand-washing process of the user, comprehensively evaluate the execution quality of the hand-washing process of the user, provide real-time feedback, and provide real-time guidance, improving the user experience.

[0070] Although the above describes specific embodiments of the present disclosure, those skilled in the art may make various modifications (combining or substituting the features of each embodiment in the above cases), improvements, or equivalents to the present disclosure within the spirit and scope of the appended claims. These modifications, improvements, or equivalents fall within the scope of protection of the present disclosure.

[0071] It should be noted that the terms "comprise" and "have" refer to the presence of features, elements, steps or components described in this specification, but do not exclude the presence or addition of one or more other features, elements, steps or components.

[0072] Furthermore, the methods of each embodiment of the present invention are not limited to being performed in the chronological order described in the specification or shown in the drawings, and may be performed in other chronological orders, or may be performed in parallel or independently. Therefore, the order of performing the methods described herein does not limit the technical scope of the present invention.

[0073] Furthermore, the following supplementary notes are also disclosed regarding the embodiments including the above-described examples, but the present invention is not limited to these supplementary notes. (Appendix 1) 1. A device for monitoring a hand washing process, comprising: a memory in which instructions are stored; a processor capable of executing the instructions retrieved from the memory to monitor the hand washing process by executing a plurality of modules in parallel; The plurality of modules include: a classification module that determines, based on a set of reference handwashing actions, an action type attribute of a selected hand-hand image in the sequence of hand-hand images generated based on the hand-washing process; a counting module that determines a number of times the selected reference hand washing motion has been performed based on a selected reference hand washing motion, the sequence of hand imagery, and determined motion type attributes of hand images in the sequence of hand imagery. (Appendix 2) The plurality of modules include: 10. The device of claim 1, further comprising: an update module that updates the sequence of images of both hands based on images of the hand washing process of a person washing their hands captured in real time by a capture device. (Appendix 3) The update module includes: If the received image is determined to be a hand-washing image, an image portion including the hand of the person washing their hands is cut out from the image as a foreground of an intermediate hand-washing image; setting the background of the intermediate two-hand image to a single color; 3. The apparatus of claim 2, wherein the sequence of hand images is updated by adding the intermediate hand image to the sequence of hand images. (Appendix 4) 4. The device of claim 3, wherein an image portion including both hands of the person washing their hands is extracted from the image based on skin color. (Appendix 5) updating the sequence of images of both hands based on the images, cropping an image portion including the hand of the person washing their hands from the received image using a rectangular frame that surrounds the hand of the person washing their hands in the image; and adding the image portion to the image sequence as a two-hand image. (Appendix 6) 4. The device of claim 3, wherein a deep learning model is used to extract an image portion from the image that includes both hands of the person washing their hands. (Appendix 7) 3. The apparatus of claim 2, wherein the update module determines an update period for updating the two-hand image sequence based on a currently determined motion type attribute. (Appendix 8) 8. The apparatus of claim 7, wherein the update module shortens the update period if a currently determined operation type attribute is a new operation type. (Appendix 9) 2. The apparatus of claim 1, wherein the classification module selects one image from the sequence of hand images as the selected hand image for every d hand images, where d is greater than or equal to 1. (Appendix 10) 2. The apparatus of claim 1, wherein the instructions cause the classification module and the counting module to operate cooperatively based on each other's processing results. (Appendix 12) 11. The apparatus of claim 10, wherein the classification module determines the selected two hand images based on a period of the current motion type determined by the counting module. (Appendix 13) When the classification module determines that the frame position of the current hand image, whose action type attribute is a new action type, corresponds to a video frame position with low recognition, the classification module adjusts the action type attribute of the current hand image to the action type attribute determined by the previous action type determination process; The device of claim 1, wherein the low-recognition video frame positions correspond to video frame positions in which the recognizability in the execution period of the reference hand-washing action of the immediately preceding action type attribute is lower than a predetermined recognition threshold. (Appendix 14) 2. The device of claim 1, wherein the classification module adjusts the action type attribute of the selected two hand images to the action type determined by the previous action type determination process if the classification confidence of the selected two hand images is lower than a predetermined confidence threshold. (Appendix 15) 2. The apparatus of claim 1, wherein the classification module, if it determines that the sequence of two hand images contains an abnormal two hand image having an abnormal action type attribute, adjusts the action type attribute of the abnormal two hand image to an action type determined by a current action type determination process. (Appendix 16) The counting module includes: determining a selected image sequence segment based on a first-appearing hand-washing image in the hand-washing image sequence, the hand-washing image having an action-type attribute corresponding to a target predetermined hand-washing action; 10. The device of claim 1, further comprising: determining a number of times the selected reference hand washing action has been performed based on the selected image sequence segment. (Appendix 17) The device described in Appendix 1, wherein the plurality of modules includes a presentation module that presents notification information to the person washing their hands based on at least the number of times the wash has been performed. (Appendix 18) 18. The device of claim 17, wherein the presentation module causes the presentation unit to present the notification information in the form of light, sound, a still image, a video, or a combination thereof. (Appendix 19) 1. A method for monitoring a hand washing process, comprising: monitoring the hand washing process by executing a plurality of modules in parallel; The plurality of modules include: a classification module that determines, based on a set of reference handwashing actions, an action type attribute of a selected hand-hand image in the sequence of hand-hand images generated based on the hand-washing process; a counting module that determines a number of times the selected reference hand washing motion has been performed based on a selected reference hand washing motion, the sequence of hand imagery, and determined motion type attributes of hand images in the sequence of hand imagery. (Appendix 20) A computer-readable storage medium having a program stored therein, the program, when executed by a processor, executing a plurality of modules in parallel to monitor a hand-washing process; The plurality of modules include: a classification module that determines, based on a set of reference handwashing actions, an action type attribute of a selected hand-hand image in the sequence of hand-hand images generated based on the hand-washing process; a counting module that determines a number of times the selected reference hand washing motion has been performed based on a selected reference hand washing motion, the sequence of hand imagery, and determined motion type attributes of hand imagery in the sequence of hand imagery.

Claims

1. 1. A device for monitoring a hand washing process, comprising: a memory in which instructions are stored; a processor capable of executing the instructions retrieved from the memory to monitor the hand washing process by executing a plurality of modules in parallel; The plurality of modules include: a classification module that determines, based on a set of reference handwashing actions, an action type attribute of a selected hand-hand image in the sequence of hand-hand images generated based on the hand-washing process; a counting module for determining a number of times the selected reference hand washing action has been performed based on a selected reference hand washing action in the set of reference hand washing actions, the sequence of hand images, and determined action type attributes of hand images in the sequence of hand images; The counting module includes: determining a selected image sequence segment in the sequence of images of both hands based on the selected image of both hands that first appears in the sequence of images of both hands and has an action type attribute of the selected reference handwashing action; The apparatus determines a number of times the selected reference hand washing action has been performed based on the selected image sequence segment.

2. The plurality of modules include: The apparatus of claim 1 , further comprising: an update module for updating the sequence of hand images based on images of the hand washing process of a person washing their hands captured in real time by a capture device.

3. The update module includes: If the received image is determined to be a hand-washing image, an image portion including the hand of the person washing their hands is cut out from the image as a foreground of an intermediate hand-washing image; setting the background of the intermediate two-hand image to a single color; The apparatus of claim 2 , further comprising: updating the sequence of hand images by adding the intermediate hand images to the sequence of hand images.

4. The apparatus of claim 2 , wherein the update module determines an update period for updating the two-hand image sequence based on a currently determined motion type.

5. The apparatus of claim 1 , wherein the instructions cause the classification module and the counting module to operate cooperatively based on each other's processing results.

6. The device of claim 1 , wherein the classification module determines the selected hand images based on a period of execution of a hand-washing action of the current action type determined by the counting module.

7. When the classification module determines that the frame position of the current hand image, whose action type attribute is a new action type, corresponds to a video frame position with a low recognition level, the classification module adjusts the action type attribute of the current hand image to the action type attribute determined by the previous action type determination process; the degree of recognition indicates the accuracy of the action type attribute determined by the classification module; The device of claim 1 , wherein the low recognizability video frame positions correspond to video frame positions in which the recognizability in the execution period of the reference hand-washing action of the immediately preceding action type attribute is lower than a predetermined recognition threshold.

8. The device of claim 1 , wherein the plurality of modules includes a presentation module that presents notification information to a person washing their hands based on at least the number of times the hand washing has been performed.

9. 1. A method for monitoring a hand washing process, comprising: monitoring the hand washing process by executing a plurality of modules in parallel; The plurality of modules include: a classification module that determines, based on a set of reference handwashing actions, an action type attribute of a selected hand-hand image in the sequence of hand-hand images generated based on the hand-washing process; a counting module for determining a number of times the selected reference hand washing action has been performed based on a selected reference hand washing action in the set of reference hand washing actions, the sequence of hand images, and determined action type attributes of hand images in the sequence of hand images; The counting module includes: determining a selected image sequence segment in the sequence of images of both hands based on the selected image of both hands that first appears in the sequence of images of both hands and has an action type attribute of the selected reference handwashing action; determining a number of times the selected reference hand washing action has been performed based on the selected image sequence segment.

10. A computer-readable storage medium having a program stored therein, the program, when executed by a processor, executing a plurality of modules in parallel to monitor a hand-washing process; The plurality of modules include: a classification module that determines, based on a set of reference handwashing actions, an action type attribute of a selected hand-hand image in the sequence of hand-hand images generated based on the hand-washing process; a counting module for determining a number of times the selected reference hand washing action has been performed based on a selected reference hand washing action in the set of reference hand washing actions, the sequence of hand images, and determined action type attributes of hand images in the sequence of hand images; The counting module includes: determining a selected image sequence segment in the sequence of images of both hands based on the selected image of both hands that first appears in the sequence of images of both hands and has an action type attribute of the selected reference handwashing action; A storage medium that determines a number of times the selected reference hand washing action has been performed based on the selected image sequence segment.

Images