Hand washing recognition system and hand washing recognition method

Abstract

The present invention relates to a handwashing recognition system and method, each intended to encourage a user to properly wash their hands based on the condition of their hands or fingers. The handwashing recognition system comprises an imaging unit, a detection unit, and an abnormality type determination unit. The detection unit detects a first abnormality candidate present in the user's hand from a first image captured by the imaging unit before handwashing, and detects a second abnormality candidate present in the user's hand from a second image captured by the imaging unit after handwashing. The abnormality type determination unit determines the type of abnormality present in the user's hand based on a comparison between the first abnormality candidate and the second abnormality candidate.

Description

Technical Field

[0001] The present invention relates to a system, method and program for recognizing a person's hand washing action. Background Art

[0002] In food preparation and processing sites, pharmaceutical factories, hospitals, and nursing facilities, worker hygiene is crucial for preventing infections. Finger hygiene, including hand washing, is particularly important, as it can easily spread viruses and bacteria. Furthermore, HACCP (Hazard Analysis and Critical Control Points) requires food-related businesses to inspect, monitor, and record hygiene practices. Therefore, technology is needed to identify and record the condition of workers' fingers.

[0003] Under such circumstances, a hand washing monitoring method including the following steps is proposed. The image acquisition step acquires a hand washing image captured by a shooting unit provided in a hand washing sink. The hand area extraction step extracts the hand area from the acquired hand washing image to generate a frame image. The hand washing start determination step determines whether hand washing has started based on the frame image. The washing method identification step identifies the type of washing method being performed by extracting the shape of the hand area from the frame image. The friction determination step determines the quality of the friction state of the washing method by adding seconds to the washing method when the identified washing method is a washing method in a prescribed order. The hand washing end determination step determines the end of hand washing based on the frame image. (For example, Patent Document 1)

[0004] In addition, a hygiene management device has been proposed that automatically determines whether hand washing is being performed correctly and prevents people from entering the hygiene management area from the finger washing area if they are not washing their hands correctly (for example, Patent Document 2). A method has been proposed that can detect motion information over a wide area without contact and analyze finger hygiene actions in real time (for example, Patent Document 3). A hand washing monitoring system has been proposed that can improve the accuracy of identifying people washing hands (for example, Patent Document 4).

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2017-134712

[0006] Patent Document 2: Japanese Patent Application Laid-Open No. 2002-085271

[0007] Patent Document 3: Japanese Patent Application Laid-Open No. 2018-117981

[0008] Patent Document 4: Japanese Patent Application Laid-Open No. 2020-018674

[0009] In previous handwashing recognition methods, it is sometimes impossible to accurately determine the state of the hands or fingers. For example, dirt, wounds, moles, tattoos, etc. may not be correctly identified. Moreover, if the state of the hands or fingers is not correctly determined, it is impossible to determine whether the hands have been washed properly. In addition, it is also difficult to provide appropriate instructions to the workers. For example, if "wounds" are mistakenly identified as "dirt", even if the hands have been washed properly, it will be determined that dirt remains. In this case, even though there is no dirt, a message is output to the worker to continue washing hands. Summary of the Invention

[0010] An object of one aspect of the present invention is to encourage a user to properly wash their hands based on the condition of the user's hands or fingers.

[0011] A hand washing identification system according to one embodiment of the present invention comprises: a photographing unit; a detection unit for detecting a first abnormality candidate existing in a user's hand from a first image photographed by the photographing unit before washing hands, and for detecting a second abnormality candidate existing in the user's hand from a second image photographed by the photographing unit after washing hands; and an abnormality type determination unit for determining the type of abnormality of the user's hand based on a comparison between the first abnormality candidate and the second abnormality candidate.

[0012] According to the above aspect, the user is prompted to wash his hands appropriately based on the state of the user's hands or fingers. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 1 is a diagram showing an example of a hand washing recognition system according to an embodiment of the present invention.

[0014] Figure 2 This is a flowchart showing an example of the processing of the hand washing recognition system.

[0015] Figure 3 This is a diagram showing an example of a method for detecting abnormality candidates.

[0016] Figure 4 This is a diagram showing an example of a method for determining the type of abnormality.

[0017] Figure 5 This is a flowchart showing an example of a process for determining the type of abnormality.

[0018] Figure 6 1 is a flowchart showing a first variation of the hand washing recognition method.

[0019] Figure 7 FIG. 1 is a diagram showing an example of a method for detecting abnormal candidates using a reference finger image.

[0020] Figure 8 1 is a diagram showing an example of the abnormal candidate detection unit used in the first modification.

[0021] Figure 9 1 is a flowchart showing a second variation of the hand washing recognition method.

[0022] Figure 10 This is a flowchart showing a third variation of the hand washing recognition method.

[0023] Figure 11 This is a diagram showing an example of hand washing action.

[0024] Figure 12 This is a diagram showing an example of a method for determining important steps when washing hands.

[0025] Figure 13 This is a diagram explaining a method of detecting the state of a nail. DETAILED DESCRIPTION

[0026] Figure 1 An example of a hand washing recognition system according to an embodiment of the present invention is shown. Figure 1 As shown, the hand washing recognition system 100 according to the embodiment of the present invention comprises: a photographing device 10, an abnormal candidate detection unit 21, an abnormal type determination unit 22, a hand washing instruction generation unit 23, and a storage unit 30. In addition, the hand washing recognition system 100 may also include Figure 1 Other functions or devices not shown.

[0027] The photographing device 10 is, for example, a digital camera, which acquires color images by photographing. Here, the photographing device 10 may also acquire images at a predetermined time interval (for example, 30 frames per second). In this case, the photographing device 10 can actually acquire dynamic images. In addition, the photographing device 10 is, for example, set above a sink where a person washes his hands. Furthermore, the photographing device 10 photographs the fingers of a person when washing his hands. In addition, in the following description, "fingers" means not only "fingers of a hand", but also "hands including fingers" or "hands and fingers". In addition, in the following description, a person whose fingers are photographed by the photographing device 10 is sometimes referred to as a "user". The user is not particularly limited, but for example, is an operator at a food cooking / processing site, a pharmaceutical factory, a hospital, a nursing facility, etc.

[0028] The hand washing recognition system 100 uses the image captured by the camera 10 to generate instructions related to hand washing and provide them to the user. For example, when the user wears accessories (rings, watches, etc.) on their fingers, the hand washing recognition system 100 can also instruct the user to remove the accessories and wash their hands. In addition, when dirt remains after washing hands, the hand washing recognition system 100 can also instruct the user to wash hands again. Moreover, when the user's fingers are injured, the hand washing recognition system 100 can also instruct the user to treat the injury with a band-aid or the like and / or wear plastic gloves. In addition, in the following description, the instructions provided to the user regarding hand washing are sometimes referred to as "hand washing instructions."

[0029] However, simple image recognition may not accurately identify the state of a finger. Furthermore, if the finger's state is not correctly identified, the handwashing recognition system 100 cannot provide the user with appropriate handwashing instructions. Therefore, the handwashing recognition system 100 uses images of the finger before and after washing to identify the user's finger's state.

[0030] The abnormal candidate detection unit 21 detects abnormal candidates present on the user's fingers from the image captured by the camera 10 before washing hands. In addition, the abnormal candidate detection unit 21 detects abnormal candidates present on the user's fingers from the image captured by the camera 10 after washing hands. Here, "abnormality" includes dirt, scratches, and decorations. Therefore, "abnormal candidates" are equivalent to image areas that may be judged as "abnormal" in image recognition. As an example, an area containing color components different from the color of a general hand is detected in the hand area extracted from the input image as an abnormal candidate. In addition, in the following description, the abnormal candidate detected from the image captured by the camera 10 before washing hands is sometimes referred to as "abnormal candidate before washing hands (first abnormal candidate)". In addition, the abnormal candidate detected from the image captured by the camera 10 after washing hands is sometimes referred to as "abnormal candidate after washing hands (second abnormal candidate)".

[0031] The abnormality type determination unit 22 determines the type of abnormality of the user's finger based on the pre-hand washing abnormality candidates and the post-hand washing abnormality candidates detected by the abnormality candidate detection unit 21. At this time, the abnormality type determination unit 22 can determine the type of abnormality of the user's finger based on the comparison of the pre-hand washing abnormality candidates and the post-hand washing abnormality candidates. Then, the hand washing instruction generation unit 23 generates a hand washing instruction based on the type of abnormality of the user's finger determined by the abnormality type determination unit 22. The generated hand washing instruction is provided to the user. In addition, the hand washing instruction generation unit 23 can also generate a warning sound based on the type of abnormality of the user's finger determined by the abnormality type determination unit 22, prompting the user to wash his hands. The hand washing action corresponding to the warning sound can also be conveyed to the user in a different manner from the hand washing identification system 100, such as using a note.

[0032] Furthermore, the processor 20 executes the handwashing recognition program to implement the candidate abnormality detection unit 21, the abnormality type determination unit 22, and the handwashing instruction generation unit 23. Specifically, the processor 20 executes the handwashing recognition program to provide the functions of the candidate abnormality detection unit 21, the abnormality type determination unit 22, and the handwashing instruction generation unit 23. In this case, the handwashing recognition program is stored in the storage device 30, for example. Alternatively, the processor 20 may execute the handwashing recognition program stored on a portable recording medium (not shown). Furthermore, the processor 20 may retrieve and execute the handwashing recognition program from a program server (not shown).

[0033] Figure 2 is a flowchart showing an example of processing of handwashing recognition system 100. Furthermore, for example, the handwashing recognition process is executed when a user issues a start instruction. Alternatively, the handwashing recognition process is executed when the user is in a predetermined location (e.g., in front of a handwashing sink).

[0034] In S1, the processor 20 acquires an image of the user's fingers before washing their hands. Preferably, the processor 20 acquires an image from the wrist to the fingertips. Furthermore, preferably, the processor 20 acquires an image of the palm side and an image of the back of the hand. Therefore, the handwashing recognition system 100 is not particularly limited, but guidance may be provided to the user indicating that an image of the palm including the fingers and wrist, and an image of the back of the hand including the fingers and wrist, are required.

[0035] In S2 to S3, the abnormal candidate detection unit 21 detects abnormal candidates (i.e., abnormal candidates before hand washing) from the input image. At this time, the abnormal candidate detection unit 21 first extracts the hand area corresponding to the user's hand from the input image. The method of extracting the hand area from the input image is not particularly limited and is implemented by well-known techniques. For example, the abnormal candidate detection unit 21 may also extract the hand area from the input image using semantic segmentation. Then, the abnormal candidate detection unit 21 detects abnormal candidates existing in the hand area. As an example, an area in the hand area that contains a color component different from the color of a general hand is detected as an abnormal candidate. In addition, the abnormal candidate detection unit 21 may also detect abnormal candidates using, for example, image recognition deep learning. In Figure 3 In the example shown in (a), abnormal candidate X and abnormal candidate Y are detected.

[0036] When more than one abnormality candidate is detected, in S4-S5, the abnormality type determination unit 22 determines the type of abnormality for each abnormality candidate. At this time, the abnormality type determination unit 22 determines whether the detected abnormality candidate meets the conditions of "dirt", "wound", or "decoration". For example, the abnormality type is determined by deep learning through image recognition. As an example, Figure 3 As shown in (b), the abnormality type determination unit 22 estimates the finger posture by acquiring the feature quantity information of the hand area. In this way, the position of each abnormal candidate relative to the user's finger is determined. Specifically, abnormal candidate X is estimated to be located at the base of the ring finger. In this case, the abnormality type determination unit 22 determines that abnormal candidate X is an ornament (ring). In addition, abnormal candidate Y is estimated to be located at the wrist. In this case, the abnormality type determination unit 22 determines that abnormal candidate Y is an ornament (watch).

[0037] Furthermore, in addition to the position of the abnormality candidate, the abnormality type determination unit 22 may also consider the shape and / or size of the abnormality candidate when determining the type of abnormality. Furthermore, the abnormality type determination unit 22 may determine the type of abnormality without evaluating the finger posture. For example, the abnormality type determination unit 22 may determine the type of abnormality based on the size and shape of the abnormality candidate.

[0038] If the abnormal candidate is a decorative item, in S6, the handwashing instruction generator 23 generates a handwashing instruction to remove the decorative item from the user's hands. This handwashing instruction is provided to the user. Alternatively, the handwashing instruction can be provided through, for example, a voice message. Alternatively, if the handwashing recognition system 100 includes a display device, the handwashing instruction can be displayed on the display device.

[0039] Afterward, the handwashing recognition system 100 returns to S1. Specifically, the handwashing recognition system 100 again captures an image of the user's finger before handwashing. If it is confirmed that the accessory has been removed from the user's hand (S5: No), the handwashing recognition system 100 proceeds to S11. Furthermore, if no abnormality candidate is detected in S3, the handwashing recognition system 100 also proceeds to S11.

[0040] In S11, the hand washing instruction generating unit 23 generates a hand washing instruction requesting hand washing. This hand washing instruction is also provided to the user in the same manner as in S6.

[0041] In S12, processor 20 acquires images of the user's fingers after washing their hands. In S12, as in S1, processor 20 preferably acquires images from the wrist to the fingertips. Furthermore, processor 20 preferably acquires images of the palm side and the back side of the hand. In other words, processor 20 preferably acquires images of the same finger posture before and after washing hands.

[0042] In S13, the abnormality candidate detection unit 21 detects abnormality candidates (i.e., abnormality candidates after hand washing) from the input image. The method of detecting abnormality candidates from the input image is substantially the same in S2 and S13. That is, abnormality candidates are detected by, for example, image recognition deep learning.

[0043] In S14, the abnormality type determination unit 22 determines the type of abnormality. At this point, the abnormality type determination unit 22 determines the type of abnormality for each abnormality candidate detected from the input image before hand washing. Furthermore, the abnormality type determination unit 22 determines the type of abnormality based on the abnormality candidates before hand washing and the abnormality candidates after hand washing. Specifically, the abnormality type determination unit 22 determines the type of abnormality based on a comparison between the abnormality candidates before hand washing and the abnormality candidates after hand washing.

[0044] exist Figure 4 In the example shown in (a), abnormal candidates Z1 and Z2 are detected in the image of the user's fingers before washing hands. In addition, abnormal candidate Z3 is detected in the image of the user's fingers after washing hands. Here, the abnormality type determination unit 22 performs finger posture estimation and size normalization on each input image. Finger posture estimation estimates the posture of the finger by analyzing the shape of the user's finger. Size normalization refers to the result of finger posture estimation and adjusts the size of one or both images so that the sizes of the hand areas extracted from the two input images are consistent with each other. Then, the abnormality type determination unit 22 compares the abnormal candidates before washing hands and the abnormal candidates after washing hands by comparing the two input images.

[0045] exist Figure 4In (a), abnormal candidates Z1 and Z2 are detected from areas A1 and A2, respectively, of the input image before hand washing. Furthermore, abnormal candidates Z3 and Z4 are detected from areas A1 and A2, respectively, of the input image after hand washing. Specifically, abnormal candidates Z1 and Z3 are detected from the same area A1 within the hand region, and abnormal candidates Z2 and Z4 are detected from the same area A2 within the hand region.

[0046] Washing hands can remove "dirt" attached to the user's fingers. Therefore, when comparing abnormal candidates detected from the same area, if the shape of an abnormal candidate detected from an image after washing hands (i.e., a post-washing abnormal candidate) changes from the shape of an abnormal candidate detected from an image before washing hands (i.e., a pre-washing abnormal candidate), the abnormal candidate is determined to be "dirt."

[0047] For example, in the same area, if the size of the abnormal candidate after washing hands is smaller than the size of the abnormal candidate before washing hands, the abnormal candidate is determined to be "dirt". Figure 4 In the example shown in (a), abnormal candidate Z1 and abnormal candidate Z3 are detected from the same area A1, and the size of abnormal candidate Z3 is smaller than that of abnormal candidate Z1. In this case, the abnormality type determination unit 22 determines that abnormal candidates Z1 and Z3 are "dirt" attached to the user's hands. In addition, if the hands are washed properly, there should be no "dirt" in the input image after washing. For example, in Figure 4 In the example shown in (b), no abnormality candidate exists in the area A1 where the abnormality candidate Z1 was detected before hand washing. In this case, the abnormality type determination unit 22 also determines that the abnormality candidate Z1 is "dirt."

[0048] In contrast, a "wound" on the user's finger cannot be removed by washing hands. Therefore, when comparing abnormal candidates detected from the same area, if the shape of the abnormal candidate detected from the image before washing hands (i.e., abnormal candidate before washing hands) and the shape of the abnormal candidate detected from the image after washing hands (i.e., abnormal candidate after washing hands) are identical or almost identical to each other, the abnormal candidate is determined to be a "wound". Figure 4 In the example shown in (a), abnormality candidates Z2 and Z4 are detected from the same area A2, and the shapes of abnormality candidates Z2 and Z4 are almost identical. In this case, the abnormality type determination unit 22 determines that abnormality candidates Z2 and Z4 are "wounds" incurred on the user's hand.

[0049] In S15, the hand washing instruction generating unit 23 determines whether "dirt" remains on the user's finger. In this case, if an abnormal candidate is detected in the same area of ​​the image before and after hand washing, and the size of the abnormal candidate after hand washing is smaller than that of the abnormal candidate before hand washing, it is determined that "dirt" remains on the user's finger. For example, Figure 4 In the example shown in (a), the size of abnormal candidate Z3 detected from the image after hand washing is smaller than the size of abnormal candidate Z1 detected from the image before hand washing, so it is determined that "dirt" remains in area A1. In this case, the processing of the hand washing recognition system 100 returns to S11. That is, the hand washing instruction generation unit 23 generates a hand washing instruction requesting hand washing. In addition, in the case where S11 is executed after executing S12 to S14, the hand washing instruction may also include an instruction to change the detergent to be used for hand washing. For example, the hand washing instruction generation unit 23 may estimate that dirt that cannot be removed by ordinary soap is attached to the user's hands and recommend the use of a medicinal soap with stronger cleaning power.

[0050] If there is no "dirt" left on the user's finger, the hand washing identification system 100 proceeds to step S16. Figure 4 In the image before hand washing shown in (a), an abnormal candidate Z1 is detected in the area A1, but Figure 4 In the image after hand washing (b), no abnormality candidate exists in the same area A1. In this case, the hand washing instruction generator 23 determines that "dirt" has been removed from area A1 and no dirt remains. Furthermore, if no abnormality candidate is determined to be "dirt" in S13 and S14, the hand washing recognition system 100 proceeds to S16.

[0051] In S16, the handwashing instruction generating unit 23 determines whether a "wound" occurs on the user's finger. Figure 4 In the example shown, abnormality candidates Z2 and Z4 are determined to be "wounds." When a "wound" occurs on the user's finger, in S17, the handwashing instruction generation unit 23 generates a handwashing instruction requesting treatment of the wound. This handwashing instruction may include, for example, instructions for applying a bandage and for wearing gloves. Furthermore, this handwashing instruction includes a message indicating that the finger, after treatment, is to be photographed. This handwashing instruction is also provided to the user, similar to S6.

[0052] In S18, the processor 20 obtains an image of the user's finger. At this time, the processor 20 obtains an image including the area where the wound occurs. In S19, the hand washing instruction generating unit 23 determines whether the wound on the user's finger has been properly treated based on the image obtained in S18. For example, if an image equivalent to a "band-aid" is detected in the area determined to be a "wound", the hand washing instruction generating unit 23 determines that the wound on the user's finger has been properly treated. In this case, the processing of the hand washing identification system 100 ends. In addition, the hand washing identification system 100 can also be used in Figure 2 When the process of the flowchart shown is completed, the server computer records that the user's finger is in a good hygienic state. On the other hand, if it is not determined that the wound on the user's finger has been properly treated, the process of the handwashing recognition system 100 returns to S17.

[0053] In this way, when dirt remains on the user's fingers, Figure 2 The process of the flowchart shown does not end. In this case, S11 to S15 are repeatedly executed. In addition, if the user's finger is injured, the process does not end. Figure 2 The process of the flowchart shown in the figure is continued until the wound is properly treated. In this case, S17 to S19 are repeatedly executed. Therefore, according to the embodiment of the present invention, the hygiene status of the user's finger can be properly managed.

[0054] Figure 5 This is a flowchart showing an example of a process for determining the type of abnormality. Figure 2 That is, after the process of detecting abnormality candidates from the image before hand washing and the process of detecting abnormality candidates from the image after hand washing, the abnormality type determination unit 22 executes the process of the flowchart.

[0055] In S21, the abnormality type determination unit 22 initializes a variable i. The variable i identifies the abnormality candidate detected from the image before hand washing. Figure 4 In the embodiment shown in (a), abnormal candidates Z1 and Z2 are identified.

[0056] In S22, the abnormality type determination unit 22 selects an abnormality candidate Zi from the abnormality candidates detected from the image before hand washing. In S23, the abnormality type determination unit 22 determines the area where the abnormality candidate Zi is detected. Figure 5 In the descriptions herein, the region where the abnormal candidate Zi is detected may be referred to as "region Ai".

[0057] In S24, the abnormality type determination unit 22 determines whether an abnormality candidate exists in the area Ai on the image after hand washing. If no abnormality candidate exists in the area Ai on the image after hand washing, in S27, the abnormality type determination unit 22 determines that the abnormality candidate Zi is "dirt" and that the dirt has been removed by hand washing.

[0058] If an abnormal candidate exists in area Ai of the hand-washing image, the abnormality type determination unit 22 compares the abnormal candidate Zi with the abnormal candidate Zk in steps S25 and S26. If the size of the abnormal candidate Zk is smaller than that of the abnormal candidate Zi, the abnormality type determination unit 22 determines in step S28 that the abnormal candidate Zi is "dirt" and that some of the dirt has been removed by hand-washing. If the abnormal candidate Zi and the abnormal candidate Zk are identical or nearly identical, the abnormality type determination unit 22 determines in step S29 that the abnormal candidate Zi is "wound." If the size of the abnormal candidate Zk is larger than that of the abnormal candidate Zi, the abnormality type determination unit 22 performs error handling in step S30.

[0059] However, the abnormality type determination unit 22 may also determine that the abnormal candidate Zi is "dirt" regardless of the size of the abnormal candidate Zi and Zk, provided that the shape of the abnormal candidate Zi differs from the shape of the abnormal candidate Zk. For example, there may be cases where dirt is attached to the user's hands, and the dirt area may expand due to rubbing the hands when washing. In this case, even if the size of the abnormal candidate Zk is larger than the size of the abnormal candidate Zi, the abnormal candidate Zi may still be determined to be "dirt."

[0060] In S31, the variable i is incremented by 1. In S32, the abnormality type determination unit 22 determines whether any abnormality candidate Zi remains for which the processes S23 to S30 have not yet been executed. If such an abnormality candidate Zi remains, the abnormality type determination unit 22 returns to S22. In other words, the next abnormality candidate is selected from the pre-handwashing image. Therefore, the abnormality type is determined for each abnormality candidate Zi detected from the pre-handwashing image.

[0061] Thus, in the handwashing recognition method according to the embodiment of the present invention, the handwashing process that the user should perform is indicated based on the change from the image before handwashing to the image after handwashing. At this time, the handwashing recognition system 100 indicates the appropriate handwashing process and the process after handwashing by identifying dirt, scratches, and decorations, so that the user's (here, the operator's) fingers are in a state suitable for the operation. Here, compared with simple image recognition, the method using the change from the image before handwashing to the image after handwashing can accurately determine the type of abnormality of the user's fingers. Therefore, according to the handwashing recognition method according to the embodiment of the present invention, it is possible to provide the user with appropriate handwashing instructions. As a result, the hygiene conditions of operators in food-related industries, for example, are improved.

[0062] <Transformation 1>

[0063] Figure 6 is a flowchart showing a first variation of the hand washing identification method. Figure 2 In addition to S1 to S6 and S11 to S19 shown, S41 to S43 are also executed.

[0064] In S41, the handwashing recognition system 100 identifies the user. The user is identified using known techniques. For example, the user can be identified through biometric authentication (facial authentication, vein authentication, fingerprint authentication, iris authentication, etc.). Alternatively, if each user possesses a personal identification device (such as an IC chip), the user is identified using this device. In short, user identification is preferably performed in a contactless manner.

[0065] In S42, the processor 20 obtains the determined reference finger image of the user from the storage unit 30. Figure 1 As shown, the storage unit 30 stores a reference finger image of each user. In this embodiment, the reference finger image is an image of the user's finger captured under normal circumstances. The reference finger image can be captured by the camera 10 or another camera. It is preferable to store an image that meets the following conditions as the reference finger image.

[0066] (1) No ornaments are worn on the fingers.

[0067] (2) There is no dirt on the fingers.

[0068] (3) The fingers are not injured.

[0069] In this embodiment, in the process of detecting abnormal candidates (in Figure 6 For example, assuming that the processor 20 obtains Figure 7 (a) shows an input image and a reference finger image. The input image is captured by the camera 10. The reference finger image is stored in the storage unit 30. In this example, an abnormal candidate Z1 is detected from area A1 on the input image. However, no abnormal candidate exists within area A1 on the reference finger image. In this case, abnormal candidate Z1 is estimated to be a foreign object that is not normally present on the user's finger.

[0070] In contrast, in Figure 7In the example shown in (b), abnormal candidate Z5 is detected from area A5 on the input image. In addition, abnormal candidate Z6 exists in area A5 on the reference finger image. Here, the shapes of abnormal candidate Z5 and abnormal candidate Z6 are identical to each other. In this case, abnormal candidate Z5 detected from the input image is estimated to be present on the user's finger from time to time. That is, abnormal candidate Z5 is estimated not to be "dirt" that can be removed by washing hands, and also not to be a "wound" incurred on the user's finger. Therefore, in subsequent processing, abnormal candidate Z5 is removed from the abnormal candidates. In addition, the abnormal candidate detected from the reference finger image is a foreign object that is present on the user's finger from time to time, such as a mole, a scar, a tattoo, etc.

[0071] In this way, even if an abnormal candidate is detected from the input image, it does not need to be removed by hand washing if it also exists in the reference finger image. In addition, no treatment is required. Moreover, such abnormal candidates are removed in the processing of S14 to S19. Thus, in variant 1, the generation of inappropriate hand washing instructions can be suppressed. For example, if a mole, scar, tattoo, etc. is determined to be "dirt", it is required to continue washing hands. In addition, if a mole, scar, tattoo, etc. is determined to be "injury", a prescribed treatment is required. In contrast, in variant 1, the generation of such inappropriate hand washing instructions can be suppressed.

[0072] exist Figure 6 In the processes of S11 to S19 shown above, if it is determined that the finger is not injured, the processor 20 may update the reference finger image in S43. For example, if the reference finger image stored in the storage unit 30 is older, the processor 20 may store the image newly captured by the camera 10 in the storage unit 30.

[0073] In S2, the abnormal candidate detection unit 21 may detect abnormal candidates by comparing the input image before hand washing with the reference finger image. In S13, the abnormal candidate detection unit 21 may detect abnormal candidates by comparing the input image after hand washing with the reference finger image.

[0074] Figure 8 An example of the abnormal candidate detection unit 21 used in the first modification is shown. In this example, the abnormal candidate detection unit 21 includes a finger posture estimation unit 21a, a position alignment unit 21b, a difference detection unit 21c, and a position / shape detection unit 21d.

[0075] The finger posture estimation unit 21a estimates the posture of the user's fingers by detecting the feature value of the hand area extracted from the input image. The alignment unit 21b uses a non-rigid transformation based on the posture estimated by the finger posture estimation unit 21a to align the input image with the reference finger image. At this time, the alignment can also be performed using wrinkles on the fingers. The difference detection unit 21c detects the difference between the input image and the reference finger image. The difference area is detected as an abnormal candidate. The position / shape detection unit 21d detects the position and shape of the abnormal candidate with reference to the posture estimated by the finger posture estimation unit 21a. In this example, abnormal candidates are detected at the base of the ring finger and the wrist.

[0076] Furthermore, the handwashing recognition system 100 may also store, in addition to the reference finger image, accessory information indicating the characteristics of a predetermined accessory in the storage unit 30. In this case, the accessory information preferably includes an image of the predetermined accessory. Furthermore, in steps S4 and S5, the abnormality type determination unit 22 may determine whether the user is wearing an accessory on their finger based on the input image and the accessory information. If it is determined that the user is wearing an accessory on their finger, the handwashing instruction generation unit 23 generates and outputs a handwashing instruction requesting the user to remove the accessory.

[0077] Alternatively, the handwashing recognition system 100 may detect a first abnormality candidate from a reference finger image stored in the storage unit 30, and a second abnormality candidate from an image captured by the camera 10 after handwashing. In this case, the handwashing instruction generator 23 generates an instruction regarding handwashing based on a comparison of the first abnormality candidate and the second abnormality candidate.

[0078] Transformation 2

[0079] Figure 9 is a flowchart showing a second variation of the hand washing identification method. Figure 2 In addition to S1 to S6 and S11 to S19 shown above, S51 to S52 are also executed. S51 to S52 are executed after S1 to S6. Specifically, S51 to S52 are executed when it is determined that the abnormal candidate detected from the input image is not a decorative item.

[0080] In S51 to S52, the abnormality type determination unit 22 determines whether the abnormality candidate detected in S2 has a specific color component. Moreover, when the abnormality candidate has a specific color component, the hand-washing instruction generation unit 23 changes the hand-washing process. Specifically, when the abnormality candidate has a specific color component, the hand-washing instruction generation unit 23 may also change the type of detergent and / or the hand-washing action. For example, the abnormality type determination unit 22 determines whether the abnormality candidate has a specific color component corresponding to oil stains. Moreover, when the abnormality candidate has a specific color component corresponding to oil stains, the hand-washing instruction generation unit 23 determines a detergent suitable for removing the oil stains. In addition, the hand-washing instruction generation unit 23 may also determine a hand-washing action suitable for removing the oil stains.

[0081] When the detergent and hand-washing action are determined in S52, the hand-washing instruction generator 23 generates a hand-washing instruction corresponding to the determination in S11. In this case, the hand-washing instruction generator 23 instructs the user to use the determined detergent. Alternatively, the hand-washing instruction generator 23 instructs the user to perform the determined hand-washing action. This allows the user to fully remove dirt from their fingers.

[0082] In addition, Figure 9 In the illustrated embodiment, the hand washing procedure is determined based on the color components of the abnormality candidate before hand washing. However, in a second variation, the hand washing procedure may be changed based on the color components of the abnormality candidate remaining after hand washing. For example, if it is determined in S15 that dirt remains, the hand washing instruction generating unit 23 may change the hand washing procedure based on the color components of the remaining abnormality candidate (i.e., dirt).

[0083] Transformation 3

[0084] Figure 10 is a flowchart showing a third variation of the hand washing identification method. Figure 2 In addition to S1 to S6 and S11 to S19 shown above, S61 is also executed. In this embodiment, S61 is executed after S4. Specifically, S61 is executed when the type of abnormality is determined based on the input image.

[0085] In S61, the abnormality type determination unit 22 receives input from the user. Here, the user enters information regarding the condition of their finger. For example, if the finger is injured, the user may also enter information indicating the location of the injury. Furthermore, if the abnormality candidate detection unit 21 detects an abnormality candidate, the user may also enter information indicating the attributes of the abnormality candidate (such as dirt, injury, mole, scar, tattoo, etc.). Furthermore, if the abnormality type determination unit 22 in S4 incorrectly identifies an abnormality type, the user may correct the determination.

[0086] Thus, in the third variation, the user inputs or corrects information regarding the type of abnormality. Furthermore, this user-input or corrected information is used when determining the handwashing procedure the user should follow. Therefore, even when the type of abnormality cannot be determined using image recognition alone, appropriate handwashing instructions can be provided to the user.

[0087] Transformation 4

[0088] In the fourth variation, the hand washing recognition system 100 recognizes a person's hand washing action and determines whether the hand washing action is performed correctly. Here, it is assumed that the hand washing action monitored by the hand washing recognition system 100 includes a plurality of predetermined action steps. Specifically, although not particularly limited, the hand washing action includes Figure 11 The operation steps 1 to 6 are shown. In addition, the operation steps 1 to 6 are as follows.

[0089] Step 1: After wetting your palms with running water, apply detergent and rub your palms.

[0090] Step 2: Rub the back of one hand with the palm of your other hand.

[0091] Step 3: Clean your fingertips and between your nails.

[0092] Step 4: Clean between your fingers.

[0093] Step 5: Twist your thumb and its base while washing.

[0094] Step 6: Wash your wrists.

[0095] The storage unit 30 stores cleaning area information. The cleaning area information indicates the area to be cleaned in each operation step. Figure 12 In the example shown, the washing area information indicates that the back of the hand is washed in action step 2 .

[0096] Although Figure 1 Although not shown, the hand washing recognition system 100 includes a key step determination unit. The key step determination unit is implemented by executing a hand washing recognition program by the processor 20. Figure 12 As shown in FIG, the key step determination unit determines the position of the abnormal candidate detected by the abnormal candidate detection unit 21 by acquiring the feature amount information of the hand area in the input image. Figure 12 In the example shown, it is recognized that an abnormality candidate (for example, dirt) exists on the back of the left hand.

[0097] The key step determination unit determines the key step by referring to the washing area information. In this example, the action step of washing the back of the hand (i.e., action step 2) is determined as the key step. At this time, the key step determination unit can also determine the key step by projecting the abnormal candidate onto the standard hand space of the washing area information based on the feature value of the hand area. Moreover, the key step determination unit notifies the hand washing instruction generation unit 23 of the determined key step. In this way, the hand washing instruction generation unit 23 gives the user an indication of the key step for careful notification. At this time, the hand washing recognition system 100 can also make the benchmark for determining whether the key step is performed correctly stricter compared with other action steps.

[0098] In addition, the key step determination unit may also output the following suggestions.

[0099] (1) When the abnormality candidate is dirt, the manager visually checks whether the dirt has been sufficiently removed.

[0100] (2) If the abnormality candidate is an injury, the manager shall check the condition of the injury afterwards.

[0101] Thus, in the fourth variation, before the user washes his hands, important steps are determined from a plurality of hand washing operation steps and notified to the user, so that the user can reliably perform the important operation steps.

[0102] In addition, in the above example, the key step is determined before the user washes his hands, but the fourth variation is not limited to this method. For example, the key step determination unit may also be Figure 2 If it is determined in S15 that dirt remains, an action step for removing the dirt is determined. In this case, the key step determination unit detects an area of ​​remaining dirt within the hand area. The key step determination unit then determines an action step for a washing area containing the dirt area. This method allows the user to be notified of an appropriate action step for residual dirt after washing their hands.

[0103] Transformation 5

[0104] The handwashing recognition system 100 uses the features of the hand region in the input image to align the position, angle, and finger posture of the hand region with a reference finger image of the subject. These features include hand posture (the indirect position of each finger and the position of the tip of each finger), hand contour, and hand wrinkles.

[0105] After alignment, the handwashing recognition system 100 determines the image area corresponding to the abnormality candidate (wounds, dirt, decorations, etc.) based on the difference between the two images. At this point, pre-processing such as adjusting white balance, contrast, and brightness is preferably performed. The difference calculation can also be performed using deep learning.

[0106] Afterwards, the handwashing recognition system 100 determines the attributes of each abnormal candidate (wounds, dirt, watches, rings, manicures, etc.). At this time, the attributes can also be determined based on the location of the abnormal candidate. For example, an abnormal candidate appearing at the base of a finger is estimated to be a ring, and an abnormal candidate appearing at the wrist is estimated to be a watch. In addition, wounds and dirt are identified based on the changes between the image before and after handwashing. For example, if the shape of the abnormal candidate does not change between the two images, it is estimated to be a wound, and if there is a change between the two images, it is estimated to be dirt. Wounds and dirt can also be identified based on the color of the abnormal candidate.

[0107] Transformation 6

[0108] The handwashing recognition system 100 can also measure the length of the user's nails when washing their hands. If the length of the nails is inappropriate (including too long or too short), the handwashing recognition system 100 outputs a reminder.

[0109] The handwashing recognition system 100 calculates the length of the nails of each finger of the user using the input image. Figure 13 In the example shown, length L is calculated. Furthermore, the ideal nail length is pre-registered for each user. The handwashing recognition system 100 then calculates the difference between length L obtained from the input image and the ideal length and compares this difference with a predetermined threshold. If the difference is greater than the threshold, the handwashing recognition system 100 outputs a reminder to the user.

[0110] The handwashing recognition system 100 can also be used Figure 13 The nail condition is determined based on the length of the nail plate L1 and the length of the nail tip L2 shown. In this case, for example, if (L1 + L2) / L1 deviates from a predetermined threshold range, a reminder is output. This method allows users who have not registered an ideal nail condition to determine whether their nail length is appropriate.

[0111] Transformation 7

[0112] When the handwashing recognition system 100 determines that the user's finger has a wound, it can compare the wound with past data to estimate whether the wound condition has improved or worsened. For example, if the area corresponding to the wound has become larger or the number of wounds has increased, it is estimated that the wound condition has worsened.

[0113] If a previously detected wound is not detected in a new input image, the handwashing recognition system 100 calculates the time from the wound detection to the present time. If this time is shorter than the estimated time required for wound healing, the handwashing recognition system 100 performs the following processing.

[0114] (1) The person confirms whether the injury has healed.

[0115] (2) Ask the administrator to confirm whether the injury has healed.

[0116] (3) Take photos to see the corresponding position more clearly and perform damage detection again.

[0117] <Transformation 8>

[0118] The handwashing recognition system 100 may also include a biometric information acquisition device to identify the user or their administrator. In this case, the handwashing recognition system 100 may perform palm vein authentication or facial authentication, for example. Furthermore, if each user or administrator carries a personal authentication IC chip, the handwashing recognition system 100 may also be equipped with a function to read the IC chip.

[0119] Transformation 9

[0120] The handwashing recognition system 100 may also have a function for receiving information indicating the user's status. For example, the user can use this function to input information indicating their status (e.g., "before work," "just after using the restroom," etc.). In this way, the handwashing recognition system 100 can also modify the handwashing routine based on the user's input. For example, if the user's status is "just after using the restroom," the handwashing recognition system 100 may instruct the user to rub their hands more frequently.

[0121] Transformation 10

[0122] The handwashing recognition system 100 can also record the results of handwashing recognition for each user. For example, the system can record the presence, location, size, and type of wounds. Furthermore, it can record any wound treatment (whether a bandage was applied, gloves were worn, etc.). It can also record the presence, location, size, and type of dirt / decorations. Furthermore, after handwashing is complete, it can record that the dirt has been removed.

[0123] In the example of recording data of each user as described above, the handwashing recognition system 100 can also use data of multiple users to adjust parameters related to handwashing recognition. For example, parameters for detecting abnormal candidates and parameters for determining the type of abnormal candidates can be adjusted.

[0124] Transformation 11

[0125] The handwashing recognition system 100 can also be equipped with a function for recognizing the shape and / or movement of the user's hands. In this case, the camera 10 captures a predetermined shape or movement. The handwashing recognition system 100 then performs processing corresponding to the shape or movement. This configuration allows users to provide instructions to the handwashing recognition system 100 in a contactless manner, even when their fingers are dirty.

[0126] Transformation 12

[0127] The handwashing recognition system 100 may also have a function for detecting contamination on the lens of the camera 10. For example, the storage unit 30 may store a reference image when the user's finger is not within the imaging range. In this case, the handwashing recognition system 100 acquires an image captured by the camera 10 before the handwashing recognition process begins (i.e., before the user places their finger within the imaging range). The acquired image is then compared with the reference image to detect contamination on the lens.

[0128] When dirt on the lens is detected, the handwashing recognition system 100 requests the administrator or user to remove the dirt from the lens. Thereafter, the handwashing recognition system 100 executes a predetermined confirmation procedure to confirm whether the dirt on the lens has been removed.

[0129] Transformation 13

[0130] The handwashing recognition system 100 may also include an image projector that optically projects images into space. In this case, the image projector projects a warning image around the detected scratches, stains, decorations, etc. Alternatively, the image projector may project an image that illustrates how to address the detected scratches, stains, or decorations. Furthermore, the handwashing recognition system 100 may also utilize the image projector to project images that guide users to the correct hand position.

[0131] Description of Reference Numerals

[0132] 10 ... imaging device, 20 ... processor, 21 ... abnormal candidate detection unit, 22 ... abnormal type determination unit, 23 ... hand washing instruction generation unit, 30 ... storage unit, 100 ... hand washing recognition system.

Claims

1. A hand washing recognition system comprising: Filming Department; a detection unit that detects a first abnormality candidate present in the user's hand from a first image captured by the imaging unit before hand washing, and detects a second abnormality candidate present in the user's hand from a second image captured by the imaging unit after hand washing; and The abnormality type determination unit determines the type of abnormality of the user's hand based on a difference between the first abnormality candidate and the second abnormality candidate when the detection unit detects the first abnormality candidate and the second abnormality candidate from the same area.

2. The hand washing identification system according to claim 1, characterized in that: The apparatus further includes an instruction generating unit configured to generate an instruction regarding hand washing based on the type of abnormality determined by the abnormality type determining unit.

3. The hand washing identification system according to claim 1, characterized in that: further comprising a storage unit for storing a reference image representing a past state of the user's hand; The detection unit detects the first abnormality candidate by comparing the reference image and the first image.

4. The hand washing identification system according to claim 1, characterized in that: further comprising a storage unit for storing a reference image representing a past state of the user's hand; The detection unit detects the second abnormality candidate by comparing the reference image and the second image.

5. The hand washing identification system according to claim 2, characterized in that: The instruction generating unit instructs the user to further wash hands when the first abnormality candidate and the second abnormality candidate are detected from the same area and the shapes of the first abnormality candidate and the second abnormality candidate are different from each other.

6. The hand washing identification system according to claim 5, characterized in that: The instruction generating unit instructs the user to change a detergent used for hand washing.

7. The hand washing identification system according to claim 5, characterized in that: The instruction generating unit instructs the user to perform hand washing based on the position of the second abnormality candidate.

8. The hand washing identification system according to claim 2, characterized in that: The instruction generating unit instructs the user on an action flow of hand washing based on a color component of the first abnormality candidate or the second abnormality candidate.

9. The hand washing identification system according to claim 2, characterized in that: When the first abnormality candidate and the second abnormality candidate are detected from the same area, and the shapes of the first abnormality candidate and the second abnormality candidate are identical or almost identical to each other, the abnormality type determination unit determines that the second abnormality candidate is a wound on the user's hand. The instruction generating unit instructs the user to treat the injury.

10. The hand washing identification system according to claim 2, characterized in that: The abnormality type determination unit determines whether the first abnormality candidate is a decorative item based on the position and shape of the first abnormality candidate. When it is determined that the first abnormality candidate is a decorative item, the instruction generating unit instructs the user to remove the decorative item.

11. The hand washing identification system according to claim 2, characterized in that: It also includes a storage unit that stores a reference image representing a predetermined decorative item. The abnormality type determination unit determines whether the first abnormality candidate is a decorative item by comparing the first image with the reference image. When it is determined that the first abnormality candidate is a decorative item, the instruction generating unit instructs the user to remove the decorative item.

12. A hand washing recognition system comprising: Filming Department; a storage unit that stores a reference image representing a user's past hand state; a detection unit that detects a first abnormality candidate present in the user's hand from the reference image and detects a second abnormality candidate present in the user's hand from the image captured by the imaging unit after washing hands; and The abnormality type determination unit determines the type of abnormality of the user's hand based on a difference between the first abnormality candidate and the second abnormality candidate when the detection unit detects the first abnormality candidate and the second abnormality candidate from the same area.

13. A hand washing identification method, characterized in that: Detecting a first abnormality candidate present in the user's hand from a first image captured by the imaging device before washing the hand, detecting a second abnormality candidate existing in the user's hand from a second image captured by the imaging device after washing the hands, When the first abnormality candidate and the second abnormality candidate are detected from the same area, the type of abnormality of the user's hand is determined based on a difference between the first abnormality candidate and the second abnormality candidate.

Images