Method for generating learning model, information processing method, computer program, and information processing apparatus

Abstract

To provide: a generation method of a learning model which generates a learning model for specifying a type of hand washing action, an information processing method for specifying, by using the learning model generated by the method, the type of hand washing action; and a computer program and an information processing device for implementing the methods.SOLUTION: Training data in which image data obtained by photographing hands and fingers of a worker who is washing the hands and a plurality of layers of labels regarding hand washing are associated with each other is acquired, and when image data is input, a learning model for outputting a plurality of pieces of information regarding the plurality of layers of labels is generated on the basis of the acquired training data. The image data obtained by photographing the hands and fingers of the worker who is washing the hands is acquired. The acquired image data is input to the generated learning model generated by the above-mentioned generation method. A plurality of pieces of information regarding the plurality of layers of labels output by the learning model is acquired. The type of hand wash action of the worker is specified on the basis of the acquired plurality of pieces of information.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a method for generating a learning model for generating a learning model for specifying the types of handwashing operations, an information processing method for specifying the types of handwashing operations using the learning model generated by this method, a computer program for implementing these methods, and an information processing device.

Background Art

[0002] In order to prevent the spread of harmful viruses, the importance of handwashing has been increasing. In Patent Document 1, a procedure detection device is proposed that learns images of a series of procedures performed by a person, estimates which of the series of procedures is in the image captured by an imaging device, determines the procedure for each image from each estimated procedure estimated from each image, and calculates the time related to each determined procedure to determine whether handwashing has been performed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Correct handwashing procedures include, for example, those published by the Ministry of Health, Labour and Welfare. However, the correct handwashing procedures are not limited to those published in this way, and there may be multiple correct handwashing procedures from a hygienic perspective. In a method of estimating handwashing procedures using machine learning, the accuracy of estimation may significantly decrease or it may not be possible to estimate at all for handwashing patterns that do not exist in the data used during learning.

[0005] The present invention has been made in view of such circumstances, and an object thereof is to provide a method for generating a learning model for generating a learning model for specifying the type of handwashing operation, an information processing method for specifying the type of handwashing operation using the learning model generated by this method, a computer program for implementing these methods, and an information processing apparatus.

Means for Solving the Problems

[0006] A method for generating a learning model according to an embodiment acquires training data in which image data obtained by photographing the fingers of an operator performing handwashing or distance data obtained by measuring the distance to the fingers is associated with a plurality of hierarchical labels related to handwashing, and based on the acquired training data, generates a learning model that outputs a plurality of pieces of information related to the plurality of hierarchical labels when the image data or the distance data is input.

Effects of the Invention

[0007] In the case of one embodiment, it is possible to generate a learning model that can be expected to specify various types of handwashing operations, and it is possible to expect to specify the types of various handwashing operations using the generated learning model.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Embodiments for Carrying Out the Invention

[0009] A specific example of the information processing system according to the embodiment of the present invention will be described below with reference to the drawings. Note that the present invention is not limited to these examples, and is defined by the claims, and is intended to include all modifications within the meaning and scope equivalent to the claims.

[0010] <System Overview> FIG. 1 is a schematic diagram for explaining the outline of the information processing system according to Embodiment 1. The information processing system according to Embodiment 1 is a system in which a monitoring device 3 installed in a washroom or the like photographs an operator performing handwashing, and monitors based on a photographed image whether the handwashing operation for keeping the fingers clean is being appropriately performed. The monitoring device 3 according to the present embodiment can be installed in various facilities such as food factories, pharmaceutical factories, medical institutions, various research institutes, cooking facilities, educational institutions, stores, and the like.

[0011] In the information processing system according to Embodiment 1, the monitoring device 3 uses a learning model 5 that has been machine-learned in advance, and determines whether the handwashing operation is being appropriately performed based on the image photographed by the camera. The learning model 5 used by the monitoring device 3 is machine-learned in advance by a learning device 1. The learning device 1 uses a plurality of training data in which pre-photographed handwashing images are associated with labels indicating the types of handwashing operations shown in these images, and performs machine learning of a learning model such as a neural network, thereby classifying the types of handwashing operations shown in the image for the input of the photographed image. A learning model 5 is generated. The learning model 5 generated by the learning device 1 is provided to the monitoring device 3 via communication or a recording medium or the like.

[0012] In conventional supervised machine learning, for input data such as an image, one label as a classification result is attached and used as training data (teacher data). In contrast, in the information processing system according to the present embodiment, the learning device 1 performs machine learning of the learning model 5 using, as training data, input data of a photographed image of a handwashing operation with a plurality of hierarchical labels attached. By using such a plurality of hierarchical labels, it can be expected that the generated learning model 5 can classify the types of handwashing operations in more detail. In the present embodiment, it is assumed that the learning device 1 generates the learning model 5, but it is not limited to this, and the monitoring device 3 may generate the learning model 5.

[0013] Figs. 2 and 3 are schematic diagrams for explaining the photographing of the handwashing operation by the monitoring device 3. Fig. 2 shows a state of looking down on a handwashing sink (handwashing area) provided in a facility from above, and Fig. 3 shows a state of looking at the handwashing sink from the side (the right side in Fig. 2). The operator performs the handwashing operation in a state of standing, for example, on the lower side in Fig. 2 and the left side in Fig. 3. The upper side in Fig. 2 and the right side in Fig. 3 are regarded as the front side of the operator.

[0014] The monitoring device 3 according to the first embodiment includes a camera 33 for photographing the handwashing operation of the operator, a display unit 35 for displaying information to the operator, and the like. The camera 33 is provided at a position where it can photograph the fingers of the operator (the situation of the handwashing operation) performing the handwashing operation at the handwashing sink. For example, as shown in Figs. 2 and 3, it is provided so as to photograph the fingers of the operator from above the front side of the operator. In Figs. 2 and 3, the image of the photographing range of the camera 33 is shown by a solid line. Note that the camera 33 only needs to be able to photograph at least the fingers of the operator, and may be configured to photograph the whole body of the operator. Further, the camera 33 may be configured to photograph not only from above the front side of the operator but also from above the lateral direction, or may be configured to photograph only from above the lateral direction.

[0015] The display unit 35 is a liquid crystal display, an organic EL display, or the like, and displays various types of information to be notified to the operator. The display unit 35 may be a touch panel capable of receiving an operation by the operator. The display unit 35 is provided at a position where it can be visually recognized by the operator using the handwashing sink, for example, as shown in Fig. 2.

[0016] The monitoring device 3 according to Embodiment 1 identifies the type of handwashing operation performed by the operator based on the image captured by the camera 33, and determines whether each handwashing operation is being properly performed. By displaying the determination results of these handwashing operations on the display unit 35, the monitoring device 3 can notify the operator of information such as the correctness of the handwashing operation, the progress of the handwashing operation, and advice regarding the handwashing operation. A learning model 5 that has been previously machine-learned by the learning device 1 is used for these processes performed by the monitoring device 3.

[0017] <Generation of Learning Model> FIG. 4 is a block diagram showing the configuration of the learning device 1 according to the present embodiment. The learning device 1 according to the present embodiment includes a processing unit 11, a storage unit 12, a communication unit 13, and the like. The processing unit 11 is configured using one or more arithmetic processing devices such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), or a GPU (Graphics Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). By reading and executing the program 12a stored in the storage unit 12, the processing unit 11 performs various processes such as a process of acquiring training data for machine learning and a process of generating a learning model using the training data.

[0018] The storage unit 12 is configured using a large-capacity storage device such as a hard disk, for example. The storage unit 12 stores various programs executed by the processing unit 11 and various data necessary for the processing of the processing unit 11. In the present embodiment, the storage unit 12 stores the program 12a executed by the processing unit 11, and also stores training data 12b for performing machine learning, a learning model 5 generated by machine learning, and the like.

[0019] In this embodiment, the program 12a is provided in a form recorded on a recording medium 99 such as a memory card or an optical disk, and the learning device 1 reads the program 12a from the recording medium 99 and stores it in the storage unit 12. However, the program 12a may be written in the storage unit 12, for example, at the manufacturing stage of the learning device 1. Also, for example, the program 12a may be acquired by the learning device 1 through communication from a remote server device or the like. For example, the program 12a may be read by a writing device from what is recorded on the recording medium 99 and written into the storage unit 12 of the learning device 1. The program 12a may be provided in a form of distribution via a network or in a form recorded on the recording medium 99.

[0020] The training data 12b is a large number of data in which image data of a handwashing operation is associated with information such as a label indicating the type of this handwashing operation. The training data 12b is, for example, created in advance by the designer or the like of the information processing system according to this embodiment and stored in the storage unit 12 of the learning device 1. In the information processing system according to this embodiment, what has a plurality of hierarchical labels associated with the image of the handwashing operation is used as the training data 12b. Details of the plurality of hierarchical labels will be described later.

[0021] The learning model 5 is generated by the learning device 1 performing machine learning processing using the training data 12b. In the storage unit 12, information regarding the structure of the learning model and information such as parameters obtained by machine learning are stored as the learning model 5. Details of the configuration of the learning model 5 used in the information processing system according to this embodiment will be described later.

[0022] The communication unit 13 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN (Local Area Network), and the Internet. In the present embodiment, the communication unit 13 communicates with one or more monitoring devices 3 via the network N. The communication unit 13 transmits the data given by the processing unit 11 to other devices and gives the data received from other devices to the processing unit 11. The learning device 1 according to the present embodiment transmits a learning model 5 generated by machine learning to the monitoring device 3 by communication. However, the transfer of the learning model 5 between the learning device 1 and the monitoring device 3 may be performed not by communication but via, for example, a recording medium or the like.

[0023] Note that the storage unit 12 may be an external storage device connected to the learning device 1. Further, the learning device 1 may be a multi-computer including a plurality of computers, or may be a virtual machine virtually constructed by software. Further, the learning device 1 is not limited to the above configuration, and may include, for example, a reading unit that reads information stored in a portable storage medium, an input unit that receives an operation input, or a display unit that displays an image.

[0024] Further, in the learning device 1 according to the present embodiment, when the processing unit 11 reads and executes the program 12a stored in the storage unit 12, a training data acquisition unit 11a, a learning model generation unit 11b, a learning model transmission unit 11c, etc. are realized in the processing unit 11 as software functional units. In this figure, a functional unit that performs processing related to the generation of the learning model 5 is illustrated as a functional unit of the processing unit 11, and functional units related to other processing are not illustrated.

[0025] The training data acquisition unit 11a acquires the training data 12b stored in the storage unit 12 by reading it, and performs a process of giving the acquired training data 12b to the learning model generation unit 11b. The training data acquisition unit 11a performs various pre-processings such as resizing, value normalization, or regularization of the image data included in the training data 12b, for example, and gives the pre-processed training data 12b to the learning model generation unit 11b.

[0026] The learning model generation unit 11b generates a desired learning model 5 by performing machine learning processing using the training data 12b acquired by the training data acquisition unit 11a. Machine learning is a process of setting appropriate values for a plurality of parameters of an unlearned learning model whose configuration is determined in advance. The learning model generation unit 11b of the learning device 1 according to the present embodiment generates a learning model by machine learning of so-called supervised learning using training data 12b in which image data and labels are associated, and determines parameters by methods such as the stochastic gradient descent method and the error backpropagation method. Further, the learning model generation unit 11b may periodically perform relearning using new training data 12b on the generated learning model 5 to update the learning model 5.

[0027] The learning model transmission unit 11c performs a process of transmitting the learning model 5 generated by the learning model generation unit 11b to the monitoring device 3. In the information processing system according to the present embodiment, the learned learning model 5 is exchanged between the learning device 1 and one or more monitoring devices 3 through communication via the network N. However, the learning model 5 may be exchanged via a recording medium. The learning model transmission unit 11c transmits the learned learning model 5 spontaneously or in response to a request from the monitoring device 3 after the generation of the learning model 5 by the learning model generation unit 11b is completed.

[0028] FIG. 5 is a schematic diagram for explaining the outline of the learning model 5 according to the present embodiment. In the information processing system according to the present embodiment, the learning model 5 generated by the learning device 1 receives an input of a "captured image" of a handwashing operation and outputs two types of classification results: a "classification result of layer 1" and a "classification result of layer 2". The classification results output by the learning model 5 are, for example, those in which N values are associated with N types of handwashing operations, and each value indicates the likelihood (probability, confidence level, etc.) that it is the corresponding type of handwashing operation. Each value is a value in the range of 0 to 1, and it can be expected that the sum of the N output values is 1 (= 100%).

[0029] The learning model 5 according to this embodiment outputs two types of classification results: the classification result of layer 1 and the classification result of layer 2. When the number of types of handwashing operations classified as layer 1 is N1 and the number of types of handwashing operations classified as layer 2 is N2, it can be expected that the sum of the N1 output values of layer 1 is 1 and the sum of the N2 output values of layer 2 is 1.

[0030] The learning model according to this embodiment outputs, as the classification result, the likelihood (probability, confidence level, etc.) of the type of handwashing operation. Such output values can be realized, for example, by using a softmax function. However, the learning model may output a value different from the likelihood as the classification result. Also, in this example, the learning model outputs the classification results of two layers, but it is not limited to this, and the learning model may be configured to output the classification results of three or more layers.

[0031] FIG. 6 is a schematic diagram for explaining the handwashing operations classified by the learning model 5 according to this embodiment. In this example, the learning model 5 classifies, as layer 1, the parts where handwashing is performed, such as "back of the hand (left)", "back of the hand (right)", "between fingertips and nails (left)", and "between fingertips and nails (right)". Also, in this example, the learning model 5 classifies, as layer 2, the ways of washing the parts specified in layer 1, such as "washing method 1", "washing method 2",..., "washing method 14". In this example, "washing methods 1 to 4" of layer 2 belong to the lower level of "back of the hand (left)" of layer 1, "washing methods 5 to 8" of layer 2 belong to the lower level of "back of the hand (right)" of layer 1, "washing methods 9 to 11" of layer 2 belong to the lower level of "between fingertips and nails (right)" of layer 1, and "washing methods 12 to 14" of layer 2 belong to the lower level of "back of the hand (right)" of layer 1.

[0032] FIG. 7 is a schematic diagram showing an example of washing methods 1 to 4 of layer 2. The images of washing methods 1 to 4 shown in FIG. 7 are photographed images of washing methods corresponding to "washing methods 1 to 4" belonging to the lower level of "back of the hand (left)" shown in FIG. 6. "Washing method 1" is a washing method in which the right hand is placed on the back of the left hand, and the back of the left hand is washed with the fingers of the right hand sandwiched between the fingers of the left hand. "Washing method 2" is a washing method in which the right hand with the fingers aligned and extended is placed on the back of the left hand with the fingers aligned and extended in the same direction, and the back of the left hand is washed. "Washing method 3" is a washing method in which the right hand with the fingers aligned and extended is placed on the back of the left hand with the fingers aligned and extended in a crossed direction, and the back of the left hand is washed. "Washing method 4" is a washing method in which the back of the left hand is turned downward, and the right hand is placed under the left hand to wash the back of the left hand. Note that the washing methods shown in FIG. 7 are only examples and are not limited thereto.

[0033] In this example, at least four of "washing methods 1 to 4" of layer 2 are associated with "back of the hand (left)" of layer 1. Although the illustration of the details of the washing methods is omitted, similarly, "washing methods 5 to 8" of layer 2 are associated with "back of the hand (right)" of layer 1. In this case, "washing methods 5 to 8" of "back of the hand (right)" can be the ones with the left-right hand positional relationship reversed in the images of "washing methods 1 to 4" shown in FIG. 7.

[0034] Similarly, for "fingertips·between nails (left)" and "fingertips·between nails (right)" etc. of layer 1, one or more washing methods of layer 2 are associated. The washing methods of layer 2 corresponding to "fingertips·between nails (left)" and "fingertips·between nails (right)" etc. can be different from the "washing methods 1 to 4" of layer 2 corresponding to "back of the hand (left)". Therefore, the number of washing methods of layer 2 corresponding to "back of the hand (left)" of layer 1 and the number of washing methods of layer 2 corresponding to "fingertips·between nails (left)" and "fingertips·between nails (right)" etc. of layer 1 can be different numbers. In this example, for "fingertips·between nails (left)" and "fingertips·between nails (right)" of layer 1, three washing methods of layer 2 are respectively associated with the lower levels.

[0035] How the learning model 5 classifies handwashing operations at each layer is determined by the designer of the information processing system according to the present embodiment at the system design stage or the like. The number of layers of the classification of handwashing operations shown in FIGS. 5 to 7, the number of classifications in each layer, and the items in each layer are examples and are not limited thereto.

[0036] FIG. 8 is a schematic diagram showing another example of the classification of handwashing operations, and an example of classifying handwashing operations into three layers is shown. In the example shown in FIG. 8, the learning model 5 classifies parts such as "palm", "back of hand", "between fingertips and nails", "between fingers", "thumb", and "wrist" as layer 1. Further, the learning model 5 classifies whether the part of handwashing is on the left, right, or both as layer 2, such as "left and right", "left", and "right". Further, the learning model 5 classifies the washing methods according to the parts specified in layer 1 and the left and right specified in layer 2, such as "washing method 1", "washing method 2", "washing method 3", and "washing method 4" as layer 3. The example shown in FIG. 8 corresponds to the case where the classification of layer 1 shown in FIG. 6 is subdivided into two layers.

[0037] In this example, "left and right" of layer 2 belongs to the lower layer of "palm" of layer 1, and "washing method 1" and "washing method 2" of layer 3 belong to the lower layer of "left and right" of layer 2. Further, "left" and "right" of layer 2 belong to the lower layer of "wrist" of layer 1, and "washing method 3" and "washing method 4" of layer 3 belong to the lower layer of "left" of layer 2. In this example, the number of classifications in layer 2 is not three types of "left and right", "left", and "right", but a number larger than the number of classifications in layer 1, such as "left and right" belonging to "palm", "left" and "right" belonging to "back of hand", "left" and "right" belonging to "between fingertips and nails",.... Similarly, the number of classifications in layer 3 is a number larger than the number of classifications in layer 2.

[0038] FIG. 9 is a schematic diagram showing an example of washing methods 1 to 4 at level 3. The images of washing methods 1 to 4 shown in FIG. 9 are captured images corresponding to "washing methods 1 and 2" which belong to the lower levels of "palm" at level 1 and "left and right" at level 2 shown in FIG. 8, and "washing methods 3 and 4" which belong to the lower levels of "wrist" at level 1 and "left" at level 2. "Washing method 1" is a washing method in which the palms of both hands are washed without tilting. "Washing method 2" is a washing method in which the palms of both hands are tilted and washed. "Washing method 3" is a washing method in which the left wrist is grasped and washed with the right hand. "Washing method 4" is a washing method in which the left wrist is grasped and washed with the left hand. Note that the washing methods shown in FIG. 9 are only examples and are not limited thereto.

[0039] FIG. 10 shows still another example of classifying handwashing operations into three levels. In the example shown in FIG. 10, the learning model 5 classifies whether handwashing is being performed or not, such as "washing hands" and "not washing hands" as level 1. Also, the learning model 5 classifies the parts where handwashing is being performed, such as "back of hand (left)", "back of hand (right)", "between fingertips and nails (left)", and "between fingertips and nails (right)" as level 2 corresponding to "washing hands" at level 1. Also, the learning model 5 classifies operations other than handwashing, such as "washing with water" and "soap", as level 2 corresponding to "not washing hands" at level 1. Also, the learning model 5 classifies the washing methods of the parts specified at level 2, such as "washing method 1", "washing method 2", "washing method 3", and "washing method 4", as level 3 corresponding to "not washing hands" at level 1. Also, in this example, the learning model 5 does not define level 3 corresponding to "not washing hands" at level 1. The example shown in FIG. 10 corresponds to adding a level for classifying "washing hands" and "not washing hands", which are higher levels than the two-level classification shown in FIG. 6.

[0040] The training data 12b used by the learning device 1 to generate the learning model 5 is data in which images of handwashing operations are associated with labels that are the classification results for each layer. The images of handwashing operations are, for example, the images shown in FIGS. 7 and 9. The labels for each layer are, for example, when classifying N handwashing operations, a vector having N values where the value of "1" is set for one correct operation and the value of "0" is set for the other N - 1 operations. In the case of the example shown in FIG. 6, for one image, the vector of the label for layer 1 and the vector of the label for layer 2 are associated. The learning device 1 determines the values of the internal parameters of the learning model 5 so that when an image of the training data 12b is input, the output of the corresponding label is obtained as the classification result, thereby generating the learning model 5.

[0041] FIG. 11 is a schematic diagram showing a configuration example of the learning model 5 according to the present embodiment. Note that the learning model 5 in this example is configured assuming classification in two layers as shown in FIG. 6. The learning model 5 according to the present embodiment includes three layers (layers): a CNN (Convolutional Neural Network) layer 51, a layer 1 output layer 52, and a layer 2 output layer 53. The CNN layer 51 has a convolutional layer that performs filter processing of a predetermined size on the input captured image and a pooling layer that reduces the size of the image, and extracts and outputs the feature amount of the captured image by performing the processing of the convolutional layer and the pooling layer one or more times. The feature amount output by the CNN layer 51 is input to the layer 1 output layer 52 and the layer 2 output layer 53, respectively.

[0042] The layer 1 output layer 52 and the layer 2 output layer 53 can each be configured as, for example, a fully connected layer of a neural network. The layer 1 output layer 52 and the layer 2 output layer 53 have the same input size and different output sizes. The layer 1 output layer 52 outputs classification results such as "back of hand (left)", "back of hand (right)", "between fingertips and nails (left)", and "between fingertips and nails (right)" of layer 1 shown in FIG. 6. The layer 2 output layer 53 outputs classification results such as "washing method 1", "washing method 2", "washing method 3", and "washing method 4" of layer 2 shown in FIG. 6. When obtaining the classification result as a probability (likelihood, confidence level, etc.), the softmax function is applied to the output of the layer 1 output layer 52 and the output of the layer 2 output layer 53, but the layer that performs the operation by the softmax function in FIG. 11 is not shown.

[0043] Note that the softmax function for outputting the classification result as a probability may not be necessary at the stage of performing the classification process of the handwashing operation using the learned learning model 5 even if it is necessary at the learning stage of the learning model 5. In the learning stage, for example, a loss function that calculates the error of the classification result and the label based on the probability output by the softmax function as the classification result and the label that is the correct answer of the classification result is used, and the parameters can be updated by backpropagating the error calculated by the loss function to each layer in the learning model 5. In the learning model 5 of this example, for example, a loss function may be used individually for the classification result of layer 1 and the classification result of layer 2, or for example, a single loss function may be used for the classification result of layer 1 and the classification result of layer 2 together.

[0044] Also, in the learning model 5 according to this embodiment, the layer that extracts the feature amount from the captured image is the CNN layer 51, but it is not limited to this, and for example, a fully connected layer or a layer having a configuration other than these may be adopted. Also, the layer 1 output layer 52 and the layer 2 output layer 53 that output the classification results of each layer based on the extracted feature amounts are fully connected layers, but it is not limited to this, and for example, a CNN layer or a layer having a configuration other than these may be adopted.

[0045] FIG. 12 is a schematic diagram showing another configuration example of the learning model 5. The learning model 5 shown in FIG. 12 has a configuration in which the output of the CNN layer 51 is input to the layer 1 output layer 52, and the output of the layer 1 output layer 52 is input to the layer 2 output layer 53. The CNN layer 51 outputs the feature amount of the input captured image. The layer 1 output layer 52 outputs the classification result of layer 1 based on the input feature amount. The layer 2 output layer 53 outputs the classification result of layer 2 based on the classification result of layer 1.

[0046] FIG. 13 is a schematic diagram showing another configuration example of the learning model 5. The learning model 5 shown in FIG. 13 has a configuration in which the output of the CNN layer 51 is input to the layer 2 output layer 53, and the output of the layer 2 output layer 53 is input to the layer 1 output layer 52. The CNN layer 51 outputs the feature amount of the input captured image. The layer 2 output layer 53 outputs the classification result of layer 2 based on the input feature amount. The layer 1 output layer 52 outputs the classification result of layer 1 based on the classification result of layer 2. The configuration of the learning model 5 shown in FIG. 13 is a configuration in which the arrangement of the layer 1 output layer 52 and the layer 2 output layer 53 is reversed with respect to the configuration of the learning model 5 shown in FIG. 12. This is suitable when the output size of the layer 2 output layer 53 is larger than the output size of the layer 1 output layer 52.

[0047] FIG. 14 is a schematic diagram showing another configuration example of the learning model 5. The learning model 5 shown in FIG. 14 has a configuration in which the output of the CNN layer 51 is input to the LSTM (Long Short Term Memory) layer 54, and the LSTM layer 54 outputs the classification result of layer 1 and the classification result of layer 2 as sequential data. Although two LSTM layers 54 are shown in FIG. 14, the two LSTM layers 54 are actually the same. The LSTM layer 54 outputs the classification result of layer 1 based on the feature amount output by the CNN layer 51, and outputs the classification result of layer 2 based on the feature amount output by the CNN layer 51 and the classification result of layer 1.

[0048] FIG. 15 is a flowchart showing the procedure of the processing performed by the learning device 1 according to the present embodiment. The training data acquisition unit 11a of the processing unit 11 of the learning device 1 acquires one or a plurality of pieces of training data from among the plurality of pieces of training data 12b stored in the storage unit 12 (step S1). The learning model generation unit 11b of the processing unit 11 performs learning processing of the learning model 5 using the training data read in step S1 (step S2). At this time, when the learning model generation unit 11b inputs the captured image included in the training data to the learning model 5, the learning model generation unit 11b updates the parameters of the learning model 5 so that the output value corresponding to the correct label of each layer included in the training data approaches 1.0 and the other output values approach 0. The parameters of the learning model 5 are, for example, the coefficients of the filter processing and compression processing of the CNN layer 51, the weights and biases of the fully connected layer, etc., and the learning device 1 performs learning processing so as to optimize these values.

[0049] The learning model generation unit 11b determines whether there is any unprocessed training data that has not been used for learning processing among all the training data 12b stored in the storage unit 12 (step S3). If there is unprocessed training data (S3: YES), the learning model generation unit 11b returns the process to step S1, acquires another piece of training data, and repeats the learning process. If there is no unprocessed training data (S3: NO), the learning model generation unit 11b ends the learning of the learning model 5, stores the parameters of the learning model 5, etc. in the storage unit 12 (step S4), and ends the process.

[0050] By performing these processes by the learning device 1, when a captured image is input, a learning model 5 is generated that classifies the type of handwashing operation performed by the operator in the captured image at a plurality of levels. Note that the already learned learning model 5 can also be relearned by performing the above-described processes. In this case, the learning device 1 can generate a learning model 5 with higher discrimination accuracy. Note that in the above-described process, learning is ended when all of the training data 12b stored in the storage unit 12 is used, but the present invention is not limited to this, and the learning of the learning model 5 may be repeated using the same training data 12b repeatedly.

[0051] The learning model 5 generated by the learning device 1 performing machine learning processing is provided to the monitoring device 3 via a communication or recording medium or the like. The monitoring device 3 acquires the learned learning model 5 from the learning device 1 and stores it in its own storage unit or the like, and uses this learning model 5 in the process of monitoring the handwashing operation of the operator.

[0052] <Monitoring process of handwashing operation> In the information processing system according to the present embodiment, the monitoring device 3 installed in the handwashing area of the facility or the like monitors the handwashing operation of the operator using the learning model 5 generated by the learning device 1. FIG. 16 is a block diagram showing the configuration of the monitoring device 3 according to the present embodiment. The monitoring device 3 according to the present embodiment includes a processing unit 31, a storage unit 32, a camera 33, a communication unit 34, a display unit 35, and the like. The processing unit 31 is configured using one or more arithmetic processing devices such as a CPU, MPU, or GPU, a ROM, a RAM, and the like. The processing unit 31 reads and executes the program 32a stored in the storage unit 32, and performs various processes such as a process of photographing the handwashing operation of the operator, a process of specifying the type of handwashing operation based on the photographed image, and a process of displaying information related to the handwashing operation of the operator.

[0053] The storage unit 32 is configured using a non-volatile storage device such as a flash memory or a hard disk. The storage unit 32 stores various programs executed by the processing unit 31 and various data necessary for the processing of the processing unit 31. In the present embodiment, the storage unit 32 stores the program 32a executed by the processing unit 31 and stores the learning model 5 generated by machine learning and the like.

[0054] In the present embodiment, the program 32a is provided in a form recorded on a recording medium 98 such as a memory card or an optical disk, and the monitoring device 3 reads the program 32a from the recording medium 98 and stores it in the storage unit 32. However, the program 32a may be written in the storage unit 32 at the manufacturing stage of the monitoring device 3, for example. Also, for example, the monitoring device 3 may acquire the program 32a distributed by a remote server device or the like through communication. For example, the program 32a may be read by a writing device from what is recorded on the recording medium 98 and written into the storage unit 32 of the monitoring device 3. The program 32a may be provided in a form of distribution via a network or in a form recorded on the recording medium 98.

[0055] The learning model 5 is a learned learning model obtained by machine learning by the learning device 1. The learning model 5 is assumed to be used as a program module that functions as part of artificial intelligence software. In the storage unit 32, information regarding the structure of the learning model 5 and information such as parameters obtained by machine learning are stored. The learning model 5 is acquired from the learning device 1 via the recording medium 98 or communication, together with or separately from the program 32a.

[0056] The camera 33 is an imaging device having a lens, an imaging element, etc., and acquires image data of a subject image through the lens. The camera 33 performs imaging according to an instruction from the processing unit 31 and sequentially acquires image data (captured image) of one frame. The camera 33 repeatedly performs imaging at a frequency of, for example, 60 frames, 30 frames, or 15 frames per second to acquire image data. That is, the monitoring device 3 according to the present embodiment can perform shooting of a moving image with the camera 33. In addition to being incorporated in the monitoring device 3, the camera 33 may be configured to be externally attached to the monitoring device 3 or may be connected to the monitoring device 3 via a network such as a LAN or the Internet. In this case, the monitoring device 3 includes a connection part capable of connecting an external camera or a camera communication part capable of wireless communication with an external camera, and acquires video data captured by the external camera via the connection part or the camera communication part.

[0057] The communication unit 34 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN, the Internet, etc. In the present embodiment, the communication unit 34 communicates with the learning device 1 via the network N. The communication unit 34 transmits the data given from the processing unit 31 to other devices and gives the data received from other devices to the processing unit 31. The monitoring device 3 according to the present embodiment acquires the learned learning model 5 by communicating with the learning device 1 and stores it in the storage unit 32. However, the monitoring device 3 may acquire the learning model 5 via a recording medium 98 or the like. In this case, the monitoring device 3 may not include the communication unit 34.

[0058] The display unit 35 is a liquid crystal display, an organic EL display, or the like, and displays various information according to an instruction from the processing unit 31. In the present embodiment, the monitoring device 3 displays information such as the progress of the handwashing operation by the operator and whether the handwashing operation is appropriately performed on the display unit 35.

[0059] Further, in the monitoring device 3 according to the present embodiment, when the processing unit 31 reads and executes the program 32a stored in the storage unit 32, the operation specifying processing unit 31a, the display processing unit 31b, etc. are realized in the processing unit 31 as software functional units. The operation specifying processing unit 31a photographs the handwashing operation by the operator with the camera 33, and classifies the type of the handwashing operation using the learned learning model 5 for the photographed image obtained by the photographing, thereby performing the process of specifying the handwashing operation being performed by the operator. Further, the operation specifying processing unit 31a determines the progress of the handwashing operation by the operator and whether the handwashing operation by the operator is appropriately performed based on the specific result of the type of the handwashing operation. The display processing unit 31b performs the process of displaying on the display unit 35 information such as the specific result of the handwashing operation by the operation specifying processing unit 31a, the determination result of the progress of the handwashing operation, and the determination result of whether the handwashing operation is appropriately performed.

[0060] The monitoring device 3 according to this embodiment inputs the data of the captured image (the data of a single still image) obtained by capturing the handwashing operation by the operator with the camera 33 into the learned learning model 5. The learning model 5 performs arithmetic processing using parameters determined by learning on the input captured image, and outputs classification results of multiple layers as a result of the arithmetic processing. The monitoring device 3 acquires the classification results of multiple layers output by the learning model 5, and identifies the handwashing operation of the operator based on the combination of the classification results of multiple layers and the like.

[0061] Hereinafter, the processing performed by the monitoring device 3 will be described assuming that the learning model 5 performs the classification of layer 1 and layer 2 shown in FIG. 6. In this example, the monitoring device 3 inputs the image captured by the camera 33 into the learning model 5, and acquires the classification result of layer 1 and the classification result of layer 2 output by the learning model 5. The learning model 5 outputs numerical values from 0 to 1 for each handwashing operation such as "back of hand (left)", "back of hand (right)", "between fingertips and nails (left)", "between fingertips and nails (right)",... The numerical value corresponding to each handwashing operation indicates the likelihood (probability, confidence level, etc.) that the operator is performing the handwashing operation. The monitoring device 3 searches for the maximum value among the output values corresponding to each handwashing operation in layer 1, and sets one handwashing operation corresponding to the maximum value as the classification result of the handwashing operation in layer 1. Similarly, the monitoring device 3 sets the handwashing operation corresponding to the maximum value among the multiple values output by the learning model 5 for layer 2 as the classification result of the handwashing operation in layer 2.

[0062] The monitoring device 3 identifies the handwashing operation of the operator based on the classification result of the handwashing operation at layer 1 and the classification result of the handwashing operation at layer 2. In this example, it is assumed that the type of the handwashing operation of the operator identified by the monitoring device 3 is the same as the type of the handwashing operation at layer 1. That is, the monitoring device 3 identifies whether the handwashing operation of the operator is "back of hand (left)", "back of hand (right)", "between fingertips and nails (left)", "between fingertips and nails (right)",... based on the classification result at layer 1 and the classification result at layer 2. However, this is just an example. The type of the handwashing operation identified by the monitoring device 3 may be the same as the type of the handwashing operation at layer 2, or may be different from the types of the handwashing operations at layer 1 and layer 2.

[0063] Based on the classification result of layer 2, which is the lowest layer, the monitoring device 3 identifies which type of handwashing operation at layer 1, which is the upper layer, this classification result corresponds to. For example, in FIG. 6, when the classification result of layer 2 is "washing method 1", the monitoring device 3 can identify that the type of the handwashing operation at layer 1 corresponding to this classification result is "back of hand (left)". The monitoring device 3 determines whether the type of the handwashing operation at layer 1 of the upper layer identified according to the classification result of layer 2 matches the type of the handwashing operation of the classification result (= identification result) of layer 1 by the learning model 5. When the two types match, the monitoring device 3 finally identifies the type of the handwashing operation that matches as the type of the handwashing operation of the operator based on the identification result of layer 1 and the identification result of layer 2.

[0064] When the type of handwashing operation specified based on the classification result of layer 2 does not match the type of handwashing operation specified based on the classification result of layer 1, the monitoring device 3 identifies the handwashing operation based on the previous captured image. Note that the monitoring device 3 repeatedly captures the handwashing operation of the operator by the camera 33, for example, at a frequency of dozens of frames per second, and identifies the handwashing operation based on the captured image each time the capture is performed. The monitoring device 3 determines whether either the type of handwashing operation specified based on the classification result of layer 2 or the type of handwashing operation specified based on the classification result of layer 1 matches the type of handwashing operation identified based on the previous captured image. When either the type of handwashing operation specified based on the classification result of layer 2 or the type of handwashing operation specified based on the classification result of layer 1 matches the type of handwashing operation identified based on the previous captured image, the monitoring device 3 finally identifies the matching type of handwashing operation as the type of the operator's handwashing operation.

[0065] When neither the type of handwashing operation specified based on the classification result of layer 2 nor the type of handwashing operation specified based on the classification result of layer 1 matches the type of handwashing operation identified based on the previous captured image, the monitoring device 3 identifies the handwashing operation based on a plurality of numerical values output by the learning model 5 for layer 2, which is the lowest layer. The monitoring device 3 calculates, for each type of handwashing operation in layer 1, which is the highest layer, for example, the average value of the numerical values output by the learning model 5 for one or more layers 2 belonging to the lower layer. The monitoring device 3 compares the average values of the output values of layer 2 calculated for each layer 1 and identifies the type of handwashing operation in layer 1 with the largest average value as the type of the operator's handwashing operation. Note that in this example, the average value of the output values of layer 2 is used, but it is not limited to this. For example, various statistical values such as a weighted total value, a minimum value, and a median value can be used.

[0066] FIG. 17 is a flowchart showing the procedure of the operation identification process performed by the monitoring device 3 according to the present embodiment. The operation identification unit 31a of the processing unit 31 of the monitoring device 3 according to the present embodiment repeatedly performs the processes shown in the flowchart of FIG. 17. The operation identification unit 31a captures an image of the handwashing operation by the operator with the camera 33 (step S11). The operation identification unit 31a inputs the captured image of the handwashing operation obtained by the shooting of the camera 33 into the learned learning model 5 (step S12). The operation identification unit 31a acquires the classification results of layer 1 and layer 2 output by the learning model 5 for the input of the captured image (step S13).

[0067] The operation identification unit 31a determines whether or not the type of the handwashing operation specified by the classification result of layer 1 matches the type of the handwashing operation specified by the classification result of layer 2 (step S14). In this example, the type of the handwashing operation specified by the classification result of layer 1 is equal to the classification result of layer 1. Also in this example, the type of the handwashing operation specified by the classification result of layer 2 is the type of the handwashing operation of the upper layer 1 to which layer 2 belongs. When the identification results of layer 1 and layer 2 match (S14: YES), the operation identification unit 31a identifies the type of the handwashing operation, which is the matching identification result in layer 1 and layer 2, as the handwashing operation of the operator captured by the camera 33 (step S15), and ends the operation identification process.

[0068] If the specific results of level 1 and level 2 do not match (S14: NO), the operation specific processing unit 31a acquires the specific result of the handwashing operation based on the image captured by the camera 33 last time (step S16). For this purpose, the operation specific processing unit 31a stores and holds the specific result of the handwashing operation in the storage unit 32 or the like for at least one time. The operation specific processing unit 31a determines whether any of the current specific results of level 1 or level 2 match the previous specific result (step S17). If any of the current specific results of level 1 or level 2 match the previous specific result (S17: YES), the operation specific processing unit 31a specifies the type of the handwashing operation of the operator shown in the current captured image as the same type of handwashing operation as that specified based on the previous captured image (step S18), and ends the operation specific processing.

[0069] If none of the current specific results of level 1 or level 2 match the previous specific result (S17: NO), the operation specific processing unit 31a calculates the average value of the values output by the learning model 5 for one or more levels 2 belonging to each type of handwashing operation of level 1 (step S19). The operation specific processing unit 31a compares the average values of the output values of level 2 calculated for each level 1, and specifies the type of handwashing operation of level 1 with the maximum average value as the handwashing operation of the operator captured by the camera 33 (step S20), and ends the operation specific processing.

[0070] The monitoring device 3 that specifies the handwashing operation of the operator determines the progress status of this type of handwashing operation, for example, by measuring the time during which the operation is performed for each specified type of handwashing operation. The monitoring device 3 displays various information including the determined progress status of the handwashing operation on the display unit 35. FIG. 18 is a schematic diagram showing an example of the screen displayed by the monitoring device 3 on the display unit 35. In the screen of this example, a display area for the captured image is provided on the upper left side, a display area for the progress status is provided on the upper right side, and a display area for the reference image is provided on the lower side.

[0071] In the display area of the captured image provided in the upper left side of the screen, an image of the handwashing operation of the operator captured by the camera 33 is displayed. The monitoring device 3 repeatedly captures images with the camera 33, and by updating the image to be displayed each time the camera 33 performs a capture, the handwashing operation of the operator can be displayed as a real-time moving image on the display unit 35. Also, in the present embodiment, the monitoring device 3 displays a message to the operator performing the handwashing operation below the captured image. In this example, a message "It is different from the model way of washing" is displayed below the captured image. Various messages such as "It is not being washed correctly" and "The washing method is insufficient" can be adopted as the message.

[0072] Based on the type of handwashing operation identified based on the captured image, the monitoring device 3 determines whether the handwashing operation is correct, and displays the above message based on the determination result. The monitoring device 3 can determine whether the handwashing operation of the operator is correct, for example, by determining whether the identified type of handwashing operation is included in the type of handwashing operation defined in advance as a model. Also, when performing this determination, the monitoring device 3 may use not only the identified type of handwashing operation but also the output values or classification results of each layer obtained from the learning model 5 during the identification process. The monitoring device 3 may, for example, store in advance in the storage unit 32 one or more messages associated with the determination result or the like together with the program 32a, and read out the message to be displayed from the storage unit 32 according to the determination result. Also, for example, a machine-learned model that generates a message according to the identification result of the type of handwashing operation or the output value of the learning model 5 may be used to generate the message to be displayed.

[0073] In the display area for the progress status provided in the upper right side of the screen, for example, progress bars (indicators) showing the progress of six types of handwashing operations are displayed. When the type of handwashing operation identified using the learning model 5 corresponds to any of the six types of handwashing operations indicating the progress degree, the monitoring device 3 calculates the progress degree by measuring the time (execution time) during which this handwashing operation is being performed. For example, when the monitoring device 3 repeatedly performs shooting by the camera 33 at a frequency of 10 times per second (10 frames) and identifies the handwashing operation using the learning model 5 for each frame, every time the handwashing operation is identified, 0.1 second is added to the execution time for the type of handwashing operation identified. Also, for example, when the monitoring device performs shooting at a frequency of 30 frames per second and identifies the handwashing operation every 3 frames, 0.1 second is added to the execution time for the type of handwashing operation identified. Thereby, the monitoring device 3 can measure the execution time for the six types of handwashing operations.

[0074] Also, for the six types of handwashing operations, the time (required time) required to perform this handwashing operation is determined in advance, and the monitoring device 3 stores in advance in the storage unit 32 the required time for each handwashing operation. The monitoring device 3 calculates the progress degree by calculating, for example, the ratio of the execution time to the required time based on the measured execution time of the handwashing operation and the corresponding required time (progress degree = execution time / required time). Note that the method for calculating the progress degree by the monitoring device 3 is not limited to the above method, and various methods may be adopted. Every time the monitoring device 3 calculates the progress degree for each handwashing operation, it displays the calculated progress degree using the progress bar corresponding to each handwashing operation provided in the display area for the progress status. In the illustrated screen, the required time set for each handwashing operation is indicated by a dashed line, and the progress degree is displayed using a progress bar in which the execution time of each handwashing operation is indicated by hatching.

[0075] In the display area for the demonstration images provided at the lower part of the screen, demonstration images for six types of handwashing operations that notify the progress status are arranged and displayed in a matrix. The monitoring device 3 stores in advance in the storage unit 32 the data of the demonstration images to be displayed in the display area for the demonstration images. In the illustrated screen, the numbers 1 to 6 attached to the demonstration images correspond to the numbers 1 to 6 attached to the progress bar. In the illustrated example, the handwashing operation numbered 1 is "wash the palm and the ventral surface of the fingers", the handwashing operation numbered 2 is "wash the back of the hand and the back of the fingers", the handwashing operation numbered 3 is "wash between the fingers (side surface) and the groin (base)", the handwashing operation numbered 4 is "wash the swollen part between the thumb and the base of the thumb", the handwashing operation numbered 5 is "wash the fingertips", and the handwashing operation numbered 6 is "wash the wrist (inner side, side surface, outer side)".

[0076] When the monitoring device 3 identifies the handwashing operation being performed by the operator, it displays the number of the progress bar corresponding to the identified handwashing operation and the number of the demonstration image in a manner different from other numbers. Thereby, the monitoring device 3 can notify the operator which handwashing operation is currently being carried out. In the illustrated screen, number 2 in the progress bar and the demonstration image is displayed in reverse, indicating that the handwashing operation numbered 2 is being executed.

[0077] FIG. 19 is a flowchart showing the procedure of the display process performed by the monitoring device 3 according to the present embodiment. The processing unit 31 of the monitoring device 3 according to the present embodiment determines whether or not the handwashing operation in the handwashing sink has started (step S31). At this time, for example, when the fingers of the operator are detected by a human sensor or the like installed in the handwashing sink, the processing unit 31 may determine that the handwashing operation has started. In this case, when starting the handwashing operation, the operator can input the start of the handwashing operation to the monitoring device 3 by holding his / her fingers in front of the human sensor. As such a human sensor, a sensor that senses the user's fingers when automatically discharging and stopping water from the faucet can be used. Also, for example, the processing unit 31 may be provided with an operation button for inputting the start of the handwashing operation in the handwashing sink, and may determine that the handwashing operation has started when the operation button is operated. Also, for example, the processing unit 31 may determine that the handwashing operation has started when the fingers of the operator appear in the image captured by the camera 33 or when the fingers of the operator who appears are making a predetermined movement. In this case, the operator can input the start of the handwashing operation to the monitoring device 3 by putting his / her fingers within the shooting range of the camera 33 or starting the handwashing operation within the shooting range of the camera 33. The processing unit 31 may determine the start of the handwashing operation by other methods, or may continuously repeat this process without determining the start of the handwashing operation.

[0078] When it is determined that the handwashing operation has not started (S31: NO), the processing unit 31 waits until the handwashing operation starts. When it is determined that the handwashing operation has started (S31: YES), the processing unit 31 displays the handwashing screen shown in FIG. 18 on the display unit 35 (step S32). The operation specifying processing unit 31a of the processing unit 31 specifies the handwashing operation of the operator based on the image captured by the camera 33 (step S33). Note that the processing performed in step S33 is the processing whose procedure is shown in the flowchart of FIG. 17. Based on the result of specifying the handwashing operation by the operation specifying processing unit 31a, the display processing unit 31b of the processing unit 31 determines whether the handwashing operation of the operator is a correct handwashing operation (step S34). The display processing unit 31b can determine whether the handwashing operation of the operator is correct by determining whether the specified handwashing operation matches the handwashing operation defined as correct in advance.

[0079] When the handwashing operation of the operator is a correct handwashing operation (S34: YES), the display processing unit 31b measures the execution time of this handwashing operation based on, for example, the number of times this handwashing operation is specified (that is, the number of frames in which this handwashing operation is captured by the camera 33) (step S35). Based on the measured execution time, the display processing unit 31b updates the display on, for example, a progress bar provided on the handwashing screen of the display unit 35 to display the progress of this handwashing operation (step S36), and proceeds to step S38. When the handwashing operation of the operator is not a correct handwashing operation (S34: NO), the display processing unit 31b displays, for example, at the lower part of the display area of the captured image on the handwashing screen of the display unit 35, a message notifying that the handwashing operation of the operator is incorrect (step S37), and proceeds to step S38.

[0080] The processing unit 31 determines whether or not the handwashing operation of the operator has been completed (step S38). At this time, for example, regarding one or more predetermined handwashing operations (for example, the six types of handwashing operations shown in FIG. 18), when the progress of all handwashing operations reaches 100%, it can be determined that the handwashing operation has ended. Also, for example, when the processing unit 31 detects the absence (non-presence) of the operator or the operator's fingers by means of a human presence sensor installed in the handwashing sink, or when the operator or the operator's fingers disappear from the captured image of the camera 33, etc., it may be determined that the handwashing operation has ended. Note that the determination of the end of the handwashing operation is not limited to the above method, and various methods may be adopted. If it is determined that the handwashing operation has not ended (S38: NO), the processing unit 31 returns the process to step S33 and repeats the processes of steps S33 to S37.

[0081] If it is determined that the handwashing operation has ended (S38: YES), the processing unit 31 determines whether or not all necessary handwashing operations have ended normally (step S39). Here, for example, when the progress of each handwashing operation displayed on the handwashing screen reaches 100% for all, it is determined that the operation has ended normally. If it is determined that the operation has ended normally (S39: YES), the processing unit 31 notifies that the handwashing operation has ended normally (step S40) by, for example, displaying a message or the like on the handwashing screen, and ends the process. For example, the processing unit 31 can display a message such as "Proper handwashing operation has been completed" on the display unit 35. If the monitoring device 3 has a speaker, the processing unit 31 may output the above message as voice from the speaker.

[0082] If it is determined that the handwashing operation has not ended normally (has been interrupted) (S39: NO), the processing unit 31 notifies that the handwashing operation has been interrupted (interruption of the operation, abnormal end) by, for example, displaying a message or the like on the handwashing screen (step S41). For example, the processing unit 31 can display a message such as "The handwashing operation has not been completed yet" or "The handwashing is insufficient" on the display unit 35.

[0083] The processing unit 31 determines whether the interrupted (not normally completed) handwashing operation has been resumed (step S42). At this time, the processing unit 31 performs processes such as the start determination of the handwashing operation in step S31 and the identification of the handwashing operation in step S33. If it is determined that the handwashing operation has been started within a predetermined time (for example, several seconds) after the notification in step S41, it is determined that the interrupted handwashing operation has been resumed as a handwashing operation by the same operator. If it is determined that the handwashing operation has been resumed (S42: YES), the processing unit 31 returns the process to step S33. When measuring the execution time of the handwashing operation in step S35, the processing unit 31 resumes the timing process from the execution time of each handwashing operation that has been stored in the storage unit 32 or a storage area in the processing unit 31, taking over the execution time that was being measured before the interruption. The processing unit 31 may reset the execution time of the handwashing operation that was being measured until the interruption when the handwashing operation is interrupted. In this case, the operator who interrupted the handwashing operation needs to start the handwashing work from the beginning. If it is determined that the handwashing operation has not been resumed (S42: NO), the processing unit 31 ends the process. At this time, the processing unit 31 may reset the execution time of each handwashing operation stored for this operator.

[0084] Through the above-described processing, the monitoring device 3 monitors the handwashing operation performed by the operator based on the captured image of the state of the operator's handwashing operation, and manages the progress of the handwashing operation. Then, when the operator appropriately performs the prescribed handwashing operation, the monitoring device 3 notifies the operator to that effect (normal completion). Further, when the operator interrupts the handwashing operation midway, the monitoring device 3 notifies the operator to that effect (interruption). Therefore, the operator can determine whether his / her handwashing operation is appropriate not only based on his / her own judgment but also based on the determination process by the monitoring device 3. In the information processing system according to the present embodiment, the progress of each handwashing operation is notified by the progress bar on the screen shown in FIG. 18, so that the operator performing the handwashing operation can easily grasp his / her own progress status for each handwashing operation. Therefore, by performing the handwashing operation while checking the progress, it is possible to suppress the occurrence of variations in the cleanliness due to the washing method (type of handwashing operation), such as overwashing or insufficient washing of the same area, and it becomes possible to perform an efficient handwashing operation.

[0085] <Summary> In the information processing system according to the present embodiment having the above configuration, the learning device 1 acquires training data in which the captured image data of the fingers of the operator performing handwashing is associated with a plurality of hierarchical labels related to handwashing, and uses the acquired training data to generate a learning model 5 that outputs a plurality of pieces of information related to the plurality of hierarchical labels when image data is input. As a result, the information processing system can generate the learning model 5 that is expected to identify various types of handwashing operations in the learning device 1, and it is expected that the monitoring device 3 can identify the types of various handwashing operations using the generated learning model 5.

[0086] Also, in the present embodiment, the labels output by the learning model 5 are two-layered, the first-layer label is a label for classifying finger parts such as the back of the hand, fingertips, and between nails, and the second-layer label is a label for classifying the detailed washing methods of each part shown in FIG. 7, for example. As a result, it is expected that the information processing system according to the present embodiment can identify the detailed washing method for each part based on the captured image.

[0087] Also, the learning model 5 generated in this embodiment includes an intermediate layer (CNN layer 51) that outputs an intermediate value according to the input image data, a first label layer (layer 1 output layer 52) that outputs information regarding the labels of the first layer according to this intermediate value, and a second label layer (layer 2 output layer 53) that outputs information regarding the labels of the second layer according to this intermediate value. Alternatively, the learning model 5 includes an intermediate layer (CNN layer 51) that outputs an intermediate value according to the input image data, a first label layer (layer 1 output layer 52) that outputs information regarding the labels of the first layer according to this intermediate value, and a second label layer (layer 2 output layer 53) that outputs information regarding the labels of the second layer according to the information output by the first label layer. Alternatively, the learning model 5 includes an intermediate layer (CNN layer 51) that outputs an intermediate value according to the input image data, a second label layer (layer 2 output layer 53) that outputs information regarding the labels of the second layer according to this intermediate value, and a first label layer (layer 1 output layer 52) that outputs information regarding the labels of the first layer according to the information output by the second label layer. Alternatively, the learning model 5 includes an intermediate layer (CNN layer 51) that outputs an intermediate value according to the input image data, and a recurrent neural network layer (LSTM layer 54) that outputs information regarding the labels of the first layer and information regarding the labels of the second layer as sequential information according to this intermediate value. By generating the learning model 5 with these configurations using the training data, it can be expected to accurately obtain information regarding the labels of two layers from the captured image using the generated learning model 5.

[0088] Also, in this embodiment, the labels output by the learning model 5 are regarded as three layers, the labels of the first layer are used as labels for classifying the finger parts, the labels of the second layer are used as labels for classifying left and right, and the labels of the third layer are used as labels for classifying detailed washing methods. Thereby, it can be expected that the information processing system according to this embodiment can specify more detailed washing methods for each part based on the captured image.

[0089] Also, in the present embodiment, the labels output by the learning model 5 are divided into three layers. The label of the first layer is a label for classifying whether handwashing is being performed or not, the label of the second layer is a label for classifying the parts of the fingers, and the label of the third layer is a label for classifying the detailed washing method. Thus, it can be expected that the information processing system according to the present embodiment can perform various classifications even in a situation where handwashing is not being performed.

[0090] Also, in the information processing system according to the present embodiment, the monitoring device 3 acquires image data obtained by photographing the operator's fingers, inputs the image data into the learning model 5 that has been machine-learned in advance as described above, acquires information regarding the multi-layer labels output by the learning model 5, and specifies the type of the operator's handwashing operation based on the acquired information. By using the learning model 5 that has been appropriately machine-learned in advance, it can be expected that the monitoring device 3 can specify various types of handwashing operations.

[0091] Also, in the information processing system according to the present embodiment, the monitoring device 3 specifies the type of handwashing operation from the information regarding the labels of each layer based on the information regarding the multi-layer labels acquired from the learning model 5, and when the specific results corresponding to the labels of each layer match, specifies the type of the operator's handwashing operation as the type where the specific results match. When the specific results corresponding to the labels of each layer do not match, the monitoring device 3 compares the previous specific result with the current multiple specific results. When there is a match between the previous specific result and one of the current multiple specific results in the current multiple specific results, the monitoring device 3 sets the type of handwashing operation that is the same as the previous specific result as the current specific result. When there is no match with the previous specific result, the monitoring device 3 calculates the statistical value (for example, average value, total value, etc.) of the likelihood of the lower layer for each top layer, and specifies the type of the operator's handwashing operation based on the calculated statistical value. Thus, the monitoring device 3 can specify the type of the operator's handwashing operation from the image photographed by the camera 33 by using the learning model 5 that outputs information regarding the multi-layer labels.

[0092] Also, in the information processing system according to the present embodiment, the monitoring device 3 determines the progress of the identified handwashing operation, and displays the determined progress, the model image of the handwashing operation, and the image obtained by photographing the handwashing operation on the display unit 35. As a result, the operator can confirm the progress of the handwashing operation being performed based on various information displayed on the display unit 35, and it is expected that an efficient handwashing operation can be performed.

[0093] <Embodiment 2> In the information processing system according to Embodiment 2, instead of the image data captured by the camera 33, identification of the handwashing operation is performed based on the depth data acquired using a depth sensor. FIG. 20 is a block diagram showing the configuration of the monitoring device 3 according to Embodiment 2. The monitoring device 3 according to Embodiment 2 has a configuration in which the camera 33 of the monitoring device 3 shown in FIG. 16 is replaced with a depth sensor 37. Since the configuration other than the depth sensor 37 is the same as the monitoring device 3 shown in FIG. 20, detailed description thereof is omitted.

[0094] The depth sensor 37 includes, for example, a light emitting unit that emits light such as laser light, near-infrared light, or infrared light, and a light receiving unit that receives the reflected light of the light emitted by the light emitting unit by an object, and measures the distance to the object based on the received reflected light. The depth sensor 37 measures the distance to the object according to an instruction from the processing unit 31, and acquires distance data indicating the measured distance. The depth sensor 37 acquires distance data at a frequency of, for example, 30 times or 15 times per second and provides it to the processing unit 31. Note that the depth sensor 37 may be configured to be built in the monitoring device 3, or may be configured to be externally attached to the monitoring device 3, or may be configured to be connected to the monitoring device 3 via a network such as a LAN or the Internet. In this case, the monitoring device 3 includes a connection unit capable of connecting an external sensor or a sensor communication unit capable of wireless communication with an external sensor, and acquires the distance data acquired by the external sensor via the connection unit or the sensor communication unit.

[0095] In Embodiment 2, the depth sensor 37 is provided at a position where it can measure the distance to the fingers of the operator performing the hand-washing operation at the hand-washing sink, and is provided, for example, at the same position as the camera 33 in FIGS. 2 and 3. Note that the depth sensor 37 only needs to be provided with at least the fingers of the operator as the distance measurement target, and may be provided with the entire body of the operator as the distance measurement target. Further, the depth sensor 37 may be configured to perform distance measurement not only from above the front side of the operator but also from above the lateral direction, or may be configured to perform distance measurement only from above the lateral direction.

[0096] The learning model 5 according to Embodiment 2 is realized by the same configuration as the learning model 5 of Embodiment 1. However, in the learning model 5 according to Embodiment 2, distance data measured by the depth sensor 37 for the distance to the fingers of the operator during the hand-washing operation is used as input. That is, the learning model 5 according to Embodiment 2 is a learned model in which machine learning is performed so as to output the likelihood of a plurality of hierarchical labels regarding the type of hand-washing operation being performed by the distance measurement target operator based on the input distance data when the distance data acquired by the depth sensor 37 is input. Even in the case of using the learning model 5 having such a configuration, the monitoring device 3 can identify the type of hand-washing operation being performed by the operator by performing the same processing as that described in Embodiment 1.

[0097] The monitoring device 3 according to Embodiment 2 including the depth sensor 37 performs the same processing as the monitoring device 3 according to Embodiment 1 by treating the two-dimensional planar distance data obtained by the depth sensor 37 in the same manner as the captured image, and can identify the type of hand-washing operation of the operator. Similarly, the learning device 1 according to Embodiment 2 can also generate the learning model 5 by performing machine learning using training data in which distance data and a plurality of hierarchical labels are associated instead of the captured image, with the same processing as the learning device 1 according to Embodiment 1.

[0098] In the information processing system according to the second embodiment with the above configuration, instead of the data of the image captured by the camera 33, the monitoring device 3 identifies the handwashing operation of the worker based on the distance data measured by the depth sensor 37. Further, the learning device 1 performs machine learning using training data in which distance data is associated with labels of multiple layers, and generates a learning model 5 that outputs information (likelihood) regarding the labels of multiple layers for the input of the distance data. Even when using the distance data measured by the depth sensor 37, similar to the information processing system according to the first embodiment, the information processing system according to the second embodiment can generate a learning model 5 that can be expected to identify various types of handwashing operations in the learning device 1, and it can be expected that the monitoring device 3 identifies the types of various handwashing operations using the generated learning model 5.

[0099] Also, since other configurations of the information processing system according to the second embodiment are the same as those of the information processing system according to the first embodiment, the same reference numerals are given to the same parts, and detailed descriptions thereof are omitted.

[0100] In each of the above-described embodiments, the process of the monitoring device 3 determining the type of handwashing operation being performed by the worker based on the captured image or the distance data may be configured to be performed by a predetermined server connected to the network. In this case, the processing unit 31 of the monitoring device 3 sequentially transmits the captured image or the distance data to be acquired to a predetermined server via the network, and acquires the determination result (the type of handwashing operation being performed by the worker in the captured image or the distance data) determined by the predetermined server, and may identify the handwashing operation being performed by the worker in the captured image or the distance data according to the acquired determination result. At this time, the server may store the learned learning model in the storage unit, and may identify the type of handwashing operation being performed by the worker based on the captured image or the distance data received from the monitoring device 3 using the stored learning model. Note that the server here may be realized using a server computer, a personal computer, or may be realized using a plurality of virtual machines provided in one server, or may be realized using a cloud server.

[0101] The embodiments disclosed this time should be considered illustrative in all respects and not restrictive. The scope of the present invention is shown not by the above description but by the claims, and it is intended that all modifications within the meaning and scope equivalent to the claims are included.

Explanation of Signs

[0102] 1 Learning device 3 Monitoring device 5 Learning model 11 Processing unit 11a Training data acquisition unit 11b Learning model generation unit 11c Learning model transmission unit 12 Storage unit 12a Program 12b Training data 13 Communication unit 31 Processing unit 31a Operation identification processing unit 31b Display processing unit 32 Storage unit 32a Program 33 Camera 34 Communication unit 35 Display unit 37 Depth sensor 51 CNN layer 52 Layer 1 output layer 53 Layer 2 output layer 54 LSTM layer 99 Recording medium N Network

Claims

1. Obtain training data in which image data obtained by photographing the fingers of an operator performing handwashing or distance data obtained by measuring the distance to the fingers is associated with a plurality of hierarchical labels related to handwashing, Generate a learning model that outputs a plurality of pieces of information related to the plurality of hierarchical labels when image data or distance data is input based on the obtained training data, The labels are provided in two layers, The label of the first layer is a label for classifying the part of the finger where handwashing is being performed, The label of the second layer is a label for classifying the way of washing the part, A method for generating a learning model.

2. The learning model is An intermediate layer that outputs an intermediate value according to the input image data or distance data, A first label layer that outputs information related to the label of the first layer according to the intermediate value, A second label layer that outputs information related to the label of the second layer according to the intermediate value And has, The method for generating a learning model according to Claim 1.

3. The learning model is An intermediate layer that outputs an intermediate value according to the input image data or distance data, A first label layer that outputs information related to the label of the first layer according to the intermediate value, A second label layer that outputs information related to the label of the second layer according to the information related to the label of the first layer And has, The method for generating a learning model according to Claim 1.

4. The learning model is An intermediate layer that outputs an intermediate value according to the input image data or distance data, A second label layer that outputs information related to the label of the second layer according to the intermediate value, A first label layer that outputs information related to the label of the first layer according to the information related to the label of the second layer And has, The method for generating a learning model according to Claim 1.

5. The learning model is An intermediate layer that outputs an intermediate value according to the input image data or distance data, A recurrent neural network layer that outputs information related to the label of the first layer and information related to the label of the second layer as sequential information according to the intermediate value And has, The method for generating a learning model according to Claim 1.

6. Obtain training data in which image data obtained by photographing the fingers of an operator performing handwashing or distance data obtained by measuring the distance to the fingers is associated with a plurality of hierarchical labels related to handwashing, Based on the obtained training data, generate a learning model that outputs a plurality of pieces of information regarding the labels of the plurality of layers when image data or distance data is input. The labels are provided in three layers. The label of the first layer is a label for classifying the finger part where handwashing is being performed. The label of the second layer is a label for classifying whether the part is on the left or right hand. The label of the third layer is a label for classifying the way of washing the part. Method for generating a learning model.

7. Obtain training data in which image data obtained by photographing the fingers of an operator performing handwashing or distance data obtained by measuring the distance to the fingers is associated with a plurality of layers of labels regarding handwashing. Based on the obtained training data, generate a learning model that outputs a plurality of pieces of information regarding the labels of the plurality of layers when image data or distance data is input. The labels are provided in three layers. The label of the first layer is a label for classifying whether handwashing is being performed. The label of the second layer is a label for classifying the finger part where handwashing is being performed. The label of the third layer is a label for classifying the way of washing the part. Method for generating a learning model.

8. Obtain image data obtained by photographing the fingers of an operator performing handwashing or distance data obtained by measuring the distance to the fingers. Based on the training data in which image data obtained by photographing the fingers of an operator performing handwashing or distance data obtained by measuring the distance to the fingers is associated with a plurality of layers of labels regarding handwashing, input the obtained image data or distance data into a learning model generated to output a plurality of pieces of information regarding the labels of the plurality of layers when image data or distance data is input. Obtain a plurality of pieces of information regarding the labels of the plurality of layers output by the learning model. Respectively identify the types of handwashing operations based on the information regarding the labels of each obtained layer. When the identified multiple types match, identify the type as the type of the operator's handwashing operation. Repeatedly identify the type of the operator's handwashing operation for a plurality of image data photographed in time series or distance data measured in time series. Respectively identify the types of handwashing operations based on the information regarding the labels of each obtained layer. When the identified multiple types do not match, determine whether at least one of the identified multiple types matches the type identified based on the previous image data or distance data. When it is determined that they match, identify the type specified last time as the type of the handwashing operation of the current operator. Information processing method.

9. The learning model outputs the likelihood of the label as information regarding the label. When it is determined that at least one of the multiple types specified for each of the multiple layers does not match the type specified based on the previous image data or distance data, calculate the statistical value of the likelihoods of the lower layers for each top layer. Based on the calculated statistical value, identify the type of the handwashing operation of the operator. The information processing method according to claim 8.

10. Obtain image data obtained by photographing the fingers of an operator performing handwashing or distance data obtained by measuring the distance to the fingers, Based on training data in which image data obtained by photographing the fingers of an operator performing handwashing or distance data obtained by measuring the distance to the fingers is associated with multiple layers of labels related to handwashing, input the obtained image data or distance data into a learning model generated to output multiple pieces of information regarding the multiple layers of labels when the image data or distance data is input, Obtain the multiple pieces of information regarding the multiple layers of labels output by the learning model. Based on the obtained multiple pieces of information, identify the type of the handwashing operation of the operator. Regarding the identified type of handwashing operation, determine the progress degree of the handwashing operation. Display the determined progress degree, a reference image of the handwashing operation, and an image of the handwashing operation being photographed. Information processing method.

11. Cause a computer to Obtain training data in which image data obtained by photographing the fingers of an operator performing handwashing or distance data obtained by measuring the distance to the fingers is associated with multiple layers of labels related to handwashing, Generate a learning model that outputs multiple pieces of information regarding the multiple layers of labels when image data or distance data is input based on the obtained training data. Perform the process. The label is provided in two layers. The label of the first layer is a label that classifies the part of the finger where handwashing is being performed. The label of the second layer is a label that classifies the way of washing the part. Computer program.

12. Cause a computer to Obtain training data in which image data obtained by photographing the fingers of an operator performing handwashing or distance data obtained by measuring the distance to the fingers is associated with multiple layers of labels related to handwashing. Generate a learning model that outputs multiple pieces of information regarding the labels of the plurality of layers when image data or distance data is input based on the obtained training data. Cause the processing to be performed. The labels are provided in three layers. The label of the first layer is a label for classifying the finger part where handwashing is being performed. The label of the second layer is a label for classifying whether the part is on the left or right hand. The label of the third layer is a label for classifying the way of washing the part. A computer program.

13. On a computer, Obtain training data in which image data obtained by photographing the fingers of an operator performing handwashing or distance data measuring the distance to the fingers is associated with multiple layers of labels related to handwashing. Generate a learning model that outputs multiple pieces of information regarding the labels of the plurality of layers when image data or distance data is input based on the obtained training data. Cause the processing to be performed. The labels are provided in three layers. The label of the first layer is a label for classifying whether handwashing is being performed. The label of the second layer is a label for classifying the finger part where handwashing is being performed. The label of the third layer is a label for classifying the way of washing the part. A computer program.

14. On a computer, Obtain image data obtained by photographing the fingers of an operator performing handwashing or distance data measuring the distance to the fingers. Input the obtained image data or distance data into a learning model generated to output multiple pieces of information regarding the labels of the plurality of layers when image data or distance data is input based on the training data in which the image data obtained by photographing the fingers of an operator performing handwashing or the distance data measuring the distance to the fingers is associated with multiple layers of labels related to handwashing. Obtain the multiple pieces of information regarding the labels of the plurality of layers output by the learning model. Respectively identify the types of handwashing actions based on the information regarding the labels of each obtained layer. When the identified multiple types match, identify the type as the type of the operator's handwashing action. Repeatedly identify the type of the operator's handwashing action for a plurality of image data photographed in time series or distance data measured in time series. Respectively identify the types of handwashing actions based on the information regarding the labels of each obtained layer. When a plurality of specified types do not match, determine whether at least one of the plurality of specified types matches the type specified based on the previous image data or distance data, When it is determined that they match, specify the type specified previously as the type of the handwashing operation of the operator this time, A computer program that causes the processing to be performed.

15. On a computer, Obtain image data obtained by photographing the fingers of an operator performing handwashing or distance data obtained by measuring the distance to the fingers, Based on training data in which image data obtained by photographing the fingers of an operator performing handwashing or distance data obtained by measuring the distance to the fingers is associated with a plurality of hierarchical labels, when the image data or distance data is input, input the obtained image data or distance data into a learning model generated to output a plurality of pieces of information regarding the plurality of hierarchical labels, Obtain a plurality of pieces of information regarding the plurality of hierarchical labels output by the learning model, Based on the obtained plurality of pieces of information, specify the type of the handwashing operation of the operator, Regarding the specified type of handwashing operation, determine the progress degree of the handwashing operation, Display the determined progress degree, the reference image of the handwashing operation, and the image of the handwashing operation being photographed A computer program that causes the processing to be performed.

16. An acquisition unit that acquires training data in which image data obtained by photographing the fingers of an operator performing handwashing or distance data obtained by measuring the distance to the fingers is associated with a plurality of hierarchical labels, A generation unit that generates a learning model that outputs a plurality of pieces of information regarding the plurality of hierarchical labels when image data or distance data is input based on the acquired training data Comprising, The labels are provided in two layers, The first layer label is a label for classifying the part of the finger where handwashing is being performed, The second layer label is a label for classifying the way of washing the part, An information processing apparatus.

17. An acquisition unit that acquires training data in which image data obtained by photographing the fingers of an operator performing handwashing or distance data obtained by measuring the distance to the fingers is associated with a plurality of hierarchical labels, A generation unit that generates a learning model that outputs a plurality of pieces of information regarding the plurality of hierarchical labels when image data or distance data is input based on the acquired training data Comprising, The labels are provided in three layers, The first layer label is a label for classifying the part of the finger where handwashing is being performed, The label of the second layer is a label for classifying whether the part is the left or right hand. The label of the third layer is a label for classifying the washing method of the part. An information processing apparatus.

18. An acquisition unit that acquires training data in which image data obtained by photographing the fingers of an operator performing hand washing or distance data obtained by measuring the distance to the fingers is associated with a plurality of hierarchical labels related to hand washing, and A generation unit that generates a learning model that outputs a plurality of pieces of information related to the plurality of hierarchical labels when image data or distance data is input based on the acquired training data. Comprising The labels are provided in three layers. The label of the first layer is a label for classifying whether hand washing is being performed. The label of the second layer is a label for classifying the part of the finger where hand washing is being performed. The label of the third layer is a label for classifying the washing method of the part. An information processing apparatus.

19. A first acquisition unit that acquires image data obtained by photographing the fingers of an operator performing hand washing or distance data obtained by measuring the distance to the fingers, and Based on training data in which image data obtained by photographing the fingers of an operator performing hand washing or distance data obtained by measuring the distance to the fingers is associated with a plurality of hierarchical labels related to hand washing, the acquired image data or distance data is input to a learning model generated to output a plurality of pieces of information related to the plurality of hierarchical labels, and a second acquisition unit that acquires the plurality of pieces of information related to the plurality of hierarchical labels output by the learning model, and 、 A specifying unit that specifies the types of hand washing operations respectively based on the information related to the labels of each acquired layer, and when the specified multiple types match, specifies the type as the type of the operator's hand washing operation. Comprising Repeatedly specifying the type of the operator's hand washing operation for a plurality of image data captured in time series or distance data measured in time series, Specifying the types of hand washing operations respectively based on the information related to the labels of each acquired layer, When the specified multiple types do not match, determining whether at least one of the specified multiple types matches the type specified based on the previous image data or distance data, When it is determined that they match, specifying the type specified previously as the type of the operator's hand washing operation this time. An information processing apparatus.

20. A first acquisition unit that acquires image data obtained by photographing the fingers of an operator performing hand washing or distance data obtained by measuring the distance to the fingers; Based on training data in which image data obtained by photographing the fingers of an operator performing hand washing or distance data obtained by measuring the distance to the fingers is associated with a plurality of hierarchical labels, when the image data or distance data is input, the image data or distance data is input to a learning model generated to output a plurality of pieces of information regarding the plurality of hierarchical labels, and a second acquisition unit that acquires the plurality of pieces of information regarding the plurality of hierarchical labels output by the learning model; 、 An identification unit that identifies the type of the operator's hand washing operation based on the acquired plurality of pieces of information; A determination unit that determines the progress of the hand washing operation for the identified type of hand washing operation; A display processing unit that performs a process of displaying the determined progress, a reference image of the hand washing operation, and an image obtained by photographing the hand washing operation on a display unit; An information processing apparatus comprising:

Images