Information processing device and program

Abstract

To provide an information processing device for presenting whether a series of predetermined operations are operations suitable for operation identification, to a user.SOLUTION: An information processing device comprises: a memory; an image interface; a reporting interface; and at least one processor. The memory stores a work pattern indicating a series of predetermined operations and an order of operations. The at least one processor identifies an operation of a user on the basis of an acquired image from the image interface in a first determination block, determines whether the identified operation of the user satisfies a prescribed condition required for operation determination, then if the prescribed condition is satisfied in the first determination block, notifies the user of execution of the operation determination, and in a second determination block, identifies the operation of the user executing a series of work on the basis of the image, determines whether the identified operation of the user matches the work pattern, and outputs information indicating the result of the operation determination.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to an information processing device and a program. [Background technology]

[0002] A system for determining whether hand washing has been performed in a predetermined sequence is provided, which identifies a user's hand washing actions and determines whether the identified hand washing actions conform to a specific hand washing pattern.

[0003] In this situation, in a system that uses images to evaluate a series of predetermined actions, such as hand-washing, there was a problem in that sufficient accuracy in identifying the action could not be ensured if the target part needed to identify the action was not captured within the camera's shooting area, or if the target part was not within an appropriate distance range from the camera. Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to present to a user whether a predetermined series of actions is suitable for action identification. [Means for solving the problem]

[0005] The information processing device according to the embodiment includes a memory, an image interface, a notification interface, and at least one processor. The memory stores a work pattern indicating the motion and sequence of a predetermined series of work. The image interface acquires a captured image of a target area. The notification interface notifies a user. The at least one processor, in a first judgment block, identifies a motion of the user based on the captured image and determines whether the identified motion of the user satisfies a predetermined condition required for motion judgment. When the predetermined condition is satisfied in the first judgment block, the at least one processor notifies the user that the motion judgment will be performed, and, in a second judgment block, identifies a motion of the user performing the series of work based on the captured image and performs the motion judgment to determine whether the identified motion of the user is consistent with the work pattern. The at least one processor outputs information indicating a result of the motion judgment. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a motion determining device according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of another configuration of the motion determining device according to the embodiment. [Diagram 3] FIG. 3 is a block diagram showing an example of a hardware configuration of the motion determining device according to the embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a hand-washing pattern according to the embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of a pattern setting table according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a feature quantity model table according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of an action class table according to the embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of an identification information table according to the embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a functional configuration of a motion determining device according to an embodiment. [Figure 10] FIG. 10 is a diagram showing an example of an instruction screen according to the embodiment. [Figure 11] FIG. 11 is a flowchart showing an example of the flow of processing executed in the movement determining device according to the embodiment. [Figure 12] FIG. 12 is a flowchart showing an example of the process flow of each block in FIG. [Figure 13] FIG. 13 is a flowchart showing an example of the process flow of each block in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an information processing device, a display terminal, a system, an information processing method, a program, and a recording medium according to embodiments will be described with reference to the drawings.

[0008] The embodiment described below is an embodiment of an information processing device, a display terminal, a system, an information processing method, a program, and a recording medium, and does not limit the configuration, specifications, etc. The embodiment described below is an example of application to a system that uses images to evaluate the hand-washing action of a person washing their hands.

[0009] The motion determination device (system) according to the embodiment is an example of a system that determines whether a user's hand washing has been performed in a predetermined procedure. The motion determination device captures an image of the user's hand in a sink or the like. The motion determination device determines a motion class of the user's hand from the captured image. For example, the motion determination device determines hand motions classified according to the parts being washed (back of the hand, between the fingers, thumb, etc.) as motion classes. The motion determination device determines whether the user's hand washing has been performed in a predetermined procedure based on the order of the motion classes, the duration of the motion, etc.

[0010] 1 is a diagram illustrating an example of the configuration of a motion determining device 1 according to an embodiment. The motion determining device 1 includes a camera 3, an input / output device 5 (display terminal), a reader 6, and a sink 20, as shown in FIG.

[0011] Sink 20 is a water tank in which a user washes his / her hands. Sink 20 has a recessed structure. A drain groove is formed in the bottom of sink 20. Faucet 21 is installed in sink 20, for example, at a position facing the drain groove, i.e., so as to be able to discharge water into a space formed by the recessed structure of sink 20. Faucet 21 discharges water for a user to wash their hands. The discharged water is discharged from the drain groove of sink 20.

[0012] Camera 3 is installed above sink 20, for example at a position facing the space formed by the recessed structure of sink 20, facing downwards with the drain side as the shooting direction. Camera 3 shoots an area (target area) including the user's hands. Camera 3 shoots the user's hands from above. Camera 3 shoots a two-dimensional image (shot image). Camera 3 outputs color information (RGB (Red Green Blue) values) for each dot. Camera 3 may also have a light that illuminates the user's hands.

[0013] The input / output device 5 is installed in a position visible to the user. In the example of Fig. 1, the input / output device 5 is installed in front of the user who is washing his / her hands at the sink 20.

[0014] The input / output device 5 is an interface that receives input of instructions from an operator and displays various information to the operator. The input / output device 5 has an operation unit that receives input of instructions and a display unit that displays information.

[0015] The input / output device 5 transmits a signal indicating an operation received from an operator to a control device 10 (described later) as an operation of the operation unit. For example, the operation unit is configured using a touch panel. The operation unit may also have a keyboard or a numeric keypad.

[0016] The input / output device 5 operates as a display unit to display an image from the control device 10. For example, the display unit is configured using a liquid crystal monitor. The display unit may be formed integrally with a touch panel serving as an operation unit.

[0017] Note that the input / output device 5 may have an output unit that outputs voice or notification sound from the control device 10, instead of or in addition to the display unit. This output unit is configured using, for example, a speaker.

[0018] The reader 6 is an acquisition device that acquires identification information for identifying a user. For example, the reader 6 acquires identification information (such as an ID) for identifying a user from a terminal (such as an IC card or an RFID (radio frequency identification) tag) held by the user. Here, it is assumed that the user connects his / her own terminal to the reader 6. The reader 6 acquires the identification information from the terminal by connecting to the terminal via a wired or wireless connection. The reader 6 may also read codes such as barcodes or two-dimensional codes. The identification information may be information for identifying an individual user, or may indicate the user's position (such as a member of the cooking department or a plating worker).

[0019] Here, another example of the movement determining device 1 will be described.

[0020] Fig. 2 is a diagram illustrating another configuration example (motion determination device 1') of the motion determination device 1 according to the embodiment. The configuration example (motion determination device 1') of Fig. 2 includes a camera 3, an input / output device 5, a reader 6, and a sink 20, similar to the configuration example (motion determination device 1) of Fig. 1.

[0021] In the configuration example of FIG. 2, the camera 3 is installed beside the sink 20 (on the left side in FIG. 2). The camera 3 captures an image of the user's hand from the side (the side). The input / output device 5 is installed beside the sink 20. The other configuration is similar to the configuration example of FIG. 1 (motion determination device 1), and therefore description thereof will be omitted.

[0022] In this embodiment, when there is no need to distinguish between the configuration example of FIG. 1 (motion determining device 1) and the configuration example of FIG. 2 (motion determining device 1′), they will be simply referred to as the motion determining device 1.

[0023] Next, a control system of the movement determining device 1 will be described.

[0024] Fig. 3 is a block diagram showing an example of a hardware configuration of the movement determining device 1 according to the embodiment. As shown in Fig. 3, the movement determining device 1 further includes a control device 10. The control device 10 is communicatively connected to the camera 3, the input / output device 5, and the reader 6.

[0025] The control device 10 (information processing device) has a processor 11, a ROM 12, a RAM 13, an NVM (Non-volatile memory) 14, a communication I / F (interface) 15, a camera I / F 16, an input / output I / F 18, and a reader I / F 19.

[0026] The processor 11 is communicatively connected to the ROM 12, the RAM 13, the NVM 14, the communication I / F 15, the camera I / F 16, the input / output I / F 18, and the reader I / F 19. The camera 3 is communicatively connected to the camera I / F 16. The input / output I / F 18 is communicatively connected to the input / output device 5. The reader 6 is communicatively connected to the reader I / F 19.

[0027] The processor 11 controls the overall operation of the control device 10. The processor 11 may have an internal cache and various interfaces, etc. The processor 11 executes programs stored in advance in the internal cache, the ROM 12, or the NVM 14 to realize various processes.

[0028] Some of the various functions realized by the processor 11 executing the programs may be realized by hardware circuits. In this case, the processor 11 controls the functions executed by the hardware circuits.

[0029] With regard to the various functions realized by processor 11 executing a program, one function may be realized by cooperation of two or more processors, or two or more functions may be realized by a common processor.

[0030] The ROM 12 is a non-volatile memory in which control programs, control data, etc. are stored in advance. The control programs and control data stored in the ROM 12 are pre-installed in accordance with the specifications of the control device 10. The ROM 12 stores, for example, a program for controlling a circuit board of the control device 10.

[0031] The RAM 13 is a volatile memory. The RAM 13 temporarily stores data being processed by the processor 11. The RAM 13 stores various application programs based on instructions from the processor 11. The RAM 13 may also store data necessary for executing the application programs and execution results of the application programs.

[0032] The NVM 14 is a non-volatile memory in which data can be written and rewritten. The NVM 14 is, for example, a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The NVM 14 stores control programs, applications, and various data according to the operational purpose of the control device 10.

[0033] The NVM 14 also stores in advance a pattern setting table 22 (see FIG. 5) showing a plurality of hand washing patterns, a feature model table 23 (see FIG. 6) showing a feature model for determining a hand action class, an action class table 24 (see FIG. 7) showing an action class, and an identification information table 25 (see FIG. 8) showing a hand washing pattern for each user. Each of the pattern setting table 22, the feature model table 23, the action class table 24, and the identification information table 25 may be updated as appropriate according to an operation by an operator. Each of the pattern setting table 22, the feature model table 23, the action class table 24, and the identification information table 25 is not limited to the configuration examples shown in FIGS. 5 to 8 described later. Each table may have a specific configuration added, deleted, or changed as necessary.

[0034] Furthermore, the NVM 14 stores in advance a feature model (classifier) ​​for determining hand motion classes (motion classes "1" to "6" described below). Motion classification using this feature model can be realized, for example, by performing image classification using machine learning such as deep learning. The feature model specifies, for example, parameters of a network (model) obtained by machine learning or the like. For example, the NVM 14 stores the configuration and parameters of a machine learning model as the feature model.

[0035] As an example, the feature model is based on a machine learning model such as a neural network whose parameters are determined by deep learning. As the model, for example, a convolutional neural network (CNN) can be used, but other networks may also be used.

[0036] As an example, the feature model is a machine learning model whose parameters are determined (trained) to output a classification result that classifies a user's action contained in a captured image into one of multiple action classes in response to an input of the captured image.

[0037] As an example, the feature quantity model is common between a hand-washing position judgment block (first judgment block, see FIG. 11) and a hand-washing action judgment block (second judgment block, see FIG. 11).

[0038] As an example, the feature quantity model differs between a hand-washing position judgment block and a hand-washing action judgment block.

[0039] The communication I / F 15 is an interface for transmitting and receiving data to and from an external device. The communication I / F 15 connects to the external device via a network, etc. For example, the communication I / F 15 supports a wired or wireless LAN (Local Area Network) connection.

[0040] The camera I / F 16 (image interface) is an interface for transmitting and receiving data to and from the camera 3. For example, the camera I / F 16 transmits a signal to instruct the camera 3 to capture an image based on the control of the processor 11. The camera I / F 16 also acquires a captured image obtained by capturing an image of a target area from the camera 3. For example, the camera I / F 16 may be one that supports a USB (Universal Serial Bus) connection, or one that supports a connection via Camera Link.

[0041] The input / output I / F 18 (notification interface) is an interface for transmitting and receiving data to and from the input / output device 5. For example, the input / output I / F 18 receives a signal indicating an operation accepted from an operator from the input / output device 5. The input / output I / F 18 transmits the received signal to the processor 11. Furthermore, the input / output I / F 18 transmits information indicating a screen to be displayed to the operator to the input / output device 5 under the control of the processor 11. For example, the input / output I / F 18 may be one that supports a USB connection or one that supports a connection via a parallel interface.

[0042] The reader I / F 19 (identification information interface) is an interface for transmitting and receiving data to and from the reader 6. For example, the reader I / F 19 transmits a signal to start the reader 6 under the control of the processor 11. The reader I / F 19 also transmits identification information from the reader 6 to the processor 11. For example, the reader I / F 19 may support a USB connection.

[0043] The control device 10 is realized by using a computer such as a desktop PC (Personal Computer), a notebook PC, a tablet PC, or a smartphone.

[0044] The control device 10 may further have other configurations as necessary in addition to the configuration example of Fig. 3, or a specific configuration may be excluded from the configuration example of Fig. 3. In addition, in the control device 10, at least two elements among the communication I / F 15, the camera I / F 16, the input / output I / F 18, and the reader I / F 19 may be integrally configured.

[0045] In the motion determining device 1, at least two of the elements of the camera 3, the input / output device 5, the reader 6, and the control device 10 may be integrally configured.

[0046] Next, the hand washing pattern will be described.

[0047] The hand-washing patterns are related to the user's hand-washing. The hand-washing patterns are predefined patterns used by the action determination device 1 to determine whether the hand-washing procedure is appropriate. Fig. 4 is a diagram showing examples of hand-washing patterns according to the embodiment. Fig. 4 illustrates hand-washing patterns "A", "B" and "C".

[0048] Hand-washing pattern "A" is used to determine the hand-washing procedure in factory "A". Hand-washing pattern "A" indicates that the hand-washing procedure continues for 5 seconds each in the order of action classes "3", "4" and "5". Hand-washing pattern "A" is also used when camera 3 captures the user's hands from above (as in FIG. 1).

[0049] Hand-washing pattern "B", like hand-washing pattern "A", is used to determine the hand-washing procedure in factory "A". Hand-washing pattern "B", like hand-washing pattern "A", indicates that the hand-washing procedure continues for 5 seconds each in the order of action classes "3", "4" and "5". Hand-washing pattern "B" is also used when camera 3 captures the user's hands from the side (left in the example of FIG. 4) (as in FIG. 2).

[0050] Hand-washing pattern "C" is used to determine the hand-washing procedure in factory "I". Hand-washing pattern "C" indicates that the hand-washing procedure is to continue the action classes "3", "4", "5" and "6" in any order for a total of 20 seconds. Hand-washing pattern "C" is also used when camera 3 captures the user's hands from above (as in FIG. 1).

[0051] Next, the pattern setting table 22 will be described.

[0052] The pattern setting table 22 shows a plurality of hand-washing patterns (task patterns), each of which indicates the action and sequence of hand-washing (a predetermined series of actions). Fig. 5 is a diagram showing an example of the pattern setting table 22 according to the embodiment. As shown in Fig. 5, the pattern setting table 22 stores a "pattern ID", "action sequence ID", "action class ID", "position determination time", "action duration", "feature model ID", and "total hand-washing time" in association with each other.

[0053] "Pattern ID" is an ID that identifies the hand-washing pattern (default pattern). "Default" indicates the hand-washing pattern when no hand-washing pattern is specified. "A", "B", and "C" indicate hand-washing patterns "A", "B", and "C", respectively.

[0054] The "action order ID" indicates the order of the action class. For example, in the hand-washing pattern "A," the pattern setting table 22 indicates "1" as the "action order ID" corresponding to the action class "3" as the "action class ID." That is, in the hand-washing pattern "A," the pattern setting table 22 indicates "1" as the order of the action class "3."

[0055] Furthermore, in the hand-washing pattern "C," the pattern setting table 22 indicates "0" as the "action order ID" corresponding to each "action class ID." In other words, the pattern setting table 22 indicates that in the hand-washing pattern "C," the action classes "3" to "6" are in no particular order.

[0056] "Action class ID" indicates the action class executed in the hand-washing pattern. Here, "1" to "6" indicate action classes "1" to "6", respectively.

[0057] The "position determination time" indicates the duration of an action class in a corresponding hand-washing pattern with respect to a hand-washing position determination block (first determination block, see FIG. 11). That is, the "position determination time" is an example of a pass threshold in a hand-washing position determination block. For example, the pattern setting table 22 indicates [as], [bs], [cs], and [ds] as the "position determination time" for the hand-washing patterns "default", "A", "B", and "C", respectively. That is, the pattern setting table 22 indicates that each action class continues for a seconds, b seconds, c seconds, and d seconds, respectively, in the hand-washing patterns "default", "A", "B", and "C". The "position determination time" may be set for each action class. This "position determination time" is, for example, shorter than the "action duration", but may be the same time.

[0058] The "action duration" indicates the duration of the corresponding action class for the judgment block of the hand-washing action (second judgment block, see FIG. 11). That is, the "action duration" is an example of a pass threshold in the judgment block of the hand-washing action. For example, the pattern setting table 22 indicates "5s" as the "action duration" corresponding to the action class "3" as the "action class ID" for the hand-washing pattern "A". That is, the pattern setting table 22 indicates that the action class "3" continues for 5 seconds for the hand-washing pattern "A". Also, for example, the pattern setting table 22 indicates "0" as the "action duration" corresponding to each "action class ID" for the hand-washing pattern "C". Also, the pattern setting table 22 indicates "20s" as the "action duration" at the end. That is, the pattern setting table 22 indicates that the action classes "3" to "6" continue for a total of 20 seconds for the hand-washing pattern "C".

[0059] The "feature model ID" is an ID that identifies the feature model used to determine the action class. The "feature model ID" corresponds to the direction in which the camera 3 captures the image of the hand. In this case, when the camera 3 captures the image of the hand from above (as in FIG. 1), the "feature model ID" indicates "H1." Also, when the camera 3 captures the image of the hand from the side (as in FIG. 2), the "feature model ID" indicates "H2."

[0060] "Total hand washing time" indicates the upper limit of the time for a user to wash their hands. For example, the pattern setting table indicates "30s" as the "total hand washing time" for hand washing pattern "A." That is, the pattern setting table 22 indicates that in hand washing pattern "A," the upper limit of the time for a user to wash their hands is 30 seconds. In addition, the "total hand washing time" does not have to be set as in hand washing pattern "C."

[0061] Next, the feature quantity model table 23 will be described.

[0062] 6 is a diagram showing an example of a feature amount model table 23 according to the embodiment. As shown in FIG 6, the feature amount model table 23 stores a "feature amount model ID", a "corresponding action class ID", and a "shooting environment" in association with each other.

[0063] The "feature model ID" is an ID for identifying a feature model.

[0064] The "Corresponding Action Class ID" indicates the action class that the feature model can determine. Here, "H1" and "H2" indicate that the feature model can determine the action classes "1" to "8" indicated by "1" to "8."

[0065] "Shooting environment" indicates the shooting environment that the feature model corresponds to. Here, the feature model table 23 indicates that "H1" corresponds to a shooting image shot from above. Also, the feature model table 23 indicates that "H2" corresponds to a shooting image shot from the side (left and right).

[0066] Next, the action class table 24 will be described.

[0067] Fig. 7 is a diagram showing an example of the action class table 24 according to the embodiment. As shown in Fig. 7, the action class table 24 stores an "action class ID", an "action class display name", and an "action class display image" in association with each other.

[0068] The "action class ID" is an ID for identifying an action class.

[0069] The "action class display name" is a character string that indicates an action class. For example, the "action class display name" indicates "whisk well" as the "action class display name" of the action class "3".

[0070] An "action class display image" is an image that indicates an action class. For example, an "action class display image" is composed of an illustration of an action class, an "action class display name", and the like.

[0071] Next, the identification information table 25 will be described.

[0072] The identification information table 25 indicates changes made to the set hand-washing pattern for each user. Fig. 8 is a diagram showing an example of the identification information table 25 according to the embodiment. As shown in Fig. 8, the identification information table 25 stores "identification information", "pattern change type", "action sequence ID", "action class ID", "action duration", "feature model ID", "total hand-washing time", "explanation of change condition", and the like in association with each other.

[0073] The "identification information" is identification information that identifies a user. Here, the "identification information" may indicate a specific user (such as a worker) or a job title (such as a member of the cooking department or a person engaged in plating).

[0074] The "pattern change type" indicates a method for changing the hand-washing pattern of the user indicated by the corresponding identification information. Here, the "pattern change type" is "difference to default pattern" or "overwrite."

[0075] "Difference to default pattern" indicates that the hand-washing pattern is changed by applying a difference to the set hand-washing pattern (default pattern).

[0076] "Overwrite" refers to changing the hand-washing pattern by overwriting the default pattern.

[0077] "Action Order ID" indicates the order of the action classes. If "Pattern Change Type" is "Difference to Default Pattern", "Action Order ID" indicates addition or deletion to the order of the action classes in the default pattern. If "Pattern Change Type" is "Overwrite", "Action Order ID" indicates the order of the action classes.

[0078] "Action class ID" indicates the action class. If the "pattern change type" is "difference to default pattern", the "action class ID" indicates the action class corresponding to the added or deleted "action order ID". If the "pattern change type" is "overwrite", the "action class ID" indicates the action class in the corresponding "action order ID".

[0079] "Action duration" indicates the duration of the corresponding action class. When "Pattern change type" is "Difference to default pattern", "Action duration" indicates an extension or shortening of the action duration of the action class in the default pattern. Also, "Action duration" indicates the action duration of the action class indicated by the added "Action class ID".

[0080] When the "pattern change type" is "overwrite", the "action class ID" indicates the action duration of the action class indicated by the corresponding "action class ID".

[0081] The "feature model ID" is an ID for identifying a feature model used to determine an action class. The "feature model ID" indicates a changed feature model ID (or no change).

[0082] "Total hand washing time" indicates the maximum time for a user to wash their hands. If "Pattern change type" is "Difference to default pattern", "Total hand washing time" indicates an extension or shortening of the total hand washing time in the default pattern. If "Pattern change type" is "Overwrite", "Total hand washing time" indicates the total hand washing time after the change.

[0083] "Description of change conditions" is text indicating the content of the change for each user. Here, "Description of change" indicates the content of the change when the set default pattern is default hand washing pattern "A". "Description of change conditions" is text indicating the content of the change to the default pattern when "Pattern change type" is "Difference to default pattern". "Description of change conditions" is text indicating the content of the changed hand washing pattern when "Pattern change type" is "Overwrite".

[0084] Next, functions realized by the movement determining device 1 will be described.

[0085] 9 is a diagram showing an example of a functional configuration of the movement determining device 1 according to the embodiment. In the movement determining device 1, a control device 10 (information processing device) realizes functions as an acquisition unit 101, a movement determining unit 102, and an output unit 103 by a processor 11 executing a program stored in an internal memory, a ROM 12, an NVM 14, or the like.

[0086] The acquisition unit 101 acquires a hand-washing pattern input to the input / output device 5 via the input / output I / F 18. For example, an operator inputs a hand-washing pattern to the input / output device 5 based on the installation location of the sink 20 and the angle of the camera 3, etc. The acquisition unit 101 acquires the operation via the input / output I / F 18.

[0087] Furthermore, the acquisition unit 101 acquires a feature model based on the hand-washing pattern. When the hand-washing pattern is set, the acquisition unit 101 refers to the pattern setting table 22 and acquires from the NVM 14 a feature model corresponding to the ID of the set hand-washing pattern.

[0088] Furthermore, when the feature quantity model is acquired, the acquisition unit 101 refers to the pattern setting table 22 and acquires the action order ID, the action class ID, and the action duration of the set hand-washing pattern.

[0089] Furthermore, the acquisition unit 101 acquires the identification information from the reader 6 through the reader I / F 19. After the instruction screen 50 (see FIG. 10) is displayed, the acquisition unit 101 waits until the reader 6 acquires the identification information. The acquisition unit 101 acquires the user's identification information from the reader 6. When the acquisition unit 101 determines whether the acquired identification information is included in the identification information table 25, it acquires from the identification information table 25 the "pattern change type," "action sequence ID," "action class ID," "action duration," "feature model ID," "total hand-washing time," and the like that correspond to the acquired identification information.

[0090] Furthermore, the acquisition unit 101 acquires a captured image from the camera 3 via the camera I / F 16 .

[0091] Furthermore, when the determination logic is constructed, the acquisition unit 101 refers to the NVM 14 or the like to acquire a display image for position determination corresponding to the acquired action class ID. The display image for position determination may be stored in the action class table 24. Furthermore, when the determination logic is constructed, the acquisition unit 101 refers to the action class table 24 to acquire an action class display image corresponding to the acquired action class ID.

[0092] The action determination unit 102 sets a hand-washing pattern (default pattern). For example, the action determination unit 102 determines whether the input hand-washing pattern exists in the pattern setting table 22. That is, the action determination unit 102 determines whether the ID of the input hand-washing pattern exists in the pattern setting table 22. If the action determination unit 102 determines that the input hand-washing pattern exists in the pattern setting table 22, it sets the input hand-washing pattern. On the other hand, if the input hand-washing pattern does not exist in the pattern setting table 22, the action determination unit 102 sets a default hand-washing pattern.

[0093] Furthermore, the action determination unit 102 constructs a determination logic for determining whether the hand-washing procedure is appropriate. For example, when the action determination unit 102 acquires the action order ID, the action class ID, and the action duration, the action determination unit 102 constructs a determination logic based on the action order ID, the action class ID, the action duration, etc.

[0094] Furthermore, the action determination unit 102 changes (sets) the hand-washing pattern based on the identification information. When the action determination unit 102 acquires the identification information, it determines whether the acquired identification information is included in the identification information table 25. When the action determination unit 102 acquires the "pattern change type", "action sequence ID", "action class ID", "action duration", "feature model ID", and "total hand-washing time", it changes the hand-washing pattern based on the acquired information. For example, when the acquired identification information indicates "worker A", the action determination unit 102 extends the duration of each action of the default pattern by 5 seconds and extends the total hand-washing time by 20 seconds. When the action determination unit 102 changes the hand-washing pattern, it updates the judgment logic based on the changed hand-washing pattern.

[0095] The action determination unit 102 may identify the user's job title from the acquired identification information. For example, the NVM 14 may store in advance a table in which identification information indicating each user is associated with the job title. The action determination unit 102 may refer to the table and identify the user's job title (cooking department member, plating worker, etc.) from the acquired identification information. The action determination unit 102 may change the hand-washing pattern based on the identification information table 25 and the identified job title.

[0096] Furthermore, the motion determination unit 102 determines whether the user's hand-washing is in accordance with a hand-washing pattern based on the determination logic. For example, the motion determination unit 102 identifies the user's motion by classifying the user's motion using a feature quantity model (classifier).

[0097] As an example, the movement determination unit 102 identifies a user's movement using a feature model based on a machine learning model in which parameters are determined (learned) so as to output a classification result that classifies the user's movement contained in the captured image into one of a plurality of movement classes in response to an input of the captured image.

[0098] As an example, the motion determination unit 102 classifies the user's motion using a feature model in each of a hand-washing position determination block (first determination block, see FIG. 11) and a hand-washing motion determination block (second determination block, see FIG. 11), thereby identifying the user's motion. For example, the motion determination unit 102 uses a common feature model between the hand-washing position determination block and the hand-washing motion determination block. For example, the motion determination unit 102 uses different feature models between the hand-washing position determination block and the hand-washing motion determination block.

[0099] As an example, the action determination unit 102, in the hand-washing position determination block, identifies the user's action with respect to some of the hand-washing actions (actions of a series of tasks) identified in the hand-washing action determination block. In other words, some of the classified actions are common between the hand-washing position determination block and the hand-washing action determination block, and the hand-washing position determination block has fewer action classifications than the hand-washing action determination block.

[0100] As an example, the motion determination unit 102 identifies a user's motion based on a captured image in a hand-washing position determination block (first determination block), and determines whether the identified user's motion satisfies a predetermined condition required for motion determination in the motion determination block (second determination block). As an example, the motion determination unit 102 determines that a predetermined condition is satisfied in the hand-washing position determination block when a proportion of a motion time identified as a part of a hand-washing motion (a series of motions) identified in the hand-washing motion determination block in a predetermined predetermined determination time (position determination time) is equal to or greater than a predetermined threshold. For example, when a captured image is acquired, the motion determination unit 102 waits until it detects a user's hand from the captured image. When the motion determination unit 102 detects the user's hand, it determines a motion class of the hand based on a feature quantity model. When the motion determination unit 102 determines the motion class, it determines whether the motion class set as a motion class for position determination is consistent with the determined motion class. When the movement determination unit 102 determines that the two are consistent, it measures the time (movement time) during which the user's hand is determined to be of the corresponding movement class. The movement determination unit 102 waits until the measured movement time exceeds the position determination time corresponding to the corresponding movement class. Note that the movement determination unit 102 may determine that the hand-washing procedure was not performed appropriately if the measured movement time does not exceed the position determination time even after a predetermined time has elapsed since the hand was detected (or the hand movement class changed).

[0101] As an example, when a predetermined condition is satisfied in the hand-washing position judgment block (first judgment block), the motion judgment unit 102 performs motion judgment to identify the motion of the user performing hand-washing (series of tasks) based on the captured image in the hand-washing motion judgment block (second judgment block) and to determine whether the identified motion of the user is consistent with a hand-washing pattern (task pattern). For example, when a captured image is acquired, the motion judgment unit 102 waits until it detects the user's hand from the captured image. When the motion judgment unit 102 detects the user's hand, it judges the motion class of the hand based on the feature model. When the motion judgment unit 102 judges the motion class, it judges whether the motion class set in the first order and the judged motion class are consistent. When the motion judgment unit 102 judges that the two are consistent, it measures the time (motion time) during which the user's hand is judged to be the motion class. The motion judgment unit 102 waits until the measured motion time exceeds the motion duration corresponding to the motion class. In addition, if the measured movement time does not exceed the movement duration even after a predetermined time has elapsed since the hand was detected (or the hand movement class was changed), the movement determination unit 102 may determine that the hand washing procedure was not performed properly.

[0102] Furthermore, when the instruction screen 50 is updated, the action determination unit 102 determines whether or not there is an action class set in the next order after the action class. If the action determination unit 102 determines that there is an action class set in the next order, it operates as described above for the action class. If there is no action class set in the next order, the action determination unit 102 determines that the user has properly washed their hands.

[0103] The action determination unit 102 measures the time that has elapsed since the hand was detected (or the time that has elapsed since the identification information was acquired). If the time that has elapsed since the hand was detected exceeds the total hand-washing time, the action determination unit 102 determines that the hand-washing procedure has not been performed properly. For example, the action determination unit 102 displays information on the instruction screen 50 indicating that the hand-washing procedure has not been performed properly.

[0104] The output unit 103 displays an instruction screen 50 (see FIG. 10) showing the hand-washing procedure on the input / output device 5. For example, when the action class display images are acquired, the output unit 103 generates an instruction screen 50 in which the action class display images are arranged according to the acquired action order IDs. When the output unit 103 generates the instruction screen 50, it outputs display information for displaying the generated instruction screen 50 to the input / output device 5 via the input / output I / F 18, thereby causing the input / output device 5 to display it.

[0105] 10 is a diagram showing an example of an instruction screen 50 according to the embodiment. The instruction screen 50 includes displays of icons 51, 52, 54, and 55, as shown in FIG.

[0106] Icon 51 displays an action class display image. Icon 51 also displays information indicating whether the action class corresponding to the action class display image has been completed. Here, icon 51 displays icon 53, which is either an "o" indicating that the action class has been completed or an "x" indicating that the action class has not been completed. Icon 51 may also display an order corresponding to the action classes. Instruction screen 50 displays the same number of icons 51 as the number of action classes indicated by the hand-washing pattern.

[0107] The icon 52 indicates the progress of the action classes. Here, the icon 52 displays the number of action classes indicated by the hand-washing pattern and the number of action classes that have been completed.

[0108] The icon 54 is an operation icon for instructing a transition to a top screen from which a hand-washing pattern input screen, an identification information acquisition screen, an instruction screen 50, or the like can be transitioned.

[0109] Icon 55 displays a display image for position determination. Icon 55 also displays information indicating whether the hand-washing position determination block has been completed (information notifying the user of a transition to hand-washing action determination). Here, icon 55 displays icon 53 with either "○" indicating that the hand-washing position determination block has been completed or "×" indicating that the hand-washing position determination block has not been completed.

[0110] In this way, when a predetermined condition is met in the hand-washing position judgment block (first judgment block), the output unit 103 notifies the user that an action judgment will be made in the hand-washing action judgment block (second judgment block) by displaying an "O" icon 53 in the icon 55 that displays the display image for position judgment.

[0111] In addition, the output unit 103 may notify the user that movement determination will be performed, for example, by displaying a text message on the instruction screen 50 to the effect that movement determination will be performed, or by outputting a notification sound or voice from a speaker provided in the movement determination device 1 to the effect that movement determination will be performed.

[0112] Note that icon 55 may display at least one of the action class display images of the action class corresponding to the judgment block of the subsequent hand-washing action.

[0113] The instruction screen 50 may further have a configuration as necessary in addition to the configuration exemplified in Fig. 10, or a specific configuration may be excluded from the configuration exemplified in Fig. 10. The configuration of the instruction screen 50 is not limited to a specific configuration.

[0114] Furthermore, when the hand-washing pattern is changed, the output unit 103 updates the instruction screen 50 based on the changed hand-washing pattern.

[0115] Furthermore, when the measured action time elapses the action duration, the output unit 103 updates the instruction screen 50. For example, the output unit 103 displays information indicating that the action class has been completed on the icon 51 of the instruction screen 50. That is, the output unit 103 displays a "◯" icon 53 on the icon 51 corresponding to the action class.

[0116] Furthermore, when it is determined that the hand-washing procedure has not been performed properly, the output unit 103 displays an "X" icon 53 on the corresponding icon 51, indicating that the hand-washing procedure has not been performed properly.

[0117] In this way, the output unit 103 outputs display information for displaying, as information indicating the result of the action judgment, an icon 53 such as an "O" indicating that the action class has been completed or an "X" indicating that the action class has not been completed, on the icon 51 of the corresponding action class.

[0118] Next, an example of the operation of the movement determining device 1 will be described.

[0119] Fig. 11 is a flowchart showing an example of the flow of processing executed in the movement determining device 1 according to the embodiment. Note that the flow of processing in Fig. 11 is merely an example, and steps can be added, deleted, and the order can be changed as desired.

[0120] First, processor 11 of motion judging device 1 executes a hand-washing position judgment block (S1). In addition, processor 11 of motion judging device 1 judges whether the hand-washing motion of the user identified based on the captured image satisfies a predetermined condition required for motion judgment in the hand-washing position judgment block (S2). If it is not judged that the predetermined condition is satisfied in the hand-washing position judgment block (S2: No), processor 11 of motion judging device 1 returns to the process of S1.

[0121] For example, the processor 11 of the movement determination device 1 determines whether the identified hand-washing movement is a user's hand-washing movement suitable for movement determination in the hand-washing movement determination block in the subsequent stage. In other words, the processor 11 of the movement determination device 1 determines whether the captured image is an image sufficient for the subsequent stage of movement determination. Specifically, when the identified hand-washing movement time for some movements in the movement class in the subsequent stage of the hand-washing movement determination block reaches the pass threshold (position determination time) in the hand-washing position determination block, the processor 11 of the movement determination device 1 determines that the predetermined condition of the hand-washing position determination block is satisfied.

[0122] On the other hand, if it is determined that the predetermined condition is satisfied in the hand-washing position judgment block (S2: Yes), the processor 11 of the action judgment device 1 notifies the user of the transition to judgment of the hand-washing action (S3), for example, as described above with reference to FIG. 10.

[0123] Furthermore, processor 11 of movement determination device 1 executes the hand-washing movement determination block (S4). After that, processor 11 of movement determination device 1 notifies the user of the result of the movement determination in the hand-washing movement determination block (S5), as described above with reference to Fig. 10, for example. Then, the flow of Fig. 11 ends.

[0124] Figures 12 and 13 are flowcharts showing an example of the process flow of each block in Figure 11. Note that the process flows in Figures 12 and 13 are merely examples, and steps can be added, deleted, and the order can be changed as desired.

[0125] The processor 11 of the movement determining device 1 executes the process flows of FIGS. 12 and 13 in the hand-washing position determining block (S1 in FIG. 11) and the hand-washing movement determining block (S4 in FIG. 11), respectively.

[0126] The processes of S11 to S21 described below do not have to be executed when the classifier is the same for the hand-washing action judgment block executed consecutively after the hand-washing position judgment block.

[0127] First, processor 11 of movement determination device 1 inputs a hand-washing pattern through input / output I / F 18 (S11). When the hand-washing pattern is input, processor 11 determines whether the input hand-washing pattern is included in pattern setting table 22 (S12).

[0128] If it is determined that the input hand washing pattern is included in the pattern setting table 22 (S12: Yes), the processor 11 refers to the feature model table 23 and acquires a feature model corresponding to the hand washing pattern (S13).

[0129] When the feature quantity model is acquired, the processor 11 constructs a decision logic (S14). After constructing the decision logic, the processor 11 displays an instruction screen 50 on the input / output device 5 via the input / output I / F 18 (S15).

[0130] When it is determined that the input hand washing pattern is not included in the pattern setting table 22 (S12: No), the processor 11 refers to the feature model table 23 and acquires a default feature model (S16). When the default feature model is acquired, the processor 11 constructs a default determination logic (S17). When the default determination logic is constructed, the processor 11 displays a default instruction screen 50 on the input / output device 5 through the input / output I / F 18 (S18).

[0131] When the instruction screen 50 is displayed on the input / output device 5 (S15) or when the default instruction screen 50 is displayed on the input / output device 5 (S18), the processor 11 determines whether identification information has been acquired through the reader 6 (S19).

[0132] If it is determined that the identification information has not been acquired (S19: No), the processor 11 returns to the process of S19.

[0133] When it is determined that the identification information has been acquired (S19: Yes), the processor 11 determines whether to change the hand-washing pattern (S20). That is, the processor 11 determines whether the identification information is included in the identification information table 25.

[0134] When it is determined that the hand-washing pattern is to be changed (S20: Yes), the processor 11 refers to the identification information table 25, changes the hand-washing pattern, and updates the determination logic and the instruction screen 50 based on the changed hand-washing pattern (S21).

[0135] When it is determined that the hand washing pattern should not be changed (S20: No) or when the instruction screen 50 is updated (S21), the processor 11 acquires a pass threshold corresponding to the first unprocessed action (S22). For example, in a hand washing position judgment block, the processor 11 acquires a position judgment time of an action class in the corresponding hand washing pattern as the pass threshold. For example, in a hand washing action judgment block, the processor 11 acquires an action duration corresponding to the first unprocessed action sequence ID as the pass threshold.

[0136] After that, the processor 11 judges whether the total hand-washing time has elapsed (S23). If it is judged that the total hand-washing time has not elapsed (S23: No), the processor 11 judges the user's hand motion class based on the captured image and the feature model (S24). After the motion class is judged, the processor 11 judges whether the judged motion class is consistent with a motion class for hand-washing position judgment or a motion class corresponding to the motion sequence ID (S25).

[0137] If it is determined that the two action classes do not match (S25: No), the processor 11 returns to the process of S24.

[0138] If it is determined that both action classes match (S25: Yes), the processor 11 adds up the action times (S26). After adding up the action times, the processor 11 determines whether the action time exceeds the action duration (S27). Here, the action time is, for example, a count value of the number of captured images (number of frames) classified into the target action class, but may also be a time measured by a timer.

[0139] If it is determined that the action time has not exceeded the action duration (S27: No), the processor 11 returns to the process of S24.

[0140] If it is determined that the action time has exceeded the action duration (S27: Yes), the processor 11 updates the instruction screen 50 (S28). After updating the instruction screen 50, the processor 11 determines whether or not the next action sequence ID exists (S29).

[0141] If it is determined that a next operation sequence ID exists (S29: Yes), the processor 11 updates the first unprocessed operation sequence ID (S30). After updating the first unprocessed operation sequence ID, the processor 11 returns to the process of S22.

[0142] When it is determined that the total hand-washing time has elapsed (S23: Yes) or when it is determined that there is no next action sequence ID (S29: No), the processor 11 displays the determination result as to whether the hand-washing procedure was performed appropriately (S31). For example, when it is determined that the total hand-washing time has elapsed (S23: Yes), the processor 11 displays information indicating that the hand-washing position was not appropriate or information indicating that the hand-washing procedure was not performed appropriately on the instruction screen 50. When it is determined that there is no next action sequence ID (S29: No), the processor 11 displays information indicating that the hand-washing position was appropriate or information indicating that the hand-washing procedure was performed appropriately on the instruction screen 50.

[0143] After displaying the determination result, the processor 11 determines whether or not to end the operation (S32). For example, the processor 11 determines whether or not an operation to end the operation has been input through the input / output device 5.

[0144] When it is determined not to end the operation (S32: No), the processor 11 returns to S19. On the other hand, when it is determined to end the operation (S32: Yes), the processor 11 ends the operation.

[0145] In addition, when the movement classes, including the judgment block for the hand-washing position, are in random order, the processor 11 may measure the movement time of each movement class regardless of which movement class the hand is in.

[0146] In addition, if the processor 11 does not detect the user's hands even after a predetermined time has elapsed since acquiring the identification information, the processor 11 may display information on the instruction screen 50 indicating that the hand washing procedure has not been performed properly.

[0147] Furthermore, the processor 11 may transmit the determination result of each determination block to an external device via the communication I / F 15.

[0148] Furthermore, the motion determination device 1 may include a distance sensor that measures a distance in a target area including the hand. The processor 11 may determine the motion class of the hand based on the measurement result of the distance sensor. The processor 11 may determine the motion class of the hand based on the measurement result of the distance sensor and the captured image.

[0149] In general, in a system that uses images to identify human actions, it is necessary that the target parts necessary for action identification are captured within the camera's shooting area, and at the same time, that the target parts are within an appropriate distance range from the camera. If the above conditions are not fully met, sufficient accuracy in identifying actions cannot be ensured. However, it is difficult to accurately determine which parts are captured by the camera without looking at the images captured by the camera. On the other hand, if the captured images are displayed on a display at any time, there is a problem that the display of the captured images reduces the computational resources for high-speed action classification according to the learning model and the image transmission bandwidth and speed.

[0150] In this situation, the motion determination device 1 according to the embodiment is configured to execute a hand-washing position determination block that checks whether the target part is within a target area for acquiring a photographed image for motion determination (whether a predetermined condition is satisfied) and a hand-washing motion determination block that identifies a motion and determines whether it is a motion that should be performed. Moreover, when it is confirmed in the hand-washing position determination block that the target part is within a target area for acquiring a photographed image for motion determination, the motion determination device 1 is configured to notify the user and then proceed to the hand-washing motion determination block.

[0151] According to this configuration, it is possible to teach the user whether the hand washing action is appropriate without allocating computational resources and image transmission bandwidth and speed to displaying the captured image. In other words, according to the configuration of the embodiment, it is possible to present to the user whether a predetermined series of actions is appropriate for action identification. In addition, since it is possible to present whether the user's action is appropriate prior to action determination, it is possible to improve the accuracy of action determination and reduce the burden on the user caused by redoing, etc.

[0152] In the above embodiment, the technology according to the present disclosure is applied to hand washing, but the present disclosure is not limited to this. The technology according to the present disclosure can be applied to a predetermined series of tasks. Here, the predetermined series of tasks is, for example, a task that includes actions that are provided with a manual and that are required to be performed continuously for a certain period of time. In addition, the predetermined series of tasks may be a sequence of actions, that is, actions for each step of the tasks. Examples of the series of tasks include hand washing, yoga, calisthenics, manufacturing processes, cooking, and the like.

[0153] Furthermore, the technology according to the present disclosure can be applied to, for example, a part or task where the user has difficulty in understanding the angle of view (target area) of the camera 3. Such tasks include, for example, tasks performed with the user facing away from the camera 3.

[0154] According to at least one of the embodiments described above, it is possible to present to the user whether a predetermined series of actions is suitable for action identification.

[0155] In the above-mentioned embodiment, "determining whether it is A" may mean "determining that it is A", or "determining that it is not A", or "determining whether it is A or not".

[0156] Each control program executed by the movement determining device 1 of this embodiment is provided in a state that it is pre-installed in a recording medium such as a ROM.

[0157] Each control program executed by the operation determination device 1 of this embodiment may be configured to be provided by being recorded in an installable or executable format on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk).

[0158] Furthermore, each control program executed by the movement determining device 1 of the present embodiment may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, each control program executed by the movement determining device 1 of the present embodiment may be configured to be provided or distributed via a network such as the Internet.

[0159] For example, the control program executed by the control device 10 in the motion determination device 1 of this embodiment has a modular configuration including execution units for the above-mentioned functions (e.g., the acquisition unit 101, the motion determination unit 102, and the output unit 103). The processor 11 of the control device 10 reads the control program from the recording medium and loads the above-mentioned units onto a main storage device such as the RAM 13 of the control device 10. As a result, the above-mentioned units are generated on the main storage device.

[0160] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims.

[0161] (Additional Note) (1) A memory for storing a work pattern indicating a predetermined series of work operations and a sequence thereof; an image interface for acquiring a captured image of a target area; a notification interface for notifying a user; In a first determination block, a motion of the user is identified based on the captured image, and it is determined whether the identified motion of the user satisfies a predetermined condition required for motion determination; When the predetermined condition is satisfied in the first judgment block, the user is notified that the motion judgment will be performed, and in a second judgment block, a motion of the user performing the series of tasks is identified based on the captured image, and the motion judgment is performed to determine whether the identified motion of the user is consistent with the work pattern. outputting information indicating a result of the motion determination; At least one processor; An information processing device comprising: (2) the at least one processor classifies the user's motion using a classifier common between the first decision block and the second decision block to identify the user's motion. The information processing device according to (1) above. (3) The at least one processor identifies the user's action based on the captured image using different classifiers between the first decision block and the second decision block. The information processing device according to (1) above. (4) The at least one processor, in the first decision block, identifies an action of the user with respect to a part of the actions of the series of tasks identified in the second decision block. An information processing device according to any one of (1) to (3) above. (5) The predetermined condition is that a proportion of a time period of the action identified as the part of the action in a predetermined determination time period is equal to or greater than a predetermined threshold value. The information processing device according to (4) above. (6) The at least one processor notifies the user that the user's motion does not satisfy a predetermined condition required for motion judgment until the predetermined condition is satisfied in the first judgment block. An information processing device according to any one of (1) to (5) above. (7) the at least one processor, in each of the first and second decision blocks, identifies the user's motion by classifying the user's motion using a classifier; The classifier is based on a model trained by deep learning so as to output a classification result in which the user's motion included in the captured image is classified into one of a plurality of motion classes in response to the input of the captured image. An information processing device according to any one of (1) to (6) above. (8) In an information processing apparatus having at least one processor, In a first determination block, a user's motion is identified based on a captured image of a target area, and it is determined whether the identified user's motion satisfies a predetermined condition required for motion determination; If the predetermined condition is satisfied in the first decision block, notifying the user that the motion determination will be performed; In a second judgment block, a motion of the user performing a predetermined series of tasks is identified based on the captured image, and the motion judgment is performed to determine whether the identified motion of the user is consistent with a task pattern indicating the motion and sequence of the series of tasks. outputting information indicating a result of the motion determination; Information processing methods. (9) In a first determination block, a user's motion is identified based on a captured image of a target area, and it is determined whether the identified user's motion satisfies a predetermined condition required for motion determination; If the predetermined condition is satisfied in the first decision block, notifying the user that the motion determination will be performed; In a second judgment block, a motion of the user performing a predetermined series of tasks is identified based on the captured image, and the motion judgment is performed to determine whether the identified motion of the user is consistent with a task pattern indicating the motion and sequence of the series of tasks. outputting information indicating a result of the motion determination; A program that makes a computer do something. (10) A recording medium (Computer Program Product) having recorded thereon the program described in (9) above, which is executed by a computer. [Explanation of symbols]

[0162] 1. Motion determination device (system) 3. Camera 5 Input / Output Devices (Display Terminals) 6. Leader 10 Control device (information processing device) 11 Processors 12 ROM 13 RAM 14 NVM 15 Communication I / F 16 Camera I / F (Image Interface) 18 Input / Output I / F (notification interface) 19 Reader I / F (Identification Information Interface) 20. Sink 21 Faucet 22 Pattern setting table 23 Feature Model Table 24 Action Class Table 25 Identification Table 101 Acquisition Department 102 Operation judgment section 103 Output section [Prior art documents] [Patent documents]

[0163] [Patent Document 1] JP 2022-187267 A

Claims

1. A memory for storing a work pattern indicating a predetermined series of work operations and a sequence thereof; an image interface for acquiring a captured image of a target area; a notification interface for notifying a user; In a first determination block, a motion of the user is identified based on the captured image, and it is determined whether the identified motion of the user satisfies a predetermined condition required for motion determination; When the predetermined condition is satisfied in the first judgment block, the user is notified that the motion judgment will be performed, and in a second judgment block, a motion of the user performing the series of tasks is identified based on the captured image, and the motion judgment is performed to determine whether the identified motion of the user is consistent with the work pattern. outputting information indicating a result of the motion determination; At least one processor; An information processing device comprising:

2. the at least one processor classifies the user's motion using a classifier common between the first decision block and the second decision block to identify the user's motion. The information processing device according to claim 1 .

3. The at least one processor, in the first decision block, identifies an action of the user with respect to a part of the actions of the series of tasks identified in the second decision block.

3. The information processing device according to claim 1 or 2.

4. The predetermined condition is that a proportion of a time period of the action identified as the part of the action in a predetermined determination time period is equal to or greater than a predetermined threshold value. The information processing device according to claim 3 .

5. The at least one processor notifies the user that the user's motion does not satisfy a predetermined condition required for motion determination until the predetermined condition is satisfied in the first determination block. The information processing device according to claim 1 .

6. In a first determination block, a user's motion is identified based on a captured image of a target area, and it is determined whether the identified user's motion satisfies a predetermined condition required for motion determination; If the predetermined condition is satisfied in the first decision block, notifying the user that the motion determination will be performed; In a second judgment block, a motion of the user performing a predetermined series of tasks is identified based on the captured image, and the motion judgment is performed to determine whether the identified motion of the user is consistent with a task pattern indicating the motion and sequence of the series of tasks. outputting information indicating a result of the motion determination; A program that makes a computer do something.

Images