Information processing device and program

Abstract

To provide an information processing apparatus and a program for effectively determining an action of a human body.SOLUTION: An action determination apparatus 1 serving as an information processing apparatus for determining a hand-washing action of a user includes an image interface 15, a distance information interface 16, and a processor 11. The image interface acquires a captured image including a human body. The distance information interface acquires distance information from a distance sensor 4 in a region including the human body. The processor is configured to: determine, as an action of the human body, a first action based on an image captured at a first timing; determine a second action based on an image captured at a subsequent second timing; select, when the actions are coincident with each other, the second action in the second timing; determine, when the actions are not coincident with each other, a third action based on distance information in the second timing; and select an action of the human body in the second timing on the basis of the second action and the third action.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an information processing device and a program. [Background technology]

[0002] A system for determining hand (human body) movements has been provided to monitor whether hand washing has been performed according to a predetermined procedure. Some such systems acquire information from multiple sensors, such as color information from a color information measurement sensor such as an RGB camera or distance information from a distance sensor that measures distance, in an area including the hand. The system determines hand movements from each sensor information according to a predetermined algorithm. The system determines hand movements based on the multiple movements determined.

[0003] However, such a system has a problem in that it requires a high computational load because it constantly executes multiple algorithms, including image information processing, simultaneously. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2017-097577 Summary of the Invention [Problem to be solved by the invention]

[0005] In order to solve the above problem, an information processing device and a program capable of effectively determining the movement of a human body are provided. [Means for solving the problem]

[0006] According to an embodiment, an information processing device includes an image interface, a distance information interface, and a processor. The image interface acquires a captured image including a human body. The distance information interface acquires distance information from a distance sensor that measures distances in an area including the human body. The processor determines a first movement of the human body based on the captured image captured at a first timing, determines a second movement of the human body based on the captured image captured at a second timing subsequent to the first timing, and selects the second movement as the movement of the human body at the second timing if the first movement and the second movement are consistent. If the first movement and the second movement are inconsistent, determines a third movement of the human body based on the distance information measured at the second timing, and selects the movement of the human body at the second timing based on the second movement and the third movement. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram schematically illustrating an example of the configuration of a motion determining system according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of the movement determining device according to the embodiment. [Figure 3] FIG. 3 is a flowchart illustrating an example of the operation of the movement determining device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments will be described with reference to the drawings. A motion determination system according to an embodiment determines hand motions associated with hand washing. The motion determination system captures an image of the user's hand in a sink or the like. The motion determination system also measures the distance from a predetermined reference plane or point in an area including the user's hand. The motion determination system determines the user's hand motion from the captured image and the measured distance. For example, the motion determination system determines the shape of the hand, which is classified according to the area being washed (e.g., the back of the hand, between the fingers, or the thumb), as the hand motion.

[0009] 1 is a schematic diagram illustrating an example of the configuration of a motion determining system 100 according to an embodiment. As shown in FIG. 1, the motion determining system 100 includes a motion determining device 1, a sink 20, and the like.

[0010] The sink 20 is a water tub where the user P washes his / her hands. The sink 20 has a recessed structure. A drain groove is formed in the bottom of the sink 20. In addition, a faucet 21 is installed in the sink 20. The faucet 21 discharges water for the user P to wash his / her hands. The discharged water is discharged from a drain in the sink 20.

[0011] Above the sink 20, a motion determining device 1 is installed. The movement determining device 1 (information processing device) determines the movement of the user P. Here, the movement determining device 1 determines the hand movement of the user P while washing his / her hands.

[0012] The movement determining device 1 includes a camera 3 and a distance sensor 4 . The camera 3 is installed above the sink 20 facing downward. The camera 3 captures an area (photographed area) including the hand of the user P. The camera 3 may be installed so as to capture an image of an item from diagonally above. The position and orientation at which the camera 3 is installed are not limited to a specific configuration. The camera 3 captures a two-dimensional image (photographed image). The camera 3 outputs color information (RGB (Red Green Blue)) for each dot.

[0013] The distance sensor 4 is installed facing downward on the top of the sink 20. The distance sensor 4 measures the distance in an area (measurement area) that includes the hand of the user P. The distance sensor 4 measures the distance up to the distance sensor 4 in the measurement area.

[0014] Based on the measurement results, the distance sensor 4 generates distance information indicating the distance from a predetermined reference plane (or a predetermined reference point) to each point. For example, the distance information may indicate the coordinates of each point in a predetermined three-dimensional coordinate system.

[0015] For example, the distance sensor 4 includes a light source and a sensor that detects reflected light of light emitted from the light source. The distance sensor 4 measures the distance based on the reflected light of the light (visible light or invisible light) emitted from the light source. For example, the distance sensor 4 may use a ToF (Time-for-Flight) method that measures the distance to the measurement object based on the time it takes for the emitted light to reflect off the measurement object and reach the distance sensor 4.

[0016] The distance sensor 4 may be a sensor that calculates distance based on the parallax between images captured by two cameras (stereo cameras), or a sensor that projects a dot pattern and measures distance based on the distortion of the dot pattern. The configuration of the distance sensor 4 is not limited to a specific configuration.

[0017] Next, the control system of the movement determining device 1 will be described. Fig. 2 is a block diagram showing an example of the configuration of a control system of the movement determination device 1. As shown in Fig. 2, the movement determination device 1 includes a processor 11, a ROM 12, a RAM 13, an NVM 14, a camera interface 15, a distance sensor interface 16, an input / output interface 17, a communication interface 18, a camera 3, a distance sensor 4, an input / output device 5, etc.

[0018] The processor 11 is communicatively connected to the ROM 12, RAM 13, NVM 14, camera interface 15, distance sensor interface 16, input / output interface 17, and communication interface 18. The camera interface 15 is communicatively connected to the camera 3. The distance sensor interface 16 is communicatively connected to the distance sensor 4. The input / output interface 17 is communicatively connected to the input / output device 5. The camera 3 and the distance sensor 4 are as described above.

[0019] The input / output device 5 is an interface that receives instructions from an operator and displays various information to the operator. The input / output device 5 is made up of an operation unit that receives instructions and a display unit that displays information.

[0020] The input / output device 5 transmits a signal indicating an operation received from an operator to the processor 11 as the operation of the operation unit. For example, the operation unit is configured with a touch panel. The operation unit may also include a keyboard or a numeric keypad.

[0021] The input / output device 5 operates as a display unit to display an image from the processor 11. For example, the display unit is configured with a liquid crystal monitor. The display unit may be formed integrally with a touch panel serving as an operation unit.

[0022] The processor 11 controls the overall operation of the motion determination device 1. That is, the processor 11 controls the overall operation of the motion determination device 1. The processor 11 may include an internal cache and various interfaces. The processor 11 realizes various processes by executing programs stored in advance in the internal cache, the ROM 12, or the NVM 14.

[0023] Some of the various functions realized by the processor 11 executing the programs may be realized by hardware circuits. In this case, the processor 11 controls the functions executed by the hardware circuits.

[0024] The ROM 12 is a non-volatile memory in which control programs, control data, etc. are stored in advance. The control programs and control data stored in the ROM 12 are pre-installed in accordance with the specifications of the movement determination device 1. The ROM 12 stores, for example, a program for controlling a circuit board of the movement determination device 1.

[0025] RAM 13 is a volatile memory. RAM 13 temporarily stores data being processed by processor 11. RAM 13 stores various application programs based on instructions from processor 11. RAM 13 may also store data necessary for executing application programs and execution results of application programs.

[0026] The NVM 14 is a nonvolatile memory to which data can be written and rewritten. The NVM 14 is configured, for example, by a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The NVM 14 stores control programs, applications, various data, and the like according to the operational use of the operation determination device 1.

[0027] Camera interface 15 (image interface) is an interface for transmitting and receiving data to and from camera 3. For example, camera interface 15 transmits a signal instructing camera 3 to take a picture, based on the control of processor 11. Camera interface 15 also acquires a captured image from camera 3. For example, camera interface 15 may be one that supports a USB (Universal Serial Bus) connection, or one that supports a connection via Camera Link.

[0028] The distance sensor interface 16 (distance information interface) is an interface for transmitting and receiving data to and from the distance sensor 4. For example, the distance sensor interface 16 transmits a signal that causes the distance sensor 4 to acquire distance information under the control of the processor 11. The distance sensor interface 16 also acquires distance information from the distance sensor 4. The distance sensor interface 16 transmits the acquired distance information to the processor 11. For example, the distance sensor interface 16 may support a USB connection.

[0029] The input / output interface 17 is an interface for transmitting and receiving data to and from the input / output device 5. For example, the input / output interface 17 receives a signal indicating an operation accepted from an operator from the input / output device 5. The input / output interface 17 transmits the received signal to the processor 11. Furthermore, the input / output interface 17 transmits information indicating a screen to be displayed to the operator to the input / output device 5 under the control of the processor 11. For example, the input / output interface 17 may be one that supports a USB connection or one that supports a connection via a parallel interface.

[0030] The communication interface 18 is an interface for transmitting and receiving data to and from an external device. The communication interface 18 connects to the external device via a network or the like. For example, the communication interface 18 supports wired or wireless LAN (Local Area Network) connections.

[0031] The movement determining device 1 may include other components as needed in addition to the components shown in FIGS. 1 and 2, or certain components may be excluded from the movement determining device 1. For example, the movement determining device 1 is a PC, a tablet PC, or a smartphone.

[0032] Next, a description will be given of functions realized by the movement determining device 1. The functions realized by the movement determining device 1 are realized by the processor 11 executing a program stored in the internal memory, the ROM 12, the NVM 14, or the like.

[0033] First, the processor 11 has a function of acquiring a photographed image including the user P's hand. The processor 11 determines whether a hand is detected within the shooting range of the camera 3 and the measurement range of the distance sensor 4. For example, the processor 11 determines whether a hand is included in the image captured by the camera 3 according to a predetermined image processing algorithm. The processor 11 may also determine whether a hand is detected based on distance information from the distance sensor 4.

[0034] When it is determined that a hand has been detected within the photographing range of camera 3 and the measurement range of distance sensor 4, processor 11 acquires the photographed image from camera 3 via camera interface 15. Note that when it is determined that a hand has been detected according to a predetermined image processing algorithm, processor 11 may acquire the photographed image as a photographed image including the hand of user P.

[0035] Furthermore, the processor 11 may acquire a captured image including the user's hand from the camera 3 in accordance with an operation input through the input / output device 5.

[0036] The processor 11 also has a function of acquiring distance information measured in a measurement region including the user P's hand. When a captured image including the hand of user P is acquired, processor 11 acquires distance information from distance sensor 4 via distance sensor interface 16. Note that, when it is determined that a hand has been detected based on the distance information, processor 11 may acquire the distance information as distance information measured in a measurement area including user P's hand.

[0037] The processor 11 also has a function of determining hand movements based on the captured image in accordance with a predetermined algorithm. After acquiring the distance information, processor 11 extracts a human body region in which a hand appears from the captured image. For example, processor 11 extracts the human body region by detecting edges from the captured image. Alternatively, processor 11 may extract the human body region based on the distance information.

[0038] After identifying the human body region, the processor 11 acquires an image of the human body region from the captured image. After acquiring the image of the human body region, the processor 11 calculates the likelihood of each motion based on the acquired image. For example, the likelihood is the probability that the motion corresponding to the likelihood is a motion of the user P.

[0039] Here, each action is a hand shape classified in relation to hand washing, for example, each action is a hand shape classified according to the part of the hand being washed (back of the hand, between the fingers, thumb, etc.).

[0040] For example, the NVM 14 stores in advance a model (e.g., a network generated by deep learning) that outputs the likelihood of an action when an image is input. The processor 11 inputs an acquired image into the model stored in the NVM 14 to calculate the likelihood of each action.

[0041] The processor 11 may extract features from the acquired image and calculate the likelihood of each action based on the extracted features. The method by which the processor 11 calculates the likelihood from the captured image is not limited to a specific method.

[0042] After calculating the likelihood of each motion, processor 11 determines the motion with the highest likelihood as the hand motion of user P. Alternatively, processor 11 may determine the motion with the highest likelihood among the likelihoods equal to or greater than a predetermined threshold as the hand motion of user P. In this case, if there is no likelihood equal to or greater than the predetermined threshold, processor 11 may output an error.

[0043] Here, the hand movement determined by the processor 11 from a captured image taken at a certain timing (t) is defined as f(t).

[0044] The processor 11 also has a function of determining whether f(t1) determined at a first timing (t1) matches f(t2) determined at a second timing (t2) following t1.

[0045] Here, t2 is the timing when the processor 11 acquires the current captured image and distance information, and t1 is the timing when the processor 11 acquires the immediately preceding captured image and distance information.

[0046] As described above, processor 11 determines the hand movement (f(t2), second movement) from the captured image. After determining f(t2), processor 11 determines whether f(t1) (first movement) and f(t2) are consistent. For example, processor 11 determines whether f(t1) and f(t2) match.

[0047] If it is determined that f(t1) and f(t2) match, the processor 11 selects f(t2) (or f(t1)) as the current hand movement of the user P.

[0048] If f(t1) does not exist (if the movement is determined from the first captured image), the processor 11 may select f(t2) as the current hand movement of the user P.

[0049] Furthermore, the processor 11 has a function of determining the hand movement based on the distance information in accordance with a predetermined algorithm when f(t1) and f(t2) do not match. The processor 11 determines the hand movement based on the distance information acquired at t2.

[0050] If f(t1) and f(t2) do not match, the processor 11 calculates the likelihood of each action based on the distance information acquired at t2.

[0051] For example, the NVM 14 stores in advance a model (e.g., a network generated by deep learning) that outputs the likelihood of an action when distance information is input. The processor 11 inputs the acquired distance information into the model stored in the NVM 14 to calculate the likelihood of each action.

[0052] The processor 11 may extract a feature amount from the acquired distance information, and may calculate the likelihood of each action based on the extracted feature amount. The method by which the processor 11 calculates the likelihood from the distance information is not limited to a specific method.

[0053] After calculating the likelihood of each motion, processor 11 determines the motion with the highest likelihood as the hand motion of user P. Alternatively, processor 11 may determine the motion with the highest likelihood among the likelihoods equal to or greater than a predetermined threshold as the hand motion of user P. In this case, if there is no likelihood equal to or greater than the predetermined threshold, processor 11 may output an error.

[0054] Here, the hand movement determined by processor 11 from distance information acquired at a certain timing (t) is defined as g(t). That is, the hand movement determined from distance information acquired at t2 is defined as g(t2) (third movement).

[0055] The processor 11 also has a function of determining whether f(t2) and g(t2) are consistent.

[0056] After determining g(t2), processor 11 determines whether f(t2) and g(t2) are consistent. For example, processor 11 determines whether f(t2) and g(t2) match.

[0057] If it is determined that f(t2) and g(t2) match, the processor 11 selects f(t2) (or g(t2)) as the current hand movement of the user P.

[0058] Furthermore, when f(t2) and g(t2) do not match, the processor 11 has a function of determining the hand movement based on the likelihood of f(t2) and the likelihood of g(t2). If f(t2) and g(t2) do not match, the processor 11 compares the likelihood of f(t2) with the likelihood of g(t2). The processor 11 selects either f(t2) or g(t2), whichever has the higher likelihood, as the current hand movement of the user P.

[0059] For example, if the likelihood of f(t2) is greater than the likelihood of g(t2), processor 11 selects f(t2) as the hand motion of the current user P. Also, if the likelihood of g(t2) is greater than the likelihood of f(t2), processor 11 selects g(t2) as the hand motion of the current user P.

[0060] Note that processor 11 may multiply the likelihood of f(t2) or the likelihood of g(t2) by a predetermined coefficient. For example, if a determination method using distance information is more reliable, processor 11 may multiply the likelihood of g(t2) by a coefficient of 1 or more. In this case, processor 11 compares the likelihood of f(t2) with the likelihood of g(t2) after multiplication.

[0061] Next, an example of the operation of the movement determining device 1 will be described. 3 is a flowchart for explaining an example of the operation of the movement determining device 1. Here, it is assumed that the processor 11 of the movement determining device 1 resets i, which counts the number of determinations (substituting 0 into i).

[0062] First, processor 11 determines whether a hand has been detected (S11). If processor 11 determines that a hand has not been detected (S11, NO), processor 11 returns to S11.

[0063] When it is determined that a hand has been detected (S11, YES), the processor 11 acquires a captured image from the camera 3 through the camera interface 15 (S12). After acquiring the captured image, the processor 11 acquires distance information from the distance sensor 4 through the distance sensor interface 16 (S13). After acquiring the distance information, the processor 11 extracts a human body region from the captured image (S14).

[0064] When the human body region is extracted from the captured image, processor 11 determines the hand movement (f(t2)) of user P based on the image within the human body region (S15). After determining f(t2), processor 11 determines whether f(t1) and f(t2) are consistent (S16).

[0065] If it is determined that f(t1) and f(t2) are consistent (S16, YES), processor 11 selects f(t2) as the hand movement of user P (S17). After selecting f(t2) as the hand movement of user P, processor 11 increments i (i=i+1) (S18).

[0066] If it is determined that f(t1) and f(t2) do not match (S16, NO), the processor 11 determines the hand movement (g(t2)) of the user P based on the distance information (S20). After determining g(t2), the processor 11 determines whether f(t2) and g(t2) match (S21).

[0067] If it is determined that f(t2) and g(t2) match (S21, YES), the processor 11 proceeds to S17.

[0068] If it is determined that f(t2) and g(t2) do not match (S21, NO), the processor 11 acquires the likelihood of f(t2) and the likelihood of g(t2) (S22). After acquiring the likelihood of f(t2) and the likelihood of g(t2), the processor 11 selects f(t2) or g(t2) as the hand movement of the user P based on the likelihood of f(t2) and the likelihood of g(t2) (S23).

[0069] When f(t2) or g(t2) is selected as the hand movement of the user P, the processor 11 proceeds to S18.

[0070] When i is incremented (i=i+1), the processor 11 determines whether to end the determination of the hand movement of the user P (S19). For example, the processor 11 determines whether the hand has moved out of the shooting range of the camera 3 and the measurement range of the distance sensor 4. The processor 11 may also determine whether an operation to end the movement determination has been input through the input / output device 5.

[0071] When it is determined that the determination of the hand movement of user P is not to be ended (S19, NO), processor 11 returns to S12. Note that processor 11 may return to S12 after a predetermined time (for example, 0.1 seconds) has elapsed since it was determined that the determination of the hand movement of user P is not to be ended.

[0072] When it is determined that the determination of the hand movement of the user P is to be ended (S19, YES), the processor 11 ends the operation.

[0073] When the processor 11 selects a hand motion (S17 or S23), the processor 11 may store information indicating the selected hand motion in the NVM 14. The processor 11 may also determine whether the user P has washed their hands in a predetermined procedure by acquiring the information indicating the selected hand motion in chronological order.

[0074] The processor 11 may also transmit information indicating the hand movement to an external device via the communication interface 18.

[0075] Furthermore, processor 11 may determine hand movements such as hand posture, wrist angle, angle of each joint of each finger, gesture, etc. The content of hand movements determined by processor 11 is not limited to a specific configuration.

[0076] Furthermore, the processor 11 may determine a motion other than that of the hand of the user P. For example, the processor 11 may determine the posture of the user P (standing, sitting, etc.).

[0077] Furthermore, the camera 3 or the distance sensor 4 may be installed separately from the movement determining device 1.

[0078] The movement determination system configured as described above determines hand movement from a captured image including a hand. When the movement determined from the immediately preceding captured image differs from the movement determined from the current captured image, the movement determination system determines the hand movement based on distance information. The movement determination system determines the hand movement based on the movement determined from the current captured image and the movement determined from the distance information. As a result, the movement determination system can determine the hand movement based on two determination methods. Therefore, the movement determination system can improve the accuracy of hand movement determination.

[0079] Furthermore, if the movement determined from the immediately preceding captured image matches the movement determined from the current captured image, the movement determination system determines the movement determined from the current captured image as a hand movement. As a result, the movement determination system can reduce the frequency of determinations based on distance information. Therefore, the movement determination system can reduce the calculation load.

[0080] The program may be transferred in a state where it is stored in a device, or in a state where it is not stored in a device. In the latter case, the program may be transferred via a network, or in a state where it is recorded on a recording medium. The recording medium is a non-transitory tangible medium. The recording medium is a computer-readable medium. The form of the recording medium is not important as long as it is a medium that can store the program and is computer-readable, such as a CD-ROM or a memory card.

[0081] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0082] 1...motion determination device, 3...camera, 4...distance sensor, 5...input / output device, 11...processor, 12...ROM, 13...RAM, 14...NVM, 15...camera interface, 16...distance sensor interface, 17...input / output interface, 18...communication interface, 20...sink, 21...faucet, 100...motion determination system

Claims

1. an image interface for acquiring a photographed image including a human body; a distance information interface for acquiring distance information from a distance sensor that measures distances in an area including the human body; determining a first movement of the human body based on the captured image captured at a first timing; determining a second movement of the human body based on the captured image captured at a second timing subsequent to the first timing; If the first motion and the second motion are consistent with each other, selecting the second motion as the motion of the human body at the second timing; If the first movement and the second movement do not match, a third movement of the human body is determined based on the distance information measured at the second timing; and selecting a motion of the human body at the second timing based on the second motion and the third motion; a processor; An information processing device comprising:

2. the processor, when the second motion and the third motion are consistent with each other, selects the second motion as the motion of the human body at the second timing; The information processing device according to claim 1 .

3. When the second motion and the third motion do not match, the processor selects the second motion or the third motion as the motion of the human body at the second timing based on the likelihood of the second motion and the likelihood of the third motion. The information processing device according to claim 2 .

4. the processor selects, as the human body motion at the second timing, one of the second motion and the third motion, whichever has a higher likelihood; The information processing device according to claim 3 .

5. The human body is a hand. The information processing device according to claim 1 .

6. A program executed by a processor, the processor, a function of determining a first motion of a human body based on a captured image including the human body, the captured image being captured at a first timing; a function of determining a second motion of the human body based on the captured image captured at a second timing subsequent to the first timing; a function of selecting the second motion as the motion of the human body at the second timing when the first motion and the second motion are consistent with each other; a function of determining a third motion of the human body based on distance information measured at the second timing from a distance sensor that measures a distance in an area including the human body when the first motion and the second motion do not match; a function of selecting a motion of the human body at the second timing based on the second motion and the third motion; A program to make this happen.

Images