Operation identification device and program

Abstract

To provide a movement specifying apparatus and a program which can guide a human body to a proper position.SOLUTION: According to an embodiment of the present invention, a movement specifying apparatus has an illumination unit, a camera, and a processor. The illumination unit irradiates a predetermined irradiated region with visible light having a predetermined wavelength. The camera acquires a captured image by capturing an image of an image captured region including the irradiated region. The processor specifies, based on the captured image, movement of a human body captured in the captured image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Embodiments of the present invention relate to an action specifying device and a program.

Background Art

[0002] There is provided an action specifying device that specifies the action of a human body performed by photographing a part of the human body such as a hand. Some such action specifying devices specify the action of a human body from an image obtained by photographing a predetermined region including the human body.

[0003] If a photographed part of the human body is not photographed at an appropriate position, the action specifying device may not be able to appropriately specify the action of the human body. Conventionally, the action specifying device has a problem that it cannot guide the human body to an appropriate position with respect to the camera to be photographed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In order to solve the above problems, there is provided an action specifying device and a program that can guide a human body to an appropriate position with respect to a camera that photographs the human body.

Means for Solving the Problems

[0006] According to an embodiment, the action specifying device includes illumination, a camera, and a processor. The illumination irradiates visible light having a predetermined wavelength onto a predetermined irradiation region. The camera photographs a photographing region including the irradiation region to obtain a photographed image. The processor specifies the action of the human body shown in the photographed image based on the photographed image. A different wavelength configuration from ambient light The processor identifies an object region in which the human body is captured from the captured image based on the wavelength, and identifies the movement of the human body based on the image contained in the object region. ​ [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 is a schematic diagram showing an example of the configuration of the operation identification system according to the embodiment. [Figure 2] Figure 2 is a schematic diagram showing the camera and lighting of the operation identification device according to the embodiment. [Figure 3] Figure 3 is a block diagram showing an example configuration of an operation identification device according to an embodiment. [Figure 4] Figure 4 shows an example of an effective area according to the embodiment. [Figure 5] Figure 5 shows an example of an object region according to the embodiment. [Figure 6] Figure 6 is a flowchart showing an example of the operation of the operation identification device according to the embodiment. [Figure 7] Figure 7 is a flowchart showing an example of operation of the operation identification device according to the embodiment. [Modes for carrying out the invention]

[0008] The embodiments will be described below with reference to the drawings. The action identification system according to this embodiment identifies the movements of the user's hands (human body) related to handwashing. The action identification system photographs the user's hands in a sink or the like. Based on the captured images, the action identification system identifies the movements of the user's hands. For example, the action determination system identifies the state of the hands, which can be classified as hand movements based on the area being washed (back of the hand, between the fingers, thumb, etc.).

[0009] Figure 1 schematically shows an example of the configuration of the operation identification system 100 according to the embodiment. As shown in Figure 1, the operation identification system 100 consists of an operation identification device 1 and a sink 20, etc.

[0010] Sink 20 is a basin where user P washes their hands. Sink 20 has a recessed structure. A drain is also formed at the bottom of sink 20. Furthermore, a faucet 21 is installed in the sink 20. The faucet 21 dispenses water for user P to wash their hands. The dispensed water is drained out of the drain of the sink 20.

[0011] An operation identification device 1 is installed on top of the sink 20. The action identification device 1 identifies the actions of user P. In this case, the action identification device 1 identifies the hand movements of user P while washing hands.

[0012] The motion detection device 1 includes a camera 3, a depth sensor 4, and lighting 6. Camera 3 is installed facing downwards above the sink 20. Camera 3 captures the area (shooting area) including the hands of user P as they wash their hands in the sink 20. Camera 3 may also be installed to photograph objects from an oblique angle above. The position and orientation of camera 3 are not limited to a specific configuration. Camera 3 captures a two-dimensional image (captured image). Camera 3 outputs color information (RGB (Red Green Blue)) for each dot.

[0013] The depth sensor 4 is installed facing downwards on top of the sink 20. The depth sensor 4 measures distance in the area including user P's hand. The depth sensor 4 measures distance from each point to the depth sensor 4. Here, the depth sensor 4 functions as a sensor that detects the position of the user's hand.

[0014] The depth sensor 4 generates distance information indicating the distance from a predetermined reference plane (or predetermined reference point) to each point based on the measurement results. For example, the distance information may indicate the coordinates of each point in a predetermined three-dimensional coordinate system.

[0015] For example, the depth sensor 4 includes a light source and a sensor that detects the reflected light of the light irradiated from the light source. The depth sensor 4 measures the distance based on the reflected light of the light (visible light or invisible light) irradiated from the light source. For example, the depth sensor 4 may perform a ToF (Time-of-Flight) method of measuring the distance to the measurement object based on the time from when the irradiated light is reflected by the measurement object until it reaches the depth sensor 4.

[0016] The depth sensor 4 may calculate the distance based on the parallax of each image taken by two cameras (a stereo camera). Also, the depth sensor 4 may measure the distance from the distortion of the dot pattern by projecting the dot pattern. The configuration of the depth sensor 4 is not limited to a specific configuration.

[0017] The illumination 6 irradiates visible light (irradiation light) having a predetermined wavelength (for example, any one or two of RGB) with a wavelength composition different from that of the ambient light. The illumination 6 irradiates the irradiation light to a predetermined area (irradiation area) included in the imaging area.

[0018] For example, the illumination 6 may be composed of a light source that irradiates visible light having a predetermined wavelength. Also, the illumination 6 may be composed of a light source that irradiates white light and a film that transmits or blocks light of a predetermined wavelength. Also, the illumination 6 may include a cap or the like that restricts the irradiated light from the light source to a predetermined irradiation area.

[0019] FIG. 2 shows the imaging area 3a of the camera 3 and the irradiation area 6a of the illumination 6. As shown in FIG. 2, the camera 3 photographs the hand of the user P from above. That is, the imaging area 3a of the camera 3 includes the hand of the user P.

[0020] Also, the illumination 6 is installed near the camera 3. The illumination 6 irradiates the irradiation light downward. The irradiation area 6a of the illumination 6 is included in the imaging area 3a within a predetermined range. That is, the irradiation area 6a is included in the imaging area 3a at the height of the hand of the user P.

[0021] Next, we will explain the control system of the operation identification device 1. Figure 3 is a block diagram showing an example configuration of the control system of the operation identification device 1. As shown in Figure 3, the operation identification device 1 includes a processor 11, ROM 12, RAM 13, NVM 14, camera interface 15, depth sensor interface 16, input / output interface 17, communication interface 18, lighting interface 19, camera 3, depth sensor 4, input / output device 5, and lighting 6, among others.

[0022] The processor 11 is connected to the ROM 12, RAM 13, NVM 14, camera interface 15, depth sensor interface 16, input / output interface 17, communication interface 18, and lighting interface 19 in a communicative manner. The camera interface 15 is connected to the camera 3 in a communicative manner. The depth sensor interface 16 is connected to the depth sensor 4 in a communicative manner. The input / output interface 17 is connected to the input / output device 5 in a communicative manner. The lighting interface 19 is connected to the lighting 6 in a communicative manner. Camera 3, depth sensor 4, and lighting 6 are as described above.

[0023] The input / output device 5 is an interface that receives instructions from the operator and displays various information to the operator. The input / output device 5 consists of an operation unit that receives instructions and a display unit that displays information.

[0024] The input / output device 5, as part of the operation of the control unit, transmits a signal to the processor 11 indicating the operation received from the operator. For example, the control unit consists of a touch panel. The control unit may also include a keyboard or a numeric keypad.

[0025] The input / output device 5 displays an image from the processor 11 as part of the display unit's operation. For example, the display unit is composed of a liquid crystal monitor. The display unit may also be integrally formed with a touch panel that serves as the operation unit.

[0026] The processor 11 controls the operation of the entire operation identification device 1. That is, the processor 11 controls the operation of the entire operation identification device 1. The processor 11 may be equipped with an internal cache and various interfaces. The processor 11 performs various processes by executing programs pre-stored in the internal cache, ROM 12, or NVM 14.

[0027] Furthermore, some of the various functions realized by the execution of a program by the processor 11 may be realized by hardware circuits. In this case, the processor 11 controls the functions executed by the hardware circuits.

[0028] ROM12 is a non-volatile memory in which control programs and control data are pre-stored. The control programs and control data stored in ROM12 are pre-programmed according to the specifications of the operation identification device 1. For example, ROM12 stores a program that controls the circuit board of the operation identification device 1.

[0029] RAM13 is volatile memory. RAM13 temporarily stores data being processed by processor 11. RAM13 stores various application programs based on instructions from processor 11. RAM13 may also store data necessary for the execution of application programs and the execution results of application programs.

[0030] NVM14 is a non-volatile memory that allows data to be written to and rewritten. NVM14 is composed of, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or flash memory. NVM14 stores control programs, applications, and various data according to the operational purpose of the operation-specific device 1.

[0031] The camera interface 15 is an interface for sending and receiving data to and from the camera 3. For example, the camera interface 15 sends a signal to the camera 3 to take a picture based on the control of the processor 11. The camera interface 15 also acquires the captured image obtained from the camera 3. For example, the camera interface 15 may support a USB (Universal Serial Bus) connection or a Camera Link connection.

[0032] The depth sensor interface 16 is an interface for sending and receiving data with the depth sensor 4. For example, the depth sensor interface 16 sends a signal to the depth sensor 4 to acquire distance information based on the control of the processor 11. The depth sensor interface 16 also acquires distance information from the depth sensor 4. The depth sensor interface 16 transmits the acquired distance information to the processor 11. For example, the depth sensor interface 16 may support a USB connection.

[0033] The input / output interface 17 is an interface for sending and receiving data with the input / output device 5. For example, the input / output interface 17 receives signals from the input / output device 5 indicating operations received from the operator. The input / output interface 17 transmits the received signals to the processor 11. The input / output interface 17 also transmits information to the input / output device 5 indicating the screen to be displayed to the operator, based on the control of the processor 11. For example, the input / output interface 17 may support a USB connection or a parallel interface connection.

[0034] The communication interface 18 is an interface for sending and receiving data with an external device. The communication interface 18 connects to the external device via a network or the like. For example, the communication interface 18 supports wired or wireless LAN (Local Area Network) connections.

[0035] The lighting interface 19 is an interface for sending and receiving data to and from the lighting 6. For example, the lighting interface 19 controls the lighting 6 to turn on and off according to the control from the processor 11. The lighting interface 19 may also supply power to the lighting 6. For example, the lighting interface 19 may support a USB connection or a parallel interface connection.

[0036] The camera interface 15, depth sensor interface 16, input / output interface 17, communication interface 18, and lighting interface 19 (or a part thereof) may be configured as an integrated unit.

[0037] Furthermore, the operation identification device 1 may have configurations other than those shown in Figure 3, or certain configurations may be excluded from the operation identification device 1. For example, the operation identification device 1 is a PC, tablet PC, or smartphone.

[0038] Next, the functions implemented by the operation identification device 1 will be described. The functions implemented by the operation identification device 1 are achieved when the processor 11 executes a program stored in the internal memory, ROM 12, or NVM 14, etc.

[0039] First, the processor 11 has a function to turn on the light 6 when it detects the user's hand. In this case, we assume that light 6 is turned off.

[0040] The processor 11 acquires distance information from the depth sensor 4 via the depth sensor interface 16. Upon acquiring the distance information, the processor 11 determines whether it has detected the user's hand based on the distance information. For example, the processor 11 determines whether it has detected an object in a predetermined space (for example, the space near the sink 20) ​​based on the distance information.

[0041] When it determines that a hand has been detected, the processor 11 turns on the light 6 via the lighting interface 19.

[0042] Furthermore, when the processor 11 turns on the lighting 6, it may display a screen on the input / output device 5 instructing the user to place their hand into the illuminated area 6a.

[0043] Furthermore, the processor 11 has the function of acquiring captured images that include the user P's hand. When the processor 11 determines that it has detected a user's hand, it acquires a captured image from the camera 3 via the camera interface 15. The processor 11 may also continuously acquire captured images after detecting a user's hand.

[0044] Furthermore, the processor 11 has a function to set an effective region in the captured image where the user's hand movements can be identified.

[0045] For example, when an image is captured, the processor 11 receives an input through the input / output device 5 to initiate the setting of the valid area. Upon receiving this input, the processor 11 displays the captured image on the input / output device 5.

[0046] Figure 4 shows an example of the operation in which the processor 11 sets the effective area. Figure 4 shows the captured image 31 displayed by the input / output device 5. The captured image 31 shows the user P and the illuminated area 6a in the captured image 31.

[0047] The processor 11 accepts input to the effective region 32 while displaying the captured image 31. For example, the processor 11 inputs the vertices of the effective region 32 through a tap operation on the input / output device 5. In this case, the effective region 32 is a rectangle.

[0048] When the vertices of the valid region 32 are input, the processor 11 obtains the coordinates of each vertex. After obtaining the coordinates of each vertex, the processor 11 stores the coordinates of each vertex in the NVM 14 as information indicating the valid region.

[0049] As shown in Figure 4, the effective region 32 includes the irradiation region 6a in the captured image 31. The user may adjust the irradiation region 6a after setting the effective region 32 so that it is included within the set effective region 32.

[0050] Furthermore, the processor 11 may input an operation to start setting the effective area before acquiring the captured image. For example, the processor 11 acquires the captured image after inputting this operation. Once the captured image 31 is acquired, the processor 11 displays the captured image 31 on the input / output device 5 and sets the effective area 32.

[0051] Furthermore, the processor 11 has the function of identifying the area (object area) in which user P's hand is visible from the captured image.

[0052] Here, it is assumed that the processor 11 has set the effective area 32. Furthermore, it is assumed that the processor 11 has acquired the captured image 31.

[0053] Upon acquiring the captured image 31, the processor 11 extracts the image contained within the effective region 32 from the captured image 31. After extracting the image contained within the effective region 32, the processor 11 extracts the object region in which the hand is visible from that image.

[0054] Here, the processor 11 extracts the object region in the irradiated area 6a based on the wavelength of the irradiated light. For example, the processor 11 extracts the gradient of the reflected light (light with a wavelength similar to the irradiated light) reflected from the irradiated light in the extracted image. Once the gradient of the reflected light is extracted, the processor 11 detects edges based on the gradient of the reflected light. Once edges are detected, the processor 11 identifies the object region based on the detected edges.

[0055] Figure 5 shows an example of the target area H. Figure 5 shows an image included in the effective area 32. Areas other than the target area H have been removed. As shown in Figure 5, the processor 11 identifies the area in the irradiation area 6a where the user P's hand is visible as the target area H.

[0056] Furthermore, the processor 11 has the function of identifying the user P's hand movements based on the image contained in the object region H.

[0057] Once the target region H is identified, the processor 11 obtains images contained within the target region H from the captured image. After obtaining the images contained within the target region H, the processor 11 estimates the user P's actions based on the obtained images.

[0058] Here, the processor 11 identifies the hand condition that is classified in relation to hand washing. For example, the processor 11 identifies the hand condition that is classified by the area being washed (back of the hand, between the fingers, thumb, etc.).

[0059] For example, NVM14 pre-stores a model (such as a network generated by deep learning) that outputs hand movements when an image is input. Processor 11 identifies the shape of the hand by inputting the acquired image into the model stored in NVM14.

[0060] Furthermore, the processor 11 may extract features from the acquired image. The processor 11 may also identify hand movements based on the extracted features.

[0061] The processor 11 may also identify hand movements such as hand posture, wrist angle, and the angles of each joint of each finger. The content of the hand movements identified by the processor 11 is not limited to a specific configuration.

[0062] Once a hand movement is identified, the processor 11 stores information indicating the hand movement (identification result) in the NVM 14. The processor 11 may also determine whether user P's handwashing was performed according to a predetermined procedure by acquiring the information indicating the hand movement in chronological order.

[0063] Furthermore, the processor 11 may transmit information indicating hand movements to an external device through the communication interface 18.

[0064] Next, we will explain an example of the operation of the operation identification device 1. First, we will explain an example of the operation in which the operation identification device 1 sets the effective area 32. Figure 6 is a flowchart illustrating an example of the operation of the operation identification device 1 to set the effective area 32.

[0065] First, the processor 11 of the motion identification device 1 determines whether a hand has been detected based on distance information (S11). If it determines that a hand has not been detected (S11, NO), the processor 11 returns to S11.

[0066] If it determines that a hand has been detected (S11, YES), the processor 11 turns on the light 6 via the lighting interface 19 (S12). When the light 6 is turned on, the processor 11 acquires the captured image from the camera 3 via the camera interface 15 (S13).

[0067] When the captured image is acquired, the processor 11 displays the captured image on the input / output device 5 and inputs the vertices of the valid region through the input / output device 5 (S14). Upon receiving the vertices of the valid region, the processor 11 acquires the coordinates of each input vertex (S15).

[0068] Once the coordinates of each vertex are obtained, the processor 11 stores the coordinates of each vertex in the NVM 14 as information indicating the valid region (S16). Once the coordinates of each vertex are stored in the NVM14 as information indicating the valid region, the processor 11 terminates its operation.

[0069] Next, we will describe an example of how the motion identification device 1 identifies hand movements. Figure 7 is a flowchart illustrating an example of how the motion identification device 1 identifies hand movements.

[0070] First, the processor 11 of the motion identification device 1 determines whether a hand has been detected based on distance information (S21). If it determines that a hand has not been detected (S21, NO), the processor 11 returns to S21.

[0071] If it determines that a hand has been detected (S21, YES), the processor 11 turns on the light 6 via the lighting interface 19 (S22). Once the light 6 is turned on, the processor 11 acquires the captured image from the camera 3 via the camera interface 15 (S23).

[0072] When the captured image is acquired, the processor 11 acquires the image contained within the valid region from the captured image (S24). After acquiring the image contained within the valid region, the processor 11 identifies the object region in which the user's hand is captured from the acquired image (S25).

[0073] Once the object region is identified, the processor 11 identifies the hand movement based on the image contained in the object region (S26). After identifying the hand movement, the processor 11 stores the identification result in the NVM 14 (S27). Once the identification result is stored in the NVM14, the processor 11 determines whether to terminate the hand movement identification (S28). For example, the processor 11 determines to terminate the hand movement identification if the hand is moved out of the active area. Alternatively, the processor 11 may receive an input from the input / output device 5 to terminate the hand movement identification.

[0074] If it is determined that the hand movement identification process is not complete (S28, NO), the processor 11 returns to S23.

[0075] When it determines that it has finished identifying the hand movements (S28, YES), the processor 11 terminates its operation. At this point, the processor 11 may turn off the light 6.

[0076] The processor 11 may also identify the object region based on the gradient of light across all wavelengths in the captured image. Furthermore, the processor 11 does not need to identify the target area. In this case, the processor 11 may identify the hand movement based on the image or captured image included in the effective area.

[0077] Furthermore, the processor 11 may detect the user's hand based on the captured image. In this case, the motion identification device 1 does not need to be equipped with a depth sensor 4.

[0078] In the above description, the motion identification device 1 uses the human hand as its target object, but the target object is not limited to this. The motion identification device 1 may identify actions other than those of user P's hand. For example, the motion identification device 1 may identify user P's posture (standing, sitting, etc.).

[0079] The motion identification system configured as described above has illumination such that the illuminated area is included in the effective area of ​​the captured image. The illumination emits visible light having a predetermined wavelength. As a result, the motion identification system can guide the user's hand to the appropriate position (i.e., the illuminated area). Therefore, the motion identification system can position the user's hand within the effective area of ​​the captured image. Thus, the motion identification system can effectively identify the movement of the user's hand.

[0080] The program according to this embodiment may be transferred while stored on an electronic device, or it may be transferred while not stored on an electronic device. In the latter case, the program may be transferred via a network, or it may be transferred while stored on a storage medium. The storage medium is a non-temporary tangible medium. The storage medium is a computer-readable medium. The storage medium can be any medium that is capable of storing a program and is readable by a computer, such as a CD-ROM or memory card, and its form is not limited.

[0081] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. The original claims of this application are included below. [C1] Illumination that irradiates a predetermined irradiation area with visible light having a predetermined wavelength, A camera that captures an image of the shooting area including the aforementioned irradiation area, and acquires an image of that area. An action identification device comprising: a processor that identifies the movement of a human body captured in the captured image based on the captured image. [C2] The aforementioned processor, Based on the aforementioned wavelength, the region of the object in which the human body is captured is identified from the captured image. Based on the image included in the object region, the movement of the human body is identified. The operation identification device described in C1. [C3] The aforementioned processor, A valid region including the irradiation area is set in the aforementioned captured image, The object region is identified from the image included in the effective region. The operation identification device described in C2. [C4] The system includes a sensor that detects the human body, The operation identification device according to C1, wherein the processor turns on the lighting when it detects the human body using the sensor. [C5] The aforementioned human body is a hand. An operation-specific device as described in any one of items C1 to C4. [C6] A program executed by a processor, The aforementioned processor, The lighting system has a function to irradiate a predetermined irradiation area with visible light having a predetermined wavelength, A function to acquire a captured image from the camera that includes the illumination area, Based on the aforementioned captured image, a function is provided to identify the movement of the human body captured in the image, A program that executes the command. [Explanation of symbols]

[0082] 1...Motion identification device, 3...Camera, 3a...Shooting area, 4...Depth sensor, 5...Input / output device, 6...Lighting, 6a...Illumination area, 11...Processor, 12...ROM, 13...RAM, 14...NVM, 15...Camera interface, 16...Depth sensor interface, 17...Input / output interface, 18...Communication interface, 19...Lighting interface, 20...Sink, 21...Faucet, 31...Captured image, 32...Effective area, 100...Motion identification system, H...Target area.

Claims

1. Illumination that irradiates a predetermined irradiation area with visible light having a predetermined wavelength with a wavelength configuration different from ambient light, A camera that captures an image of the shooting area including the aforementioned irradiation area, and acquires an image of that area. The system includes a processor that identifies the movement of a human body captured in the captured image based on the captured image, The aforementioned processor, Based on the aforementioned wavelength, the region of the object in which the human body is captured is identified from the captured image. A motion identification device that identifies the movement of the human body based on an image included in the object region.

2. The aforementioned processor, A valid region including the irradiation area is set in the aforementioned captured image, The object region is identified from the image included in the effective region. The operation identification device according to claim 1.

3. The human body is a hand, The system includes a sensor that detects the human body, The operation identification device according to claim 1 or 2, wherein the processor turns on the lighting when it detects the human body using the sensor.

4. A program executed by a processor, The aforementioned processor, The lighting system has a function to irradiate a predetermined irradiation area with visible light having a predetermined wavelength and a different wavelength configuration from the ambient light, A function to acquire a captured image from the camera that includes the illumination area, Based on the aforementioned captured image, a function is provided to identify the movement of the human body captured in the image, Make it run, The aforementioned processor, Based on the aforementioned wavelength, the region of the object in which the human body is captured is identified from the captured image. A program that further performs a function to identify the movement of the human body based on an image included in the aforementioned object region.

5. The processor, A valid region including the irradiation area is set in the aforementioned captured image, The program according to claim 4, further comprising the function of identifying the object region from an image included in the effective region.

Images