Gesture recognition device, gesture recognition system, gesture recognition method, and program

Abstract

To provide a gesture recognition device, a gesture recognition system, a gesture recognition method, and a program that can deal with people with hearing or speech disabilities and can be used by a user without confusion.SOLUTION: A smart controller 100 includes a gesture estimation unit 130 that estimates a user's gesture based on an image acquired by an imaging unit 110, a zone identification unit 140 that identifies a zone Z in which the gesture is imaged, where one or more zones Z are set in an imaging range of the imaging unit 110, a command generation unit 150 that generates a command based on the gesture and the zone Z in which the gesture is imaged, and a communication unit 160 that transmits the command to a smart device.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a gesture recognition device, a gesture recognition system, a gesture recognition method, and a program.

Background Art

[0002] Smart controllers that recognize user voice commands, such as smart speakers, and control other smart devices based on the recognition results have been developed. For example, Patent Document 1 describes a smart speaker that starts analyzing a voice command after detecting a wake word.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the case of a smart controller that uses voice recognition, a person with a hearing or speech impairment cannot operate the smart controller. Also, in the future, when many smart devices are placed in the home due to the spread of smart devices, when the smart controller controls individual smart devices, the user needs to identify the individual smart devices and give instructions to the smart controller. For example, if there are multiple lights in the living room, even if the user tells the smart speaker to "turn on the light", all the lights placed in the home will turn on. And when turning on an individual light with the smart speaker, for example, it becomes necessary to say "turn on the kitchen light". And when there are many lights in the home, it is expected that the user will confuse the names of the lights.

[0005] An object of the present disclosure is to provide a gesture recognition device, a gesture recognition system, a gesture recognition method, and a program that can accommodate people with hearing and speech impairments and can be used without confusing the user.

Means for Solving the Problems

[0006] The gesture recognition device according to the present disclosure includes a gesture estimation unit that estimates a user's gesture based on an image acquired by an imaging unit, one or more zones are set in the imaging range of the imaging unit, a zone identification unit that identifies the zone in which the gesture was imaged, a command generation unit that generates a command based on the zone in which the gesture was imaged and the gesture, and a communication unit that transmits the command to a smart device.

[0007] The gesture recognition system according to the present disclosure includes a gesture recognition device that recognizes a user's gesture and at least one smart device that can communicate with the gesture recognition device. The gesture recognition device includes a gesture estimation unit that estimates the user's gesture based on an image acquired by an imaging unit, one or more zones are set in the imaging range of the imaging unit, a zone identification unit that identifies the zone in which the gesture was imaged, a command generation unit that generates a command based on the zone in which the gesture was imaged and the gesture, and a communication unit that transmits the command to the smart device.

[0008] The gesture recognition method according to the present disclosure is a method in which a processor estimates a user's gesture based on an image acquired by an imaging unit, identifies the zone in which the gesture was imaged from one or more zones set in the imaging range of the imaging unit, generates a command based on the zone in which the gesture was imaged and the gesture, and transmits the command to a smart device.

[0009] The program according to the present disclosure causes a processor to execute a process of estimating a user's gesture based on an image acquired by an imaging unit, a process of specifying a zone in which the gesture is imaged from one or more zones set in the imaging range of the imaging unit, a process of generating a command based on the zone in which the gesture is imaged and the gesture, and a process of transmitting the command to a smart device.

Effect of the Invention

[0010] According to the present disclosure, it is possible to provide a gesture recognition device, a gesture recognition system, a gesture recognition method, and a program that can support people with hearing or speech impairments and can be used without confusing the user.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

MODE FOR CARRYING OUT THE INVENTION

[0012] Embodiment 1 Hereinafter, with reference to FIG. 1, an example of the configuration of the smart controller 100 according to Embodiment 1 will be described. The smart controller 100 functions as a gesture recognition device that estimates a user's gesture from an image acquired by an imaging unit 110 (described later) such as a CCD (Charge-Coupled Device) camera, and transmits a command pre-associated with the gesture to a smart device (not shown). In the following description, as an example of a gesture, a hand gesture (also referred to as a "hand pose") will be described, but the gesture is not limited to a hand gesture. Specifically, the smart controller 100 includes an imaging unit 110, an initial setting unit 120 as a zone setting unit, a gesture estimation unit 130, a zone specifying unit 140, a command generation unit 150, a communication unit 160, a display unit 170, a memory 180, and a processor 190. Note that the smart controller 100 may include a voice output unit (not shown) such as a speaker or earphone, and a voice input unit (not shown) such as a microphone.

[0013] The imaging unit 110 can be configured using a camera equipped with an imaging device such as a CCD image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The imaging unit 110 images a predetermined imaging range and acquires an image. For example, when the smart controller 100 is installed in the living room, the imaging unit 110 images the living room. Also, as shown in FIG. 2, as the imaging unit 110, instead of the camera built into the smart controller 100, or together with the camera built into the smart controller 100, an external camera 600 not built into the smart controller 100 may be used. The external camera 600 can communicate with the smart controller 100, and the image captured by the external camera 600 is transmitted to the smart controller 100. Also, the external camera 600 may be built into an external device such as another smart controller or a smart speaker. Also, the smart controller 100 does not have a built-in camera, and the smart controller 100 may receive the image captured by the external camera 600. Also, the external camera 600 may be installed at a location different from the smart controller 100. For example, the smart controller 100 is installed in the living room, and the external camera 600 may be installed in the bedroom, entrance, washroom, etc. Thereby, regardless of the installation location of the smart controller 100, the smart controller 100 can control the smart device based on the gesture performed by the user at the location where the external camera 600 is installed. FIG. 3 shows an example of an image acquired by the imaging unit 110.

[0014] As shown in FIG. 3, the initial setting unit 120 sets one or more zones Z1, Z2, Z3, ··· (hereinafter, simply referred to as "zone Z" when not particularly distinguished) in the imaging range of the imaging unit 110. Each zone Z is set in advance by the user, for example. Specifically, as shown in FIG. 4, the display unit 170 includes a touch panel, and the initial setting unit 120 displays an image acquired by the imaging unit 110 on the touch panel. Then, when the user performs a drag operation or the like on the image, one or more zones Z are set in the imaging range of the imaging unit 110. More specifically, when the zone Z is rectangular, the user performs a drag operation from the upper left point to the lower right point of the zone Z to set the zone Z. The setting of the zone Z includes setting the name of the zone Z, the image coordinates of the upper left point and the lower right point of the zone Z, and the like. Note that the number of zones Z set in the imaging range of the imaging unit 110 is not limited to the number shown in FIG. 3. Also, the sizes of the respective zones Z may be the same or different. Further, the shape of the zone Z is not limited to a rectangular shape, and may be other shapes such as a circular shape or an elliptical shape. Also, a part of one zone Z and a part of at least one other zone Z may overlap.

[0015] Note that the zone Z may be set in the imaging range of the imaging unit 110 by other methods. For example, the imaging unit 110 may image the imaging range for a predetermined period, and the initial setting unit 120 may create a heat map indicating the frequency of the user's presence within the imaging range, and set a zone Z of a predetermined size centered on a position where the frequency of the user's presence is higher than a predetermined frequency. Also, the initial setting unit 120 may perform object recognition on the image captured by the imaging unit 110, extract a predetermined piece of furniture such as a sofa or a predetermined electrical product, and set a predetermined range including the furniture or the electrical product as the zone Z. Further, the user may select a desired zone Z from the zones Z set by the initial setting unit 120 by these methods.

[0016] In addition, the initial setting unit 120 defines the definition of the table 181 stored in the memory 180. An example of the table 181 is shown in FIG. 5. As shown in FIG. 5, the table 181 is information in which the name 181A of the zone Z, the name 181B of the hand gesture, and the command information 181C are associated with each other. Specifically, the initial setting unit 120 causes the display unit 170 to display an operation screen for the user to define the table 181. Then, the user defines the name 181A of the zone Z, the name 181B of the hand gesture, and the command information 181C by performing operations on the operation screen. For example, on the operation screen, lists of candidates for the name 181A of the zone Z, the name 181B of the hand gesture, and the command information 181C may be displayed in a pull-down manner, and the definition may be performed by the user selecting a desired item from the candidate lists. In the example shown in FIG. 5, two zones Z, "sofa" and "other than sofa", are defined as the name 181A of the zone Z, but there may be one zone, and in that case, the name 181A of the zone is defined as, for example, "whole". Further, the name or identification information of the smart device to which the command defined in the command information 181C is transmitted may also be defined in the command information 181C. In addition, when a plurality of cameras, for example, a camera built in the smart controller 100 or an external camera 600 not built in the smart controller 100, are used as the imaging unit 110, the memory 180 may store a table 184 as shown in FIG. 6 instead of the table 181. As shown in FIG. 6, the table 184 is information in which an identification number 184A as identification information of the camera, a name 184B of the zone Z, a name 184C of the hand gesture, and command information 184D are associated with each other. For example, in the table 184 shown in FIG. 6, "0001" is defined as the identification number 184A of the camera built in the smart controller 100 installed in the living room, and "0002" is defined as the identification number 184A of the external camera 600 installed in the bedroom. Note that the identification information of the camera may be the name of the camera, the IP address (Internet Protocol Address), the user name of the RTSP (Real Time Streaming Protocol), the password of the RTSP, the URL (Uniform Resource Locator) of the RTSP, the model number, the serial number, etc. Thereby, the smart controller 100 can transmit a hand gesture imaged in the zone Z set in the imaging range of the external camera 600 installed at a location different from the smart controller 100 and a command based on the zone Z to the smart device.

[0017] Also, when a part of one zone Z overlaps with a part of at least one other zone Z, in Table 181 or Table 184, for all of the zones Z where the parts overlap, one type of command may be associated with one type of hand gesture. Specifically, when the initial setting unit 120 determines that different command information 181C or 184D has been selected for the name 181B or 184C of one type of gesture with respect to the names 181A or 184B of all of the zones Z where the parts overlap, as input by the user, the display unit 170 is caused to output an error. This can prevent confusion caused by defining different commands for one type of hand gesture in a plurality of overlapping zones Z.

[0018] The gesture estimation unit 130 estimates the user's hand gesture based on the image captured by the imaging unit 110. Specifically, the gesture estimation unit 130 estimates the hand gesture using a pre-trained hand gesture estimation model. The hand gesture estimation model estimates the hand gesture based on, for example, a skeletal shape formed from bones connecting key points, where the key points are the joints of the user's hand extracted from the image, as shown in FIGS. 7 to 9. More specifically, the hand gesture estimation model is, for example, a neural network such as a CNN (Convolutional Neural Network). In FIGS. 7 to 9, the key points are indicated by black circles and the bones are indicated by solid lines. The hand gesture shown in FIG. 7 is a gesture called "Calling", the hand gesture shown in FIG. 8 is a gesture called "OK", and the hand gesture shown in FIG. 9 is a gesture called "Palm Open". The types of hand gestures are not limited to the examples shown in FIGS. 7 to 9.

[0019] The training data used for machine learning of the hand gesture estimation model is, for example, an image of a hand gesture with the name of the hand gesture as the correct label prepared in advance. Note that the images of the hand gesture may include images of one hand gesture taken from different directions, or images of different forms of hand gestures recognized as one type of hand gesture with the same correct label attached. Alternatively, the training data (also referred to as "teacher data") used for machine learning of the hand gesture estimation model may be generated by the imaging unit 110 of the smart controller 100 installed at a location desired by the user capturing the user's hand gesture, and the user attaching the name of the hand gesture as the correct label to the image acquired by the imaging unit 110. Also, the user may select a desired one from the images with correct labels prepared in advance.

[0020] The zone specifying unit 140 specifies the zone Z in which the hand gesture is imaged by the imaging unit 110. Specifically, the zone specifying unit 140 specifies the zone Z in which the hand gesture is imaged based on which zone Z the image coordinates of the hand gesture estimated by the gesture estimating unit 130 are included in. Here, the image coordinates of the hand gesture are, for example, the image coordinates of the center of the image area occupied by the hand gesture in the image captured by the imaging unit 110. Also, the image coordinates of the hand gesture may be, for example, the upper left image coordinates and the lower right image coordinates of the bounding box set by the gesture estimating unit 130 for detecting the gesture.

[0021] Also, when a part of one zone Z overlaps with a part of at least one other zone Z, the zone specifying unit 140 calculates, for each zone Z, the area of the overlap between the bounding box set by the gesture estimating unit 130 for detecting a gesture and the plurality of zones Z, and may specify the zone Z with the largest such area as the zone Z in which the gesture was imaged. The area of the overlap between the bounding box and the zone Z may be represented by the ratio of the number of pixels in the overlapping portion between the bounding box and the zone Z to the number of pixels in the entire bounding box. For example, in the example shown in FIG. 10, in the imaging range of the imaging unit 110, three zones Z1, Z2, and Z3 are set, a part of zone Z1 overlaps with a part of zone Z2, and a part of zone Z2 overlaps with a part of zone Z3. Then, for example, when a "Palm Open" gesture surrounded by the bounding box B1 is estimated by the gesture estimating unit 130, the zone specifying unit 140 calculates the area of the overlap between the bounding box B1 and zone Z1 and the area of the overlap between the bounding box B1 and zone Z2. In the example shown in FIG. 10, the area of the overlap between the bounding box B1 and zone Z1 is larger than the area of the overlap between the bounding box B1 and zone Z2. Therefore, the zone specifying unit 140 specifies zone Z1 as the zone Z in which the gesture surrounded by the bounding box B1 was imaged. Also, in the example shown in FIG. 10, the bounding box B2 is completely included in the overlapping portion between zone Z2 and zone Z3, and the area of the overlap between the bounding box B2 and zone Z2 is the same as the area of the overlap between the bounding box B2 and zone Z3. To handle such cases, the initial setting unit 120 may determine the priorities of zones Z1 to Z3 in advance, and the zone specifying unit 140 may specify the zone Z with the highest priority among the plurality of zones Z overlapping with the bounding box as the zone Z in which the gesture surrounded by the bounding box B2 was imaged. Thereby, even when there are a plurality of zones Z with partial overlaps, one zone is always specified as the zone Z in which the gesture was imaged.Therefore, in this case, in Table 181 or Table 184, even if different commands are associated with one type of hand gesture in a plurality of zones Z where a part thereof overlaps, no confusion will occur. In other words, in a plurality of zones Z where a part thereof overlaps, even if the command associated with a certain hand gesture in one zone Z is different from the command associated with a certain hand gesture in another zone Z, no confusion will occur.

[0022] The command generation unit 150 generates a command based on the zone Z in which the hand gesture is imaged and the hand gesture. Specifically, the command generation unit 150 refers to Table 181 stored in the memory 180 and generates a command associated with the zone Z in which the hand gesture is imaged and the hand gesture. Alternatively, the command generation unit 150 refers to Table 184 stored in the memory 180 and generates a command associated with the identification number of the camera that imaged the hand gesture, the zone Z in which the hand gesture is imaged, and the hand gesture. The identification number of the camera that imaged the hand gesture and the like may be input to the gesture estimation unit 130 together with the image acquired by the camera. Then, the gesture estimation unit 130 may input the identification number of the camera and the like to the command generation unit 150 together with the estimation result.

[0023] In addition, when a predetermined hand gesture is estimated by the gesture estimation unit 130, the command generation unit 150 starts generating a command based on the hand gesture estimated by the gesture estimation unit 130 after the predetermined hand gesture and the zone specified by the zone specifying unit 140. Here, the predetermined hand gesture is, for example, "Calling" shown in FIG. 7. Thereby, it is possible to prevent the smart controller 100 from recognizing a hand gesture made by the user without intending to give an instruction to the smart controller 100 and accidentally controlling the smart device.

[0024] The communication unit 160 communicates between the smart controller 100 and other electrical products. Specifically, the communication unit 160 communicates with other smart devices through wireless communication such as Bluetooth Low Energy (registered trademark), Thread (registered trademark), etc. Also, the communication unit 160 communicates with a Thread Border Router, a Wi-Fi gateway, an Internet router, etc. through wireless communication such as Wi-Fi (registered trademark).

[0025] The display unit 170 is equipped with a touch panel and displays, for example, an image captured by the imaging unit 110 and an operation screen for the user to operate the smart controller 100. Also, the display unit 170 displays information regarding the smart devices controlled by the smart controller 100. Further, the display unit 170 may display other useful information such as the date, information regarding today's schedule, weather information, etc.

[0026] The memory 180 is composed of a combination of a volatile memory and a non-volatile memory. The memory 180 is used to store programs executed by the processor 190 and data used for various processes. The processor 190 may be, for example, an MPU (Micro Processor Unit) or a CPU (Central Processing Unit), etc. The processor may include a plurality of processors.

[0027] Also, the memory 180 stores the table 181 or the table 184. As shown in FIG. 5, the table 181 is information in which the name 181A of the zone Z, the name 181B of the hand gesture, and the command information 181C are associated. Also, the table 184 is information in which the identification number 184A as the identification information of the camera, the name 184B of the zone Z, the name 184C of the hand gesture, and the command information 184D are associated. Also, the table 181 or the table 184 is information defined in advance by the user.

[0028] Next, while referring to FIG. 11, the gesture recognition method according to Embodiment 1 will be described. First, the imaging unit 110 images a predetermined imaging range and acquires an image (step S101).

[0029] Next, the gesture estimation unit 130 determines whether a hand is detected from the image acquired in step S101 (step S102).

[0030] In step S102, if the gesture estimation unit 130 determines that no hand is detected from the image (step S102; No), the process returns to the process of step S101.

[0031] In step S102, if the gesture estimation unit 130 determines that a hand is detected from the image (step S102; Yes), the gesture estimation unit 130 estimates the hand gesture and determines whether the hand gesture is a predetermined hand gesture, for example, the "Calling" gesture (step S103).

[0032] In step S103, if the gesture estimation unit 130 determines that the hand gesture is not a predetermined hand gesture (step S103; No), the process returns to the process of step S101.

[0033] In step S103, if the gesture estimation unit 130 determines that the hand gesture is a predetermined hand gesture (step S103; Yes), the gesture estimation unit 130 estimates the next hand gesture (step S104).

[0034] Next, the zone identification unit 140 identifies the zone Z in which the hand gesture estimated in step S104 is imaged (step S105).

[0035] Next, the command generation unit 150 refers to Table 181 or Table 184 and generates a command associated with the hand gesture estimated in step S104 and the zone specified in step S105 (step S106).

[0036] Next, the communication unit 160 transmits the command generated in step S106 to an appropriate smart device (step S107), and ends this process.

[0037] Note that the smart controller 100 may check the operating state of the smart device before transmitting the command to the smart device in step S107.

[0038] In the smart controller 100 according to Embodiment 1, one or more zones Z are set in the imaging range of the imaging unit 110, a command is generated based on the zone Z in which the hand gesture is imaged and the hand gesture, and the command is transmitted to the smart device. Therefore, it is possible to cause the smart controller 100 to transmit a command to the smart device using a hand gesture, and even a person with a hearing or speech disorder can use the smart controller 100. In addition, since a command corresponding to a hand gesture can be generated for each zone Z, even if many smart devices are arranged in the house, it is possible to prevent the user from being confused about which smart device to specify. As a result, it is possible to provide a smart controller 100, a gesture recognition method, and a program that can support a person with a hearing or speech disorder and can be used without confusing the user.

[0039] Specifically, in a survey published by psychologist Wendy Wood et al. in 2006, 100 subjects aged 17 to 79 were asked to record their daily behaviors, and it was revealed that 47% (average value) of the subjects' behaviors were "habits" performed in the same place and at the same time. From this, it is considered that the will to perform a certain action occurs at a specific place and a specific time. Specifically, examples include "always sitting on the sofa when watching TV.", "Checking at the entrance whether the lights in all rooms are turned off when going out.", "Adjusting the air conditioner when working at the desk.", "Getting out of bed and opening the curtains when waking up in the morning.", etc. Therefore, the smart controller 100 according to Embodiment 1 can control the smart devices arranged in the home without confusion by associating a hand gesture: action, in which an operation (command) on the smart device is defined, with a zone Z: place.

[0040] Also, when a predetermined hand gesture is estimated by the gesture estimation unit 130, the generation of a command by the command generation unit 150 is started. Thereby, it is possible to prevent the smart controller 100 from recognizing a hand gesture made by the user unintentionally instructing the smart controller 100 and accidentally controlling the smart device.

[0041] Also, when a part of one zone Z overlaps with a part of at least one other zone Z, in the table 181 or the table 184, for all of the plurality of zones Z where the part overlaps, one type of command may be associated with one type of hand gesture. Thereby, it is possible to prevent confusion from occurring due to different commands being defined for one type of hand gesture in a plurality of zones Z where the part overlaps.

[0042] Also, when a part of one zone Z overlaps with a part of at least one other zone Z, the zone specifying unit 140 calculates, for each zone Z, the overlapping area between the bounding box set by the gesture estimation unit 130 for detecting a gesture and the plurality of zones Z. Then, the zone specifying unit 140 may specify the zone Z having the largest such area as the zone Z in which the gesture was imaged. In this case, in the table 181 or the table 184, even if different commands are associated with a plurality of zones Z in which the parts overlap, no confusion will occur. Therefore, it becomes possible to associate the same hand gesture with different commands, and it is possible to prevent an increase in the number of types of hand gestures to be estimated by the gesture estimation unit 130.

[0043] Also, the gesture estimation unit 130 estimates a hand gesture using a machine-learned hand gesture estimation model. Specifically, the hand gesture estimation model extracts the joints of the user's hand from the image as key points, and estimates the hand gesture based on the skeletal shape formed by the bones connecting the key points. Thereby, the hand gesture imaged by the imaging unit 110 can be accurately estimated.

[0044] In addition, when a large number of smart devices are to be placed in a home, the types of commands for controlling the smart devices also increase, and it is necessary to increase the types of hand gestures to be associated with the commands. And, to increase the types of hand gestures, it becomes necessary to include, as hand gestures, for example, hand gestures including complex movements or hand gestures by multiple hands. However, in the smart controller 100 according to Embodiment 1, since hand gestures and commands are associated for each of a plurality of zones Z, it is possible to associate different commands with the same hand gesture in different zones Z. In other words, it is possible to prevent an increase in the types of hand gestures to be estimated by the gesture estimation unit 130. Therefore, there is no need for the gesture estimation unit 130 to estimate hand gestures including complex movements or hand gestures by multiple hands. Therefore, the hand gesture estimation model used by the gesture estimation unit 130 can be a lightweight model with a small amount of computation capable of recognizing hand gestures from real-time images captured by the imaging unit 110. Thereby, the operation of the smart device by hand gestures via the smart controller 100 can be realized in real time.

[0045] In addition, the smart controller 100 receives an image acquired by a camera built in the smart controller 100 or an external camera 600, and in the table 184, a zone Z set in the imaging range of the camera built in the smart controller 100 or the external camera 600, a hand gesture, a command, and further identification information of the camera built in the smart controller 100 or the external camera 600 are associated with each other. Thereby, the smart controller 100 can transmit to the smart device a hand gesture imaged in a zone Z set in the imaging range of the external camera 600 installed at a location different from the smart controller 100 and a command based on the zone Z.

[0046] In addition, the display unit 170 includes a touch panel. On the image acquired by the imaging unit 110 and displayed on the touch panel, one or more zones Z are set in the imaging range of the imaging unit 110 when the user performs a drag operation or the like. As a result, the user can set the zone Z in a simple manner.

[0047] Embodiment 2 Next, with reference to FIG. 12, an example of the configuration of the smart home system 200 according to Embodiment 2 as a gesture recognition system will be described. The smart home system 200 includes smart controllers 300A, 300B,... (hereinafter, simply referred to as "smart controller 300" when not particularly distinguished) installed at a plurality of locations in one indoor area, and smart devices such as lighting 400A, TV 400B, curtain 400C, lighting 400D, curtain 400E, window 400F,... (hereinafter, simply referred to as "smart device 400" when not particularly distinguished) that can communicate with the smart controller 300. The smart controller 300 and the smart device 400 can communicate with each other via a network N. In addition, the smart controller 300A and the smart controller 300B may be able to communicate with each other via the network N. The network N includes, for example, wireless communications such as Bluetooth Low Energy, Thread, and Wi-Fi.

[0048] In the example shown in FIG. 12, the smart controller 300A, the lighting 400A, the TV 400B, and the curtain 400C are arranged in the living room, and the smart controller 300B, the lighting 400D, the curtain 400E, and the window 400F are arranged in the bedroom. Further, the smart controller 300 is different from the smart controller 100 according to Embodiment 1 in that the memory 180 stores the table 182 instead of the table 181 or the table 184. Further, the smart controller 300 stores in advance in the memory 180 the name of its installation location, for example, "living room", "bedroom", etc. Among the configurations of the smart controller 300, the same configurations as those of the smart controller 100 are denoted by the same reference numerals, and the description thereof is omitted.

[0049] FIG. 13 shows an example of the table 182. As shown in FIG. 13, the table 182 is information in which the name 182B of the zone Z, the name 182C of the hand gesture, the command information 182D, and further the installation location 182A of the smart controller 300 are associated. Further, in the table 182, as the command information 182D, the command transmitted to the smart device 400 and the information for identifying the smart device 400 to which the command is transmitted may be defined. Here, the information for identifying the smart device 400 is, for example, the name or identification information of the smart device 400. Further, the table 182 is information defined in advance by the user by the same method as the method described in Embodiment 1.

[0050] Then, the command generation unit 150 generates a command based on the installation location of the smart controller 300, the zone Z where the hand gesture is imaged, and the hand gesture. Specifically, the command generation unit 150 refers to the table 182 stored in the memory 180 and generates a command associated with the installation location of the smart controller 300, the zone Z where the hand gesture is imaged, and the hand gesture.

[0051] In the smart home system 200 according to Embodiment 2, in Table 182, in addition to the name 182B of zone Z, the name 182C of the hand gesture, and the command information 182D, the installation location 182A of the smart controller 300 is further associated. Also, in Table 182, as the command information 182D, together with the command transmitted to the smart device 400, the name and identification information of the smart device 400 to which the command is transmitted can also be defined. As a result, the smart controller 300A installed in the living room can control the lighting 400A, TV 400B, and curtain 400C arranged in the living room, and the smart controller 300B installed in the bedroom can control the lighting 400D, curtain 400E, and window 400F arranged in the bedroom. Also, for example, when a hand gesture is made to the smart controller 300A installed in the living room, the smart controller 300A can control at least a part of the smart devices 400 arranged in the bedroom in addition to the smart devices 400 arranged in the living room. For example, when a hand gesture for controlling the curtain is made to the smart controller 300A installed in the living room, the smart controller 300A may control both the curtain in the living room and the curtain in the bedroom. This is possible by defining the command "open the curtain" together with the names and identification information of the curtains in the living room and the bedroom as the command information 182D associated with the installation location 182A "living room", the zone name 182B "other than sofa", and the gesture name 182C "Palm open" in Table 182. On the other hand, when a hand gesture is made to the smart controller 300B installed in the bedroom, the smart controller 300B can also control only the smart devices 400 arranged in the bedroom. That is, the smart controller 300A installed in the living room can seamlessly control the smart devices 400 arranged throughout the house, and the smart controller 300B installed in the bedroom can locally control the smart devices 400 arranged in the bedroom. Also, when a hand gesture is made to the smart controller 300A installed in the living room, the smart controller 300A can control at least a part of the smart devices 400 arranged in the bedroom in addition to the smart devices 400 arranged in the living room. Also, when a hand gesture is made to the smart controller 300B installed in the bedroom, the smart controller 300B can control at least a part of the smart devices 400 arranged in the living room in addition to the smart devices 400 arranged in the bedroom.

[0052] Also, in Table 182, hand gestures are defined for each installation location 182A of the smart controller 300. Therefore, when the installation location of the smart controller 300 is different, different commands can be defined for the same type of hand gesture. This prevents the number of types of hand gestures to be recognized in the smart controller 300 from becoming too large, and can speed up the hand gesture estimation process in the gesture estimation unit 130.

[0053] Also, in Table 182, the hand gestures associated with the command information 182D in which the same command for the same type of smart device 400 is defined may be the same type of hand gesture. Specifically, in Table 182, for smart devices 400 arranged in any room, for example, lighting 400A and lighting 400D, and curtain 400C and curtain 400E, the same command or a similar command is associated with the same type of hand gesture. This enables the user to easily remember the hand gestures and perform the hand gestures without being conscious. Specifically, when the user defines Table 182, the initial setting unit 120 proposes to define the hand gestures for the same command for the same type of smart device to be of the same type. For example, when the user selects different types of hand gestures for the same command for the same type of smart device when defining Table 182, the initial setting unit 120 causes the display unit 170 to display a message such as "It is recommended to define the same type of hand gesture for the same command for the same type of smart device."

[0054] Embodiment 3 Next, with reference to FIG. 14, an example of the configuration of the smart home system 200A according to Embodiment 3 as a gesture recognition system will be described. The smart home system 200A includes a smart mirror 300C as a gesture recognition device, a smart speaker 500, and smart devices such as lighting 400A, TV 400B, curtain 400C, lighting 400D, and curtain 400E that can communicate with the smart mirror 300C and the smart speaker 500. The smart mirror 300C, the smart speaker 500, and the smart device 400 can communicate with each other via the network N.

[0055] In the example shown in FIG. 14, the lighting 400A, TV 400B, and curtain 400C are arranged in the living room, and the lighting 400D and curtain 400E are arranged in the bedroom. Also, the smart mirror 300C and the smart speaker 500 may be arranged anywhere indoors. In the following description, it is assumed that the smart mirror 300C is arranged in the living room. Embodiment 3 can assume a case where, for example, in a system that already includes the smart speaker 500 indoors, the smart mirror 300C is incorporated into the system. Then, both the smart mirror 300C and the smart speaker 500 control the lighting 400A, TV 400B, curtain 400C, lighting 400D, and curtain 400E. Further, the smart mirror 300C is different from the smart controller 100 according to Embodiment 1 in that the memory 180 stores the table 183 instead of the table 181 or the table 184. Also, the smart mirror 300C further includes a mirror on the display unit 170. Among the configurations of the smart mirror 300C, the same configurations as those of the smart controller 100 are denoted by the same reference numerals, and the description thereof is omitted.

[0056] FIG. 15 shows an example of a table 183. As shown in FIG. 15, the table 183 is information in which the name 183A of the zone Z, the name 183B of the hand gesture, and the command information 183C are associated with each other. Specifically, in the table 183, as the zone where the hand gesture for operating the smart device 400 arranged in the bedroom is performed, "entire" is defined. Here, "entire" means that the zone defined as "entire" is the entire imaging range of the imaging unit 110 of the smart mirror 300C. Also, in the table 183, the hand gestures associated with the command information 183C in which commands for operating the smart devices 400 arranged at different locations indoors are defined are different. Specifically, in the table 183, the hand gesture associated with the command information 183C for operating the smart device 400 in the living room is different from the hand gesture associated with the command information 183C for operating the smart device 400 in the bedroom. In other words, in the table 183, for example, different types of hand gestures are associated with the command information 183C for opening and closing the curtain 400C in the living room and the command information 183C for opening and closing the curtain 400E in the bedroom. Thereby, when a hand gesture for operating the smart device 400 arranged in the bedroom is performed in the living room, the command generation unit 150 can generate a command for operating the smart device 400 arranged in the bedroom regardless of whether the zone where the hand gesture is imaged is "sofa" or "other than sofa". Also, the table 183 is information defined in advance by the user in the same manner as the method described in Embodiment 1.

[0057] In the smart home system 200A according to Embodiment 3, in Table 183, the hand gestures associated with the commands for operating the smart device 400 in the living room are different from the hand gestures associated with the commands for operating the smart device 400 in the bedroom. Thereby, even if the smart mirror 300C arranged in the living room is the same type of smart device 400 as the smart device 400 arranged in the living room, it can be controlled without confusing the smart device 400 arranged in the bedroom. That is, a person with an auditory or speech disorder can control the smart device 400 arranged in the bedroom using the smart mirror 300C arranged in the living room as a UI (User Interface).

[0058] Note that the present invention is not limited to the above-described embodiments, and can be appropriately modified without departing from the gist thereof. For example, in Tables 181, 182, and 184, the zone Z, the hand gesture, and the commands for the smart controllers 100 and 300 including the tables 181, 182, and 184 may be associated with each other. Similarly, in Table 183, the zone Z, the hand gesture, and the commands for the smart mirror 300C including the table 183 may be associated with each other.

[0059] In the above-described embodiments, the present disclosure has been described as a hardware configuration, but the present disclosure is not limited thereto. The present disclosure can also be realized by causing the processor 190 to execute a computer program for the processing procedures described in the flowchart of FIG. 11 and the processing procedures described in other embodiments.

[0060] In the above example, the program can be stored using various types of non-transitory computer readable media and supplied to a computer. Non-transitory computer readable media include various types of tangible storage media. Non-transitory computer readable media include, for example, magnetic recording media, magneto-optical recording media, CD-ROM (Read Only Memory), CD-R, CD-R / W, and semiconductor memories. Semiconductor memories include, for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory), etc. Also, the program may be supplied to the computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer readable media can supply the program to the computer via wired communication paths such as electric wires and optical fibers, or wireless communication paths.

[0061] As described above, the present disclosure has been described with reference to the embodiments, but the present disclosure is not limited to the above-described embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. And each embodiment can be combined with other embodiments as appropriate.

[0062] The drawings are merely illustrative for explaining one or more embodiments. Each drawing may be associated with not only one specific embodiment but also one or more other embodiments. As can be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with the features or steps shown in one or more other drawings to create, for example, embodiments that are not explicitly illustrated or described. Not all of the features or steps shown in any one drawing for explaining exemplary embodiments are necessarily essential, and some features or steps may be omitted. The order of the steps described in any drawing may be changed as appropriate.

Description of Reference Numerals

[0063] 100, 300A, 300B, ··· Smart Controller (Gesture Recognition Device) 110 Imaging Unit 120 Initial Setting Unit (Zone Setting Unit) 130 Gesture Estimation Unit 140 Zone Identification Unit 150 Command Generation Unit 160 Communication Unit 170 Display Unit 180 Memory 181, 182, 183, 184 Tables 190 Processor 200, 200A Smart Home System (Gesture Recognition System) 300C Smart Mirror (Gesture Recognition Device) 400A, 400D Lighting (Smart Device) 400B TV (Smart Device) 400C, 400E Curtain (Smart Device) 400F Window (Smart Device) 500 Smart Speaker

Claims

1. a gesture estimation unit that estimates a gesture of a user based on an image acquired by the imaging unit; a zone specifying unit that specifies a zone in which an image of the gesture is captured, the zone specifying unit being configured to specify one or more zones in an image capturing range of the image capturing unit; a command generating unit that generates a command based on the gesture and a zone in which the gesture is imaged; A communication unit that transmits the command to a smart device; A gesture recognition device comprising:

2. when a predetermined gesture is estimated by the gesture estimation unit, the command generation unit starts generating a command based on the gesture estimated by the gesture estimation unit after the predetermined gesture and the zone in which the gesture is captured and identified by the zone identification unit. The gesture recognition device according to claim 1 .

3. the command generation unit generates a command based on a table in which a zone set in an imaging range of the imaging unit is associated with a gesture and command information; The gesture recognition device according to claim 1 .

4. The gesture recognition devices are installed at multiple locations in a single room, The table is information that associates the zone, the gesture, the command information, and further the installation location of the gesture recognition device. The gesture recognition device according to claim 3 .

5. In the table, a command to be transmitted to the smart device and information for identifying the smart device to which the command is transmitted are defined as the command information. The gesture recognition device according to claim 4 .

6. In the table, the gestures associated with the command information in which the same command for the smart device of the same type is defined are gestures of the same type. The gesture recognition device according to claim 4 .

7. In the table, the gestures associated with the command information, in which commands for operating the smart devices placed at different locations indoors are defined, are different. The gesture recognition device according to claim 4 .

8. When a plurality of zones are set in an imaging range of the imaging unit, and a part of one zone overlaps with a part of at least one other zone, in the table, one type of gesture is associated with one type of command in all of the plurality of zones that overlap. The gesture recognition device according to claim 3 .

9. When a plurality of zones are set in an imaging range of the imaging unit, and a part of one zone overlaps with a part of at least one other zone, the zone identification unit calculates an overlap area between a bounding box set by the gesture estimation unit for detecting the gesture and the plurality of zones for each zone, and identifies the zone having the largest overlap area as the zone in which the gesture was captured. The gesture recognition device according to claim 3 .

10. The gesture recognition device receives images captured by the plurality of imaging units, The table is information that associates zones set in an imaging range of the imaging unit, gestures, commands, and further identification information of the imaging unit. The gesture recognition device according to claim 3 .

11. the gesture is a hand gesture of the user; The gesture estimation unit estimates a hand gesture using a machine-learned hand gesture estimation model; and The hand gesture estimation model extracts joints of the user's hand as key points from the image, and estimates a hand gesture based on a skeletal shape formed by bones connecting the key points. The gesture recognition device according to claim 1 .

12. The gesture recognition device includes a touch panel. one or more zones are set in an imaging range of the imaging unit based on an operation of the user on the image acquired by the imaging unit and displayed on the touch panel; The gesture recognition device according to claim 1 .

13. The imaging unit further includes a zone setting unit that creates a heat map indicating the frequency with which the user was present within the imaging range based on an image obtained by imaging the imaging range for a predetermined period of time, and sets a range of a predetermined size as a zone centered on a position where the frequency with which the user was present is higher than the predetermined frequency. The gesture recognition device according to claim 1 .

14. A gesture recognition device that recognizes a gesture of a user and at least one smart device that can communicate with the gesture recognition device, The gesture recognition device comprises: a gesture estimation unit that estimates a gesture of the user based on an image acquired by an imaging unit; a zone specifying unit that specifies a zone in which an image of the gesture is captured, the zone specifying unit being configured to specify one or more zones in an image capturing range of the image capturing unit; a command generating unit that generates a command based on the gesture and a zone in which the gesture is imaged; a communication unit that transmits the command to the smart device; Equipped with Gesture recognition system.

15. The processor: Estimating a gesture of a user based on an image acquired by the imaging unit; Identifying a zone in which the gesture is captured from one or more zones set in an imaging range of the imaging unit; generating a command based on the gesture and a zone in which the gesture was imaged; Sending the command to a smart device; Gesture recognition methods.

16. The processor: A process of estimating a gesture of a user based on an image acquired by an imaging unit; A process of identifying a zone in which the gesture is captured from one or more zones set in an image capturing range of the image capturing unit; generating a command based on the gesture and the zone in which the gesture was imaged; sending the command to a smart device; A program to execute.

Images