Gesture recognition device, gesture recognition system, gesture recognition method, and program

Abstract

This invention provides a gesture recognition device, a gesture recognition system, a gesture recognition method, and a program which can even handle a person who has impaired hearing or speech and which can be used without a user becoming confused. A smart controller (100) comprises: a gesture estimation unit (130) that estimates a gesture of a user on the basis of an image acquired by an imaging unit (110); a zone identification unit (140) that, with one or more zones Z being set in the imaging range of the imaging unit (110), identifies a zone Z in which the gesture was captured; a command generation unit (150) that generates a command on the basis of the gesture and the zone Z in which the gesture was captured; and a communication unit (160) that transmits the command to a smart device.

Description

Gesture recognition device, gesture recognition system, gesture recognition method, and program

[0001] The present invention relates to a gesture recognition device, a gesture recognition system, a gesture recognition method, and a program.

[0002] Smart controllers, such as smart speakers, have been developed that recognize user voice commands and control other smart devices based on the recognition results. For example, Patent Literature 1 describes a smart speaker that starts analyzing voice commands after detecting a wake word.

[0003] US Patent Application Publication No. 2019 / 156818

[0004] However, smart controllers that use voice recognition cannot be operated by people with hearing or speech impairments. Furthermore, as smart devices become more widespread in the future, and more smart devices are placed throughout the home, users will need to identify each smart device and issue commands to the smart controller when controlling each smart device. For example, if there are multiple lights in the living room, even if a user says "Turn on the lights" to a smart speaker, all the lights in the house will turn on. Furthermore, to turn on an individual light, the user will need to say, for example, "Turn on the kitchen light." Furthermore, with many lights throughout the home, users are likely to get confused about the names of the lights.

[0005] An object of the present disclosure is to provide a gesture recognition device, a gesture recognition system, a gesture recognition method, and a program that can be used by people with hearing or speech disabilities and that can be used without causing confusion to the user.

[0006] The gesture recognition device according to the present disclosure includes a gesture estimation unit that estimates a user's gesture based on an image acquired by an imaging unit, a zone identification unit that identifies a zone in which the gesture is imaged, with one or more zones set in the imaging range of the imaging unit, a command generation unit that generates a command based on the zone in which the gesture is imaged and the gesture, and a communication unit that transmits the command to a smart device.

[0007] A gesture recognition system according to the present disclosure includes a gesture recognition device that recognizes a user's gesture, and at least one smart device that can communicate with the gesture recognition device. The gesture recognition device includes a gesture estimation unit that estimates the user's gesture based on an image acquired by an imaging unit, one or more zones set in an imaging range of the imaging unit, a zone identification unit that identifies a zone in which the gesture is imaged, a command generation unit that generates a command based on the zone in which the gesture is imaged and the gesture, and a communication unit that transmits the command to the smart device.

[0008] The gesture recognition method according to the present disclosure is a method in which a processor estimates a user's gesture based on an image acquired by an imaging unit, identifies a zone in which the gesture is imaged from one or more zones set in the imaging range of the imaging unit, generates a command based on the zone in which the gesture is imaged and the gesture, and transmits the command to a smart device.

[0009] The program according to the present disclosure causes a processor to execute the following processes: estimating a user's gesture based on an image acquired by an imaging unit; identifying a zone in which the gesture is imaged from one or more zones set in the imaging range of the imaging unit; generating a command based on the zone in which the gesture is imaged and the gesture; and transmitting the command to a smart device.

[0010] According to the present disclosure, it is possible to provide a gesture recognition device, a gesture recognition system, a gesture recognition method, and a program that can be used by people with hearing or speech impairments and that can be used without causing confusion to the user.

[0011] FIG. 1 is a block diagram showing an example of the configuration of a smart controller according to embodiment 1. FIG. 2 is a diagram showing an example of the relationship between the smart controller according to embodiment 1 and an external camera not built in the smart controller. FIG. 3 is a diagram showing an example of an image acquired by an imaging unit according to embodiment 1. FIG. 4 is a diagram showing an example of a display unit of the smart controller according to embodiment 1. FIG. 5 is a diagram showing an example of a table according to embodiment 1. FIG. 6 is a diagram showing another example of a table according to embodiment 1. FIG. 7 is a diagram explaining an example of estimation processing by a gesture estimation unit according to embodiment 1. FIG. 8 is a diagram explaining an example of estimation processing by a gesture estimation unit according to embodiment 1. FIG. 9 is a diagram explaining an example of estimation processing by a gesture estimation unit according to embodiment 1. FIG. 10 is a diagram explaining an example of zone identification processing by a zone identification unit according to embodiment 1. FIG. 11 is a flowchart explaining an example of a gesture recognition method according to embodiment 1. FIG. 12 is a block diagram showing an example of a gesture recognition system according to embodiment 2. FIG. 13 is a diagram showing an example of a table according to embodiment 2. FIG. 14 is a block diagram showing an example of a gesture recognition system according to embodiment 3. FIG. 15 is a diagram showing an example of a table according to embodiment 3.

[0012] Embodiment 1 An example of the configuration of a smart controller 100 according to embodiment 1 will be described below with reference to FIG. 1 . The smart controller 100 functions as a gesture recognition device that estimates a user's gesture from an image captured by an imaging unit 110 (described below) such as a CCD (Charge-Coupled Device) camera and transmits a command pre-associated with the gesture to a smart device (not shown). In the following description, a hand gesture (also referred to as a "hand pose") will be used as an example of a gesture; however, the gesture is not limited to a hand gesture. Specifically, the smart controller 100 includes an imaging unit 110, an initial setting unit 120 as a zone setting unit, a gesture estimation unit 130, a zone identification unit 140, a command generation unit 150, a communication unit 160, a display unit 170, a memory 180, and a processor 190. The smart controller 100 may also include an audio output unit (not shown) such as a speaker or earphones, and an audio input unit (not shown) such as a microphone.

[0013] The imaging unit 110 can be configured using a camera equipped with an imaging element such as a CCD image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The imaging unit 110 captures an image of a predetermined imaging range. For example, when the smart controller 100 is installed in a living room, the imaging unit 110 captures an image of the living room. As shown in FIG. 2 , an external camera 600 not built into the smart controller 100 may be used as the imaging unit 110 instead of or in addition to the camera built into the smart controller 100. The external camera 600 can communicate with the smart controller 100, and images captured by the external camera 600 are transmitted to the smart controller 100. The external camera 600 may also be built into an external device such as another smart controller or smart speaker. Alternatively, the smart controller 100 may not have a built-in camera, and the smart controller 100 may receive images captured by the external camera 600. The external camera 600 may be installed in a location different from the smart controller 100. For example, the smart controller 100 may be installed in a living room, and the external camera 600 may be installed in a bedroom, an entrance, a bathroom, etc. This allows the smart controller 100 to control the smart device based on gestures made by the user at the location where the external camera 600 is installed, regardless of the location where the smart controller 100 is installed. Figure 3 shows an example of an image captured by the imaging unit 110.

[0014] As shown in FIG. 3 , the initial setting unit 120 sets one or more zones Z1, Z2, Z3, ... (hereinafter, simply referred to as "zones Z" unless otherwise specified) in the imaging range of the imaging unit 110. Each zone Z is set in advance by, for example, a user. Specifically, as shown in FIG. 4 , the display unit 170 includes a touch panel, and the initial setting unit 120 displays an image acquired by the imaging unit 110 on the touch panel. The user then performs a drag operation or the like on the image to set one or more zones Z in the imaging range of the imaging unit 110. More specifically, if the zone Z is rectangular, the user sets the zone Z by performing a drag operation from the upper left point to the lower right point of the zone Z. Setting the zone Z includes setting the name of the zone Z and setting the image coordinates of the upper left point and the lower right point of the zone Z. Note that the number of zones Z set in the imaging range of the imaging unit 110 is not limited to the number shown in FIG. 3 . The sizes of the zones Z may be the same or different. Furthermore, the shape of the zone Z is not limited to a rectangular shape, and may be other shapes such as a circle, an ellipse, etc. Furthermore, a part of one zone Z may overlap a part of at least one other zone Z.

[0015] Note that the setting of the zone Z in the imaging range of the imaging unit 110 may be performed by other methods. For example, the imaging unit 110 may capture an image of the imaging range for a predetermined period, and the initial setting unit 120 may create a heat map indicating the frequency with which the user has been present within the imaging range, and set a zone Z of a predetermined size centered on a position where the frequency with which the user has been present is higher than a predetermined frequency. Alternatively, the initial setting unit 120 may perform object recognition from the image captured by the imaging unit 110 to extract a predetermined piece of furniture such as a sofa or a predetermined electrical appliance, and set a predetermined range including the furniture or the electrical appliance as the zone Z. Alternatively, the user may select a desired zone Z from the zones Z set by the initial setting unit 120 using these methods.

[0016] The initial setting unit 120 also defines a table 181 to be stored in the memory 180. An example of the table 181 is shown in FIG. 5 . As shown in FIG. 5 , the table 181 is information in which a zone Z name 181A, a hand gesture name 181B, and command information 181C are associated with each other. Specifically, the initial setting unit 120 causes the display unit 170 to display an operation screen for the user to define the table 181. The user then performs operations on the operation screen to define the zone Z name 181A, the hand gesture name 181B, and the command information 181C. For example, the operation screen may display a list of candidates for the zone Z name 181A, the hand gesture name 181B, and the command information 181C in a pull-down menu, and the user may select a desired one from the list of candidates to define the table 181. In the example shown in FIG. 5 , two zone Z names 181A, "sofa" and "non-sofa," are defined. However, there may be only one zone. In that case, the zone name 181A may be defined as, for example, "all." The command information 181C may also define the name and identification information of a smart device to which the command defined in the command information 181C is transmitted. When multiple cameras, such as a camera built into the smart controller 100 and an external camera 600 not built into the smart controller 100, are used as the imaging unit 110, the memory 180 may store a table 184 as shown in FIG. 6 instead of the table 181. As shown in FIG. 6 , the table 184 is information in which an identification number 184A as identification information of the camera, a zone Z name 184B, a hand gesture name 184C, and command information 184D are associated with each other. For example, in the table 184 shown in FIG. 6, the identification number 184A of the camera built into the smart controller 100 installed in the living room is defined as "0001," and the identification number 184A of the external camera 600 installed in the bedroom is defined as "0002."The camera identification information may be the camera name, IP address (Internet Protocol Address), RTSP (Real Time Streaming Protocol) username, RTSP password, RTSP URL (Uniform Resource Locator), model no., serial no., etc. This enables the smart controller 100 to transmit to the smart device a hand gesture captured in zone Z set in the imaging range of the external camera 600 installed at a location separate from the smart controller 100, and a command based on the zone Z.

[0017] Furthermore, when a portion of one zone Z overlaps with a portion of at least one other zone Z, one type of command may be associated with one type of hand gesture in all of the multiple zones Z that partially overlap in table 181 or table 184. Specifically, when the user selects different command information 181C or 184D for one type of gesture name 181B or 184C for all names 181A or 184B of the multiple zones Z that partially overlap, initial setting unit 120 causes display unit 170 to output an error. This makes it possible to prevent confusion caused by different commands being defined for one type of hand gesture in the multiple overlapping zones Z.

[0018] The gesture estimation unit 130 estimates a user's hand gesture based on the image captured by the imaging unit 110. Specifically, the gesture estimation unit 130 estimates the hand gesture using a machine-learned hand gesture estimation model. The hand gesture estimation model extracts the user's hand joints as key points from the image and estimates the hand gesture based on a skeletal shape formed by bones connecting the key points, as shown in, for example, FIGS. 7 to 9 . More specifically, the hand gesture estimation model is a neural network such as a CNN (convolutional neural network). In FIGS. 7 to 9 , key points are indicated by black circles and bones are indicated by solid lines. The hand gesture shown in FIG. 7 is a gesture for "calling," the hand gesture shown in FIG. 8 is a gesture for "OK," and the hand gesture shown in FIG. 9 is a gesture for "palm open." The types of hand gestures are not limited to the examples shown in FIGS.

[0019] The learning data used for machine learning of the hand gesture estimation model are, for example, images of hand gestures prepared in advance and labeled with the names of the hand gestures as correct labels. The hand gesture images may include images of a single hand gesture captured from different directions, or images of different hand gestures recognized as a single type of hand gesture, each labeled with the same correct label. Alternatively, the learning data (also referred to as "teacher data") used for machine learning of the hand gesture estimation model may be generated by capturing images of the user's hand gestures using the imaging unit 110 of the smart controller 100 installed at a location desired by the user, and then assigning the names of the hand gestures as correct labels to the images captured by the imaging unit 110. Alternatively, the user may select a desired image from among images prepared in advance and labeled with correct labels.

[0020] The zone identification unit 140 identifies the zone Z in which the hand gesture is imaged by the imaging unit 110. Specifically, the zone identification unit 140 identifies the zone Z in which the hand gesture is imaged based on which zone Z the image coordinates of the hand gesture estimated by the gesture estimation unit 130 are included. Here, the image coordinates of the hand gesture are, for example, the image coordinates of the center of the image area occupied by the hand gesture in the image captured by the imaging unit 110. Furthermore, the image coordinates of the hand gesture may be, for example, the image coordinates of the upper left and lower right of a bounding box set by the gesture estimation unit 130 to detect the gesture.

[0021] Furthermore, when a portion of one zone Z overlaps with a portion of at least one other zone Z, the zone identification unit 140 may calculate the area of ​​overlap between a bounding box set by the gesture estimation unit 130 to detect a gesture and the multiple zones Z, for each zone Z, and identify the zone Z with the largest overlap area as the zone Z in which the gesture was captured. The overlap area between a bounding box and a zone Z may be expressed as a ratio of the number of pixels in the overlapping portion between the bounding box and the zone Z to the total number of pixels in the bounding box. For example, in the example shown in FIG. 10 , three zones Z1, Z2, and Z3 are set in the imaging range of the imaging unit 110, and a portion of zone Z1 overlaps with a portion of zone Z2, and a portion of zone Z2 overlaps with a portion of zone Z3. For example, when the gesture estimation unit 130 estimates a "Palm Open" gesture surrounded by a bounding box B1, the zone identification unit 140 calculates the overlap area between the bounding box B1 and zone Z1 and the overlap area between the bounding box B1 and zone Z2. In the example shown in FIG. 10 , the overlap area between the bounding box B1 and zone Z1 is larger than the overlap area between the bounding box B1 and zone Z2. Therefore, the zone identification unit 140 identifies zone Z1 as the zone Z in which the gesture surrounded by the bounding box B1 was captured. In addition, in the example shown in FIG. 10 , the bounding box B2 is completely included in the overlap area between zone Z2 and zone Z3, and the overlap area between the bounding box B2 and zone Z2 is the same as the overlap area between the bounding box B2 and zone Z3. To deal with such cases, the initial setting unit 120 may determine the priorities of the zones Z1 to Z3 in advance, and the zone identification unit 140 may identify the zone Z with the highest priority among the multiple zones Z that overlap with the bounding box B2 as the zone Z in which the gesture surrounded by the bounding box B2 was captured. In this way, even if there are multiple zones Z that partially overlap, one zone is always identified as the zone Z in which the gesture was captured.Therefore, in this case, no confusion occurs even if different commands are associated with one type of hand gesture in the multiple zones Z that partially overlap in table 181 or table 184. In other words, no confusion occurs even if a command associated with a certain hand gesture in one zone Z is different from a command associated with a certain hand gesture in another zone Z in the multiple zones Z that partially overlap.

[0022] The command generation unit 150 generates a command based on the hand gesture and the zone Z in which the hand gesture is captured. Specifically, the command generation unit 150 references a table 181 stored in the memory 180 and generates a command associated with the hand gesture and the zone Z in which the hand gesture is captured. Alternatively, the command generation unit 150 references a table 184 stored in the memory 180 and generates a command associated with the identification number of the camera that captured the hand gesture, the zone Z in which the hand gesture is captured, and the hand gesture. The identification number of the camera that captured the hand gesture and the like may be input to the gesture estimation unit 130 along with the image acquired by the camera. The gesture estimation unit 130 may then input the camera's identification number and the like along with the estimation result to the command generation unit 150.

[0023] Furthermore, when a predetermined hand gesture is estimated by the gesture estimation unit 130, the command generation unit 150 starts generating a command based on the hand gesture estimated by the gesture estimation unit 130 after the predetermined hand gesture and the zone identified by the zone identification unit 140. Here, the predetermined hand gesture is, for example, "calling" shown in Fig. 7. This prevents the smart controller 100 from recognizing a hand gesture made by the user unintentionally as an instruction to the smart controller 100 and erroneously controlling the smart device.

[0024] The communication unit 160 communicates between the smart controller 100 and other electrical appliances. Specifically, the communication unit 160 communicates with other smart devices via wireless communication such as Bluetooth Low Energy (registered trademark) or Thread (registered trademark). The communication unit 160 also communicates with a Thread Border Router, a Wi-Fi gateway, an Internet router, etc. via wireless communication such as Wi-Fi (registered trademark).

[0025] The display unit 170 includes a touch panel and displays, for example, images captured by the imaging unit 110 and an operation screen for the user to operate the smart controller 100. The display unit 170 also displays information about the smart devices controlled by the smart controller 100. The display unit 170 may also display other useful information, such as the date, information about today's schedule, weather information, etc.

[0026] The memory 180 is configured by a combination of volatile memory and non-volatile memory. The memory 180 is used to store programs executed by the processor 190, data used for various processes, etc. The processor 190 may be, for example, an MPU (Micro Processor Unit) or a CPU (Central Processing Unit). The processor may include multiple processors.

[0027] Memory 180 also stores table 181 or table 184. As shown in Fig. 5 , table 181 is information in which zone Z names 181A, hand gesture names 181B, and command information 181C are associated with each other. Table 184 is information in which identification numbers 184A serving as camera identification information are associated with zone Z names 184B, hand gesture names 184C, and command information 184D. Table 181 or table 184 is information that is defined in advance by the user.

[0028] Next, a gesture recognition method according to the first embodiment will be described with reference to Fig. 11. First, the imaging unit 110 captures an image of a predetermined imaging range and acquires an image (step S101).

[0029] Next, the gesture estimation unit 130 determines whether or not a hand has been detected from the image acquired in step S101 (step S102).

[0030] In step S102, if the gesture estimation unit 130 determines that a hand has not been detected from the image (step S102; No), the process returns to step S101.

[0031] In step S102, if the gesture estimation unit 130 determines that a hand has been detected from the image (step S102; Yes), the gesture estimation unit 130 estimates a hand gesture and determines whether the hand gesture is a predetermined hand gesture, for example, a "calling" gesture (step S103).

[0032] In step S103, if the gesture estimation unit 130 determines that the hand gesture is not a predetermined hand gesture (step S103; No), the process returns to step S101.

[0033] In step S103, if the gesture estimation unit 130 determines that the hand gesture is a predetermined hand gesture (step S103; Yes), the gesture estimation unit 130 estimates the next hand gesture (step S104).

[0034] Next, the zone identification unit 140 identifies the zone Z in which the image of the hand gesture estimated in step S104 was captured (step S105).

[0035] Next, the command generating unit 150 refers to the table 181 or the table 184 and generates a command associated with the hand gesture estimated in step S104 and the zone identified in step S105 (step S106).

[0036] Next, the communication unit 160 transmits the command generated in step S106 to the appropriate smart device (step S107), and the process ends.

[0037] Note that the smart controller 100 may check the operating state of the smart device before transmitting the command to the smart device in step S107.

[0038] In the smart controller 100 according to the first embodiment, one or more zones Z are set in the imaging range of the imaging unit 110, and a command is generated based on the zone Z in which a hand gesture is captured and the hand gesture itself, and the command is transmitted to the smart device. Therefore, the smart controller 100 can transmit a command to the smart device using a hand gesture, enabling even people with hearing or speech disabilities to use the smart controller 100. Furthermore, because a command corresponding to a hand gesture can be generated for each zone Z, even if many smart devices are placed throughout the house, the user can be prevented from becoming confused about which smart device to identify. This makes it possible to provide a smart controller 100, a gesture recognition method, and a program that are compatible with people with hearing or speech disabilities and that can be used without user confusion.

[0039] Specifically, in a study published in 2006 by psychologist Wendy Wood et al., 100 subjects aged 17 to 79 were asked to record their daily activities. The results revealed that 47% (on average) of the subjects' behaviors were "habits" that occurred in the same place and at the same time. This suggests that the intention to perform a certain behavior occurs in a specific place and at a specific time. Specific examples include "I always sit on the sofa when watching TV," "I check that all the lights are off in the entrance before leaving," "I adjust the air conditioner when working at my desk," and "I get out of bed and open the curtains when I wake up in the morning." Therefore, the smart controller 100 according to the first embodiment can seamlessly control smart devices located throughout the house by associating hand gestures (actions) that define operations (commands) for the smart devices with zone Z (locations).

[0040] Furthermore, when a predetermined hand gesture is estimated by the gesture estimation unit 130, generation of a command is started by the command generation unit 150. This prevents the smart controller 100 from recognizing a hand gesture made by the user unintentionally as an instruction to the smart controller 100 and erroneously controlling the smart device.

[0041] Furthermore, when a portion of one zone Z overlaps with a portion of at least one other zone Z, one type of command may be associated with one type of hand gesture in all of the multiple zones Z that partially overlap in table 181 or table 184. This prevents confusion caused by different commands being defined for one type of hand gesture in the multiple zones Z that partially overlap.

[0042] Furthermore, when a portion of one zone Z overlaps with a portion of at least one other zone Z, the zone identification unit 140 calculates the area of ​​overlap between the bounding box set by the gesture estimation unit 130 for detecting the gesture and the multiple zones Z for each zone Z. The zone identification unit 140 may then identify the zone Z with the largest overlapping area as the zone Z in which the gesture was captured. In this case, even if different commands are associated with one type of hand gesture in the multiple partially overlapping zones Z in table 181 or table 184, confusion does not occur. This makes it possible to associate the same hand gesture with different commands, thereby preventing the number of types of hand gestures to be estimated by the gesture estimation unit 130 from increasing.

[0043] The gesture estimation unit 130 estimates hand gestures using a machine-learned hand gesture estimation model. Specifically, the hand gesture estimation model extracts the joints of the user's hand from the image as key points and estimates the hand gesture based on the skeletal shape formed by the bones connecting the key points. This allows the hand gestures captured by the imaging unit 110 to be estimated with high accuracy.

[0044] Furthermore, as more smart devices are installed throughout the home, the number of commands for controlling the smart devices increases, necessitating an increase in the number of hand gestures associated with the commands. To increase the number of hand gestures, the hand gestures must include, for example, hand gestures involving complex movements or hand gestures using multiple hands. However, with the smart controller 100 according to the first embodiment, hand gestures and commands are associated with each of multiple zones Z, making it possible to associate different commands with the same hand gesture in different zones Z. In other words, the number of hand gestures to be estimated by the gesture estimation unit 130 can be prevented from increasing. Therefore, the gesture estimation unit 130 does not need to estimate hand gestures involving complex movements or hand gestures using multiple hands. Therefore, the hand gesture estimation model used by the gesture estimation unit 130 can be a lightweight model with a low computational load that can recognize hand gestures from real-time images captured by the imaging unit 110. This allows the smart device to be operated in real time by hand gestures via the smart controller 100.

[0045] The smart controller 100 also receives images captured by the camera built into the smart controller 100 or the external camera 600, and in table 184, a zone Z set in the imaging range of the camera built into the smart controller 100 or the external camera 600, a gesture, a command, and further identification information of the camera built into the smart controller 100 or the external camera 600 are associated with each other. This enables the smart controller 100 to transmit to the smart device a hand gesture captured in zone Z set in the imaging range of the external camera 600 installed at a different location from the smart controller 100, and a command based on the zone Z.

[0046] Furthermore, the display unit 170 is equipped with a touch panel, and the user can perform a drag operation or the like on the image acquired by the imaging unit 110, which is displayed on the touch panel, to set one or more zones Z in the imaging range of the imaging unit 110. This allows the user to set the zones Z in a simple manner.

[0047] Embodiment 2 Next, with reference to FIG. 12 , an example of the configuration of a smart home system 200 according to embodiment 2 as a gesture recognition system will be described. The smart home system 200 includes smart controllers 300A, 300B, ... (hereinafter, simply referred to as "smart controllers 300" unless otherwise specified) installed at multiple locations indoors, and smart devices capable of communicating with the smart controller 300, such as a light 400A, a TV 400B, a curtain 400C, a light 400D, a curtain 400E, a window 400F, ... (hereinafter, simply referred to as "smart devices 400" unless otherwise specified). The smart controllers 300 and the smart devices 400 can communicate with each other via a network N. The smart controllers 300A and 300B may also be capable of communicating with each other via the network N. The network N includes wireless communication, such as Bluetooth Low Energy, Thread, and Wi-Fi.

[0048] 12 , the smart controller 300A, the light 400A, the TV 400B, and the curtain 400C are placed in the living room, while the smart controller 300B, the light 400D, the curtain 400E, and the window 400F are placed in the bedroom. The smart controller 300 differs from the smart controller 100 according to the first embodiment in that the memory 180 stores a table 182 instead of the table 181 or the table 184. The smart controller 300 also stores in advance in the memory 180 the name of its installation location, such as "living room" or "bedroom." The same components of the smart controller 300 as those of the smart controller 100 are denoted by the same reference numerals, and their description will be omitted.

[0049] 13 shows an example of the table 182. As shown in FIG. 13 , the table 182 contains information in which a name 182B of a zone Z, a name 182C of a hand gesture, command information 182D, and an installation location 182A of the smart controller 300 are associated with each other. Furthermore, the table 182 may contain, as the command information 182D, a command to be sent to the smart device 400 and information for identifying the smart device 400 to which the command is sent. Here, the information for identifying the smart device 400 is, for example, the name or identification information of the smart device 400. Furthermore, the table 182 contains information that is defined in advance by the user using a method similar to that described in the first embodiment.

[0050] Then, the command generation unit 150 generates a command based on the installation location of the smart controller 300, the zone Z in which the image of the hand gesture was captured, and the hand gesture. Specifically, the command generation unit 150 refers to the table 182 stored in the memory 180, and generates a command associated with the installation location of the smart controller 300, the zone Z in which the image of the hand gesture was captured, and the hand gesture.

[0051] In the smart home system 200 according to the second embodiment, the table 182 associates the installation location 182A of the smart controller 300 with the names 182B of the zone Z, the names 182C of the hand gestures, and the command information 182D. The table 182 may also define, as the command information 182D, the names and identification information of the smart devices 400 to which the commands are sent, along with the commands to be sent to the smart devices 400. This allows the smart controller 300A installed in the living room to control the lights 400A, TV 400B, and curtains 400C arranged in the living room, and the smart controller 300B installed in the bedroom to control the lights 400D, curtains 400E, and window 400F arranged in the bedroom. For example, by performing hand gestures on the smart controller 300A installed in the living room, the smart controller 300A can control at least some of the smart devices 400 arranged in the bedroom in addition to the smart devices 400 arranged in the living room. For example, when a hand gesture for controlling curtains is performed on the smart controller 300A installed in the living room, the smart controller 300A may control both the living room curtains and the bedroom curtains. This is possible by defining the command "open curtains" as well as names and identification information of the living room curtains and the bedroom curtains as command information 182D associated with the installation location 182A "living room," the zone name 182B "other than sofa," and the gesture name 182C "Palm open" on the table 182. On the other hand, when a hand gesture is performed on the smart controller 300B installed in the bedroom, the smart controller 300B can control only the smart device 400 installed in the bedroom.That is, the smart controller 300A installed in the living room can seamlessly control the smart devices 400 installed throughout the house, and the smart controller 300B installed in the bedroom can locally control the smart devices 400 installed in the bedroom. Furthermore, by performing hand gestures on the smart controller 300A installed in the living room, the smart controller 300A can control at least some of the smart devices 400 installed in the bedroom in addition to the smart devices 400 installed in the living room, and by performing hand gestures on the smart controller 300B installed in the bedroom, the smart controller 300B can control at least some of the smart devices 400 installed in the living room in addition to the smart devices 400 installed in the bedroom.

[0052] Furthermore, in the table 182, hand gestures are defined for each installation location 182A of the smart controller 300. Therefore, when the installation location of the smart controller 300 is different, different commands can be defined for the same type of hand gesture. This prevents the number of types of hand gestures to be recognized by the smart controller 300 from becoming too large, and enables the speed of the hand gesture estimation process in the gesture estimation unit 130 to be increased.

[0053] Furthermore, hand gestures associated with command information 182D in table 182, in which the same command is defined for smart devices 400 of the same type, may be the same type of hand gesture. Specifically, in table 182, the same or similar commands are associated with hand gestures of the same type for smart devices 400 placed in every room, such as lights 400A and 400D and curtains 400C and 400E. This allows the user to easily memorize the hand gestures and perform them without even thinking about it. Specifically, when the user defines table 182, the initial setting unit 120 suggests that hand gestures defined for the same command for smart devices of the same type be the same type. For example, if the user selects different types of hand gestures for the same command for smart devices of the same type when defining table 182, the initial setting unit 120 displays a message on the display unit 170 such as, "We recommend that you define the same type of hand gesture for the same command for smart devices of the same type."

[0054] Next, an example of the configuration of a smart home system 200A according to a third embodiment as a gesture recognition system will be described with reference to Fig. 14. The smart home system 200A includes a smart mirror 300C as a gesture recognition device, a smart speaker 500, and a light 400A, a TV 400B, a curtain 400C, a light 400D, and a curtain 400E as smart devices capable of communicating with the smart mirror 300C and the smart speaker 500. The smart mirror 300C, the smart speaker 500, and the smart device 400 can communicate with each other via a network N.

[0055] In the example shown in FIG. 14 , the light 400A, TV 400B, and curtain 400C are located in the living room, and the light 400D and curtain 400E are located in the bedroom. The smart mirror 300C and smart speaker 500 may be located anywhere indoors. In the following description, the smart mirror 300C is assumed to be located in the living room. In the third embodiment, for example, a case may be considered in which a system including a smart speaker 500 is already established indoors, and the smart mirror 300C is incorporated into the system. Then, both the smart mirror 300C and the smart speaker 500 control the light 400A, TV 400B, curtain 400C, light 400D, and curtain 400E. The smart mirror 300C differs from the smart controller 100 according to the first embodiment in that the memory 180 stores a table 183 instead of the table 181 or the table 184. The smart mirror 300C also includes a mirror in the display unit 170. Among the configurations of the smart mirror 300C, the same configurations as those of the smart controller 100 are given the same reference numerals, and the description thereof will be omitted.

[0056] FIG. 15 shows an example of table 183. As shown in FIG. 15, table 183 is information in which zone Z names 183A, hand gesture names 183B, and command information 183C are associated with each other. Specifically, table 183 defines "whole" as the zone in which hand gestures for operating smart devices 400 located in the bedroom are performed. Here, "whole" means that the zone defined as "whole" is the entire imaging range of the imaging unit 110 of smart mirror 300C. Furthermore, in table 183, hand gestures associated with command information 183C defining commands for operating smart devices 400 located in different indoor locations are different. Specifically, in table 183, hand gestures associated with command information 183C for operating smart devices 400 in the living room are different from hand gestures associated with command information 183C for operating smart devices 400 in the bedroom. In other words, in the table 183, for example, different types of hand gestures are associated with command information 183C for opening and closing the living room curtain 400C and command information 183C for opening and closing the bedroom curtain 400E. Thus, when a hand gesture for operating the smart device 400 placed in the bedroom is performed in the living room, the command generator 150 can generate a command for operating the smart device 400 placed in the bedroom, regardless of whether the zone in which the hand gesture is captured is "sofa" or "other than sofa." The table 183 is information that is defined in advance by the user using the same method as described in the first embodiment.

[0057] In the smart home system 200A according to the third embodiment, the hand gestures associated with commands for operating the smart device 400 in the living room and the hand gestures associated with commands for operating the smart device 400 in the bedroom are different in the table 183. This allows the smart mirror 300C placed in the living room to control the smart device 400 placed in the bedroom without confusion, even if the smart device 400 is the same type as the smart device 400 placed in the living room. In other words, a person with a hearing or speech impairment can control the smart device 400 placed in the bedroom using the smart mirror 300C placed in the living room as a UI (User Interface).

[0058] The present invention is not limited to the above embodiment and can be modified as appropriate without departing from the spirit of the present invention. For example, in tables 181, 182, and 184, zone Z, hand gestures, and commands for the smart controllers 100 and 300 including the tables 181, 182, and 184 may be associated with each other. Similarly, in table 183, zone Z, hand gestures, and commands for the smart mirror 300C including the table 183 may be associated with each other.

[0059] In the above-described embodiment, the present disclosure has been described as a hardware configuration, but the present disclosure is not limited to this. The present disclosure can also be realized by having the processor 190 execute a computer program to perform the processing steps shown in the flowchart of FIG. 11 and the processing steps described in other embodiments.

[0060] In the above example, the program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media (tangible storage media). Non-transitory computer-readable media include, for example, magnetic recording media, magneto-optical recording media, CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories. Semiconductor memories include, for example, mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory). The program may also be supplied to a computer by various types of transient computer-readable media (Transitory Computer-Readable Media). Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transient computer-readable media can be supplied to a computer via wired communication paths such as electrical wires and optical fibers, or wireless communication paths.

[0061] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0062] Each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0063] This application claims priority based on Japanese Patent Application No. 2023-206989, filed December 7, 2023, and Japanese Patent Application No. 2024-113380, filed July 16, 2024, the disclosures of which are incorporated herein in their entireties.

[0064] It is possible to provide a gesture recognition device, a gesture recognition system, a gesture recognition method, and a program that can be used by people with hearing or speech disabilities and that can be used without causing confusion to the user.

[0065] 100, 300A, 300B, ... Smart controller (gesture recognition device) 110 Imaging unit 120 Initial setting unit (zone setting unit) 130 Gesture estimation unit 140 Zone identification unit 150 Command generation unit 160 Communication unit 170 Display unit 180 Memory 181, 182, 183, 184 Table 190 Processor 200, 200A Smart home system (gesture recognition system) 300C Smart mirror (gesture recognition device) 400A, 400D Lighting (smart device) 400B TV (smart device) 400C, 400E Curtain (smart device) 400F Window (smart device) 500 Smart speaker

Claims

1. A gesture recognition device comprising: a gesture estimation unit that estimates a user's gesture based on an image acquired by an imaging unit; one or more zones are set in the imaging range of the imaging unit, a zone identification unit that identifies a zone in which the gesture is imaged; a command generation unit that generates a command based on the zone in which the gesture is imaged and the gesture; and a communication unit that transmits the command to a smart device.

2. The gesture recognition device according to claim 1, wherein, when a predetermined gesture is estimated by the gesture estimation unit, the command generation unit starts generating a command based on the gesture estimated by the gesture estimation unit after the predetermined gesture and the zone in which the gesture was captured and identified by the zone identification unit.

3. The gesture recognition device according to claim 1, wherein the command generation unit generates a command based on a table that associates zones set in an imaging range of the imaging unit, gestures, and command information.

4. The gesture recognition device according to claim 3, wherein the gesture recognition device is installed at a plurality of locations in a single room, and the table is information associating the zone, the gesture, the command information, and further the installation location of the gesture recognition device.

5. The gesture recognition device according to claim 4, wherein the table defines, as the command information, a command to be transmitted to the smart device and information for identifying the smart device to which the command is transmitted.

6. The gesture recognition device according to claim 4, wherein in the table, the gestures associated with the command information in which the same command for smart devices of the same type is defined are gestures of the same type.

7. The gesture recognition device according to claim 4, wherein the gestures associated with the command information in the table, in which commands for operating the smart devices placed at different locations indoors, are different.

8. The gesture recognition device according to claim 3, wherein when a plurality of zones are set in the imaging range of the imaging unit and a part of one zone overlaps with a part of at least one other zone, in the table, one type of gesture is associated with one type of command in all of the plurality of zones that have overlapping parts.

9. The gesture recognition device according to claim 3, wherein, when a plurality of zones are set within the imaging range of the imaging unit and a part of one zone overlaps with a part of at least one other zone, the zone identification unit calculates an overlap area between a bounding box set by the gesture estimation unit to detect the gesture and the plurality of zones for each zone, and identifies the zone with the largest overlap area as the zone in which the gesture was captured.

10. The gesture recognition device according to claim 3, wherein the gesture recognition device receives images captured by a plurality of the imaging units, and the table is information that associates zones set in the imaging range of the imaging units, gestures, commands, and further identification information of the imaging units.

11. The gesture recognition device of claim 1, wherein the gesture is a hand gesture of the user, and the gesture estimation unit estimates the hand gesture using a machine-learned hand gesture estimation model, and the hand gesture estimation model extracts the joints of the user's hand from the image as key points and estimates the hand gesture based on a skeletal shape formed by bones connecting the key points.

12. The gesture recognition device according to claim 1, further comprising a touch panel, and one or more zones are set in an imaging range of the imaging unit based on an operation of the user on an image acquired by the imaging unit and displayed on the touch panel.

13. The gesture recognition device according to claim 1, further comprising a zone setting unit that creates a heat map indicating the frequency with which the user was present within the imaging range based on an image obtained by the imaging unit capturing an image of the imaging range for a predetermined period of time, and sets a range of a predetermined size as a zone centered on a position where the frequency with which the user was present is higher than the predetermined frequency.

14. A gesture recognition system comprising: a gesture recognition device that recognizes a user's gesture; and at least one smart device capable of communicating with the gesture recognition device, wherein the gesture recognition device comprises: a gesture estimation unit that estimates the user's gesture based on an image acquired by an imaging unit; one or more zones are set in an imaging range of the imaging unit; a zone identification unit that identifies a zone in which the gesture is imaged; a command generation unit that generates a command based on the zone in which the gesture is imaged and the gesture; and a communication unit that transmits the command to the smart device.

15. A gesture recognition method, comprising: a processor: estimating a user's gesture based on an image acquired by an imaging unit; identifying a zone in which the gesture is imaged from one or more zones set in the imaging range of the imaging unit; generating a command based on the zone in which the gesture is imaged and the gesture; and transmitting the command to a smart device.

16. A program that causes a processor to execute the following processes: estimating a user's gesture based on an image acquired by an imaging unit; identifying a zone in which the gesture is imaged from one or more zones set in the imaging range of the imaging unit; generating a command based on the zone in which the gesture is imaged and the gesture; and transmitting the command to a smart device.

Images