Remote control device, and remote control method

Abstract

To provide a remote control device that can properly identify an operation object instrument even when a plurality users exists.SOLUTION: A user identification unit (a registration face image recognition unit 2 and face image registration unit 3) is configured to identify a user. Location information on a plurality of operation object instruments is registered in an operation object instrument registration unit 6. A visual line detection unit 4 is configured to detect a visual line of the user the user identification unit identifies. An operation object instrument identification unit 7 is configured to identify the operation object instrument located in a direction of the visual line the visual line detection unit 4 detects of the plurality of operation object instruments registered in the operation object instrument registration unit 6 as the operation object instrument targeted for being controlled. A voice response control unit 8 is configured to control the identified operation object instrument according to a voice a microphone 10 collects.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a remote control device and a remote control method. [Background technology]

[0002] 2. Description of the Related Art Remote control devices called smart speakers that can recognize voice and operate various devices by voice have become widespread. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 142295 [Patent Document 2] JP 2017-102687 A Summary of the Invention [Problem to be solved by the invention]

[0004] For example, two televisions may be operated as target devices by a remote control device. In this case, it is necessary to give each of the two televisions a unique name and register it in the remote control device, identify the television to be operated by speaking the name, and then give voice instructions on the control content. Such voice instructions are cumbersome, and there is a demand for simplifying voice instructions. Patent Document 1 describes a technology for identifying a target device by detecting a user's line of sight from image information captured by an imaging device. However, when multiple users are present within the imaging range of the imaging device, the imaging device may detect the line of sight of a user other than the user who wants to operate the target device, and the target device may not be identified properly. Therefore, it is desirable to have a remote control device and a remote control method that can identify a target device more reliably.

[0005] An object of the present invention is to provide a remote control device and a remote control method that can appropriately identify a target device even when there are multiple users. [Means for solving the problem]

[0006] The present invention provides a remote control device including a user identification unit that identifies a user, an operation target device registration unit in which position information of a plurality of operation target devices is registered, a gaze detection unit that detects the gaze of the user identified by the user identification unit, an operation target device identification unit that identifies, among the plurality of operation target devices registered in the operation target device registration unit, an operation target device located in the direction of the gaze detected by the gaze detection unit as an operation target device to be controlled, and a voice response control unit that controls the operation target device identified by the operation target device identification unit in accordance with voice picked up by a microphone.

[0007] The present invention provides a remote control method in which a user identification unit identifies a user, a gaze detection unit detects the direction of the gaze of the user identified by the user identification unit, and an operation target device identification unit identifies, among a plurality of operation target devices registered in an operation target device registration unit, an operation target device located in the direction of the gaze detected by the gaze detection unit as the operation target device to be controlled, and a voice response control unit controls the operation target device identified by the operation target device identification unit in accordance with a voice picked up by a microphone. Effect of the Invention

[0008] According to the remote control device and the remote control method of the present invention, even if there are a plurality of users, it is possible to appropriately identify the device to be operated. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of a situation in which there is a remote control device according to the first or second embodiment and two televisions as operation target devices. [Diagram 2] FIG. 2 is a block diagram showing the remote control device according to the first embodiment. [Diagram 3] FIG. 3 is an external perspective view showing the remote control device according to the first or second embodiment. [Figure 4]FIG. 4 is a diagram showing an example of map information registered in an operation target device registration unit in the remote control device according to the first or second embodiment. [Diagram 5] FIG. 5 is a conceptual diagram showing an example of map information registered in an operation target device registration unit in the remote control device according to the first or second embodiment. [Figure 6] FIG. 6 is a conceptual diagram showing a face image of a user registered in a face image registration section in the remote control device according to the first or second embodiment. [Figure 7] FIG. 7 is a block diagram showing a remote control device according to the second embodiment. [Figure 8] FIG. 8 is a conceptual diagram showing a user's voice registered in a voice registration unit in a remote control device according to the second embodiment. [Figure 9] FIG. 9 is a block diagram showing a remote control device according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] In FIG. 1, it is assumed that the television Tv1 is placed in the living room, and the television Tv2 is placed in the dining room. The televisions Tv1 and Tv2 are examples of devices to be operated. A user 60 operates the television Tv1 or Tv2 by the remote control device 100 according to the first embodiment, the remote control device 200 according to the second embodiment, or the remote control device 300 according to the third embodiment. Hereinafter, the configurations and operations of the remote control device 100 according to the first embodiment, the remote control device 200 according to the second embodiment, and the remote control device 300 according to the third embodiment, and the remote control methods executed by the remote control devices 100, 200, and 300 will be described with reference to the accompanying drawings.

[0011] First Embodiment 2, the remote control device 100 according to the first embodiment includes a camera 1, a registered face image recognition unit 2, a face image registration unit 3, a gaze detection unit 4, an operation target device position calculation unit 5, an operation target device registration unit 6, an operation target device identification unit 7, a voice response control unit 8, and a remote control signal generation unit 9. The remote control device 100 according to the first embodiment also includes a microphone 10, a voice recognition unit 11, a voice synthesis control unit 12, a voice synthesis unit 13, a speaker 14, an operation unit 15, and a registration control unit 16.

[0012] As shown in Fig. 3, remote control device 100 has a configuration in which camera 1 is attached to the top of housing 50. The components shown in Fig. 2 other than camera 1 are housed inside housing 50. Camera 1 is preferably a wide-angle camera with a wide imaging range, for example a 360-degree camera.

[0013] The registered face image recognition unit 2, the operation target device position calculation unit 5, the operation target device identification unit 7, the voice recognition unit 11, the voice response control unit 8, the voice synthesis control unit 12, the voice synthesis unit 13, and the registration control unit 16 can be configured by a central processing unit (CPU) of a microcomputer. The face image registration unit 3 and the operation target device registration unit 6 can be configured by a memory of a microcomputer. The gaze detection unit 4 can be configured by applying the technology described in Patent Document 2.

[0014] The remote control signal generator 9 can be configured by a generating circuit including a light emitting diode that generates an infrared code. The voice recognition unit 11 and the voice synthesis unit 13 may be configured by an integrated circuit separate from the CPU. The operation unit 15 may be an operation button provided on the housing 50, or may be a touch pad.

[0015] (Registering the device to be operated) When the user 60 (or another user) operates the operation unit 15, the registration control unit 16 registers the names of the televisions Tv1 and Tv2 in the operation target device registration unit 6. As shown in Fig. 4, it is assumed that the television Tv1 is registered as the living room television, and the television Tv2 is registered as the dining room television.

[0016] When the user 60 operates the operation unit 15 to set a registration mode for registering the position information of the televisions Tv1 and Tv2, the registration control unit 16 controls the voice synthesis unit 13 to generate a voice from the speaker 14 instructing the user 60 to watch the televisions Tv1 and Tv2 in that order. The method of giving instructions to the user 60 is not limited to voice. When the user 60 looks at the television Tv1, the gaze detection unit 4 detects the direction of the user 60's gaze. The operation target device position calculation unit 5 calculates at least the direction of the television Tv1 as seen from the remote control device 100, based on the gaze direction detected by the gaze detection unit 4.

[0017] The target device position calculation unit 5 may calculate an approximate distance from the remote control device 100 to the television Tv1 by calculating the intersection of the left and right lines of sight detected by the line of sight detection unit 4. The position of the target device such as the television Tv1 is at least the direction of the target device as seen from the remote control device 100, and preferably the direction of the target device as seen from the remote control device 100 and the distance from the remote control device 100 to the target device.

[0018] The registration control unit 16 registers the position information of the television Tv1 calculated by the operation target device position calculation unit 5 in the operation target device registration unit 6. When the user 60 subsequently watches the television Tv2, the operation target device position calculation unit 5 similarly calculates the position of Tv2, and the registration control unit 16 registers the position information of Tv2 in the operation target device registration unit 6.

[0019] As shown in Fig. 5, it is assumed that television Tv1 is located at a distance Ds1 in a direction Dr1 as viewed from the remote control device 100, and television Tv2 is located at a distance Ds2 in a direction Dr2 as viewed from the remote control device 100. As shown in Fig. 4, the operation target device registration unit 6 registers the direction Dr1 and distance Ds1 as position information of television Tv1, which is a television in the living room, and the direction Dr2 and distance Ds2 as position information of television Tv2, which is a television in the dining room. In this manner, the registration control unit 16 registers each operation target device in the operation target device registration unit 6 as map information.

[0020] The method of registering the target devices is not limited to the above, and the user may manually input the location information of each target device.

[0021] (Registering user's face image) In Fig. 2, when user 60 (or another user) operates operation unit 15 to set a registration mode for registering a face image, registration control unit 16 registers the face image of user 60 captured by camera 1 in face image registration unit 3. As shown in Fig. 6, face images of users who operate remote control device 100 are registered in face image registration unit 3 in association with user IDs. The face images registered in face image registration unit 3 may be face images of each user captured by camera 1, or may be face images of each user captured by a different camera.

[0022] (Operation of equipment) The user 60 shown in FIG. 1 is a user whose user ID is 0001. The registered face image recognition unit 2 recognizes whether or not a person photographed by the camera 1 is a user registered in the face image registration unit 3, based on a face image registered in the face image registration unit 3. The gaze detection unit 4 detects the gaze direction of a person who the registered face image recognition unit 2 recognizes as a user registered in the face image registration unit 3. By detecting the gaze direction of a person who the gaze detection unit 4 recognizes as a user registered in the face image registration unit 3, it is possible to avoid erroneously detecting the gaze of a person other than the registered user. The registered face image recognition unit 2 and the face image registration unit 3 function as a user identification unit that identifies a user who operates the remote control device 100.

[0023] The operation target device identifying unit 7 identifies an operation target device located in the direction of the line of sight detected by the line of sight detection unit 4 as an operation target device to be operated by voice, among the multiple operation target devices registered in the operation target device registration unit 6. As shown in Fig. 1, when a user 60 is watching a television Tv1, the operation target device identifying unit 7 identifies the television Tv1 as an operation target device to be operated by voice.

[0024] With the target device identification unit 7 identifying the television Tv1 as the target device, suppose that the user 60 utters, for example, "Please turn on the power." The voice recognition unit 11 recognizes the voice and instructs the voice response control unit 8 to turn on the power. The voice response control unit 8 acquires information from the target device identification unit 7 that the television Tv1 has been identified as the target device. The voice response control unit 8 instructs the remote control signal generation unit 9 to generate a remote control signal to turn on the power of the television Tv1.

[0025] The remote control signal generating section 9 generates a remote control signal to turn on the power of the television Tv1 in response to an instruction from the voice response control section 8. This causes the power of the television Tv1 to be turned on.

[0026] Similarly, when the operation target device identification unit 7 has identified the television Tv1 as the target operation target device, if the user 60 utters, for example, "Please turn up the volume," the volume of the television Tv1 can be turned up.

[0027] In this way, even if there are televisions Tv1 and Tv2 as multiple operation target devices, the user 60 does not need to instruct the remote control device 100 using long sentences such as "Please turn on the television in the living room" or "Please turn up the volume of the television in the living room." The remote control device 100 can simplify voice instructions.

[0028] Now, suppose that the user 60 utters "Please turn on the power" when the operation target device specification unit 7 has not specified the operation target device to be operated by voice. In this case, the voice synthesis control unit 12 controls the voice synthesis unit 13 to synthesize an inquiry sentence such as "Shall I turn on the power of the living room TV or the dining room TV?" and output it from the speaker 14.

[0029] Upon hearing the inquiry, user 60 utters, "Please turn on the TV in the living room." Voice recognition unit 11 recognizes the voice uttered by user 60 and instructs voice response control unit 8 to turn on the power of television Tv1, which is the television in the living room. Voice response control unit 8 instructs remote control signal generation unit 9 to generate a remote control signal to turn on the power of television Tv1.

[0030] In this way, if the target device identification unit 7 does not identify the target device, it may be necessary to generate voice from the speaker 14, pick up the voice with the microphone 10, and recognize the voice with the voice recognition unit 11, before the voice response control unit 8 can identify the target device. By having the target device identification unit 7 identify the target device, it becomes possible to operate the device by speaking extremely simple sentences.

[0031] Furthermore, even if there are multiple users, including users other than registered users, within the imaging range of camera 1, the device to be operated can be appropriately identified by detecting the line of sight of the registered user and identifying the device to be operated.

[0032] In the above explanation, an example was given of when a device to be operated is identified based on the gaze of a registered user and voice is detected, and then device operation is performed based on the detected voice. However, it is not necessary to detect the gaze of a registered user all the time, and a configuration may be used in which the gaze of a registered user is detected and the device to be operated is identified when voice is detected.

[0033] According to the remote control device 100, the device to be operated is identified based on the line of sight of a specific user registered in the face image registration unit 3, so that even when there are multiple users, the device that a specific user wants to operate can be accurately identified.

[0034] <Second embodiment> 7 shows a remote control device 200 according to the second embodiment. In the remote control device 200, the same parts as those of the remote control device 100 are given the same reference numerals, and the description thereof will be omitted. The remote control device 200 includes a voice registration unit 17 and a registered voice recognition unit 18, which are configurations that the remote control device 100 does not include. Registration of a device to be operated is the same as in the first embodiment, and the description thereof will be omitted.

[0035] (User voice registration) 7, when user 60 (or another user) operates operation unit 15 to set a user registration mode for registering user 60, registration control unit 16 registers the voice of user 60 in voice registration unit 17. In the registration mode, microphone 10 picks up a predetermined sentence uttered by user 60. Registration control unit 16 registers the voice picked up by the microphone in voice registration unit 17.

[0036] 8, the registration control unit 16 registers the user's voice in the voice registration unit 17 in association with the user ID. The user ID registered in the voice registration unit 17 corresponds one-to-one with the user ID registered in the face image registration unit 3. In other words, if the user registered in the face image registration unit 3 and the user registered in the voice registration unit 17 have the same user ID, they are the same user. Therefore, the voice registration unit 17 registers the user's voice in association with the user's face image registered in the face image registration unit 3.

[0037] During normal operation of the remote control device 200, suppose that the user 60 utters, for example, "Please turn on the power." The registered voice recognition unit 18 recognizes whether the voice picked up by the microphone 10 is the voice of a user registered in the voice registration unit 17. The registered voice recognition unit 18 can recognize whether the voice is the voice of a registered user based on the voiceprint of the voice.

[0038] When the registered voice recognition unit 18 recognizes that the voice picked up by the microphone 10 is the voice of the user 60 registered in the voice registration unit 17, it identifies the user ID and identifies a user face image corresponding to the identified user ID based on the user face image data registered in the face image registration unit 3. The registered face image recognition unit 2 recognizes the face image of the identified user. The gaze detection unit 4 detects the gaze direction of the user recognized by the registered face image recognition unit 2. The operation target device identification unit 7 identifies, among the multiple operation target devices registered in the operation target device registration unit 6, an operation target device located in the gaze direction detected by the gaze detection unit 4 as the operation target device to be controlled.

[0039] When the registered voice recognition unit 18 recognizes that the voice picked up by the microphone 10 is the voice of the user 60 registered in the voice registration unit 17, it notifies the voice response control unit 8 that it has recognized the voice of the user registered in the voice registration unit 17. When the voice response control unit 8 is notified that it has recognized the voice of the registered user, it instructs the remote control signal generation unit 9 to generate a remote control signal to turn on the power of the operation target device identified by the operation target device identification unit 7, in accordance with the instruction from the voice recognition unit 11 to turn on the power.

[0040] 7, the face image registration unit 3 and the voice registration unit 17 are shown separately, but the face image registration unit 3 and the voice registration unit 17 may be configured as one user registration unit. The registered face image recognition unit 2, the face image registration unit 3, the voice registration unit 17, and the registered voice recognition unit 18 function as a user identification unit that identifies the user who operates the remote control device 200.

[0041] According to the remote control device 200, even if there are multiple registered users and each registered user gives a voice instruction, the user is identified for each detected voice and the device to be operated is identified based on the gaze direction of the identified user, so that the device to be operated can be identified more appropriately than the remote control device 100.

[0042] According to the remote control device 200, the configuration of the remote control device 100 is provided with a registered voice recognition unit 18 that identifies a user based on an operation instruction voice, so that even if there are multiple registered users, it is possible to identify the user who issued the operation instruction voice and to identify the device to be operated based on the line of sight of the identified user. Therefore, according to the remote control device 200, even if there are multiple users, it is possible to more accurately identify the device that a specific user wants to operate.

[0043] <Third embodiment> 9 shows a remote control device 300 according to the third embodiment. In the remote control device 300, the same parts as those in the remote control device 100 are given the same reference numerals, and their description will be omitted. In the remote control device 100, the registered face image recognition unit 2 recognizes a face registered in the face image registration unit 3 to identify the user operating the remote control device 100. In contrast, the remote control device 300 includes a voice detection unit 30 that estimates the voice generation direction, and identifies the user operating the remote control device 300 based on the voice generation direction estimated by the voice detection unit 30.

[0044] 9, the voice detection unit 30 detects voice and estimates the direction of the detected voice. The voice detection unit 30 transmits the detected voice to the voice recognition unit 11, and transmits information indicating the estimated voice generation direction to the user identification unit 31. The voice detection unit 30 can estimate the voice generation direction using existing technology. For example, the voice detection unit 30 includes multiple microphones, and estimates the voice generation direction based on the phase difference of the voices detected by the multiple microphones.

[0045] The voice detection unit 30 does not need to be configured to include multiple microphones as an integrated unit, and multiple microphones may be installed in different locations in a room. The voice detection unit 30 may estimate the voice generation direction based on the magnitude or phase difference of the voice detected by each of the multiple microphones installed in different locations in a room. The method of estimating the voice generation direction is not limited to the method using multiple microphones, and other methods may be used.

[0046] The user identification unit 31 identifies the user operating the remote control device 300 based on the information indicating the voice generation direction supplied from the voice detection unit 30 and the imaging result of the camera 1. Specifically, the user identification unit 31 identifies a person existing in the voice generation direction estimated by the voice detection unit 30 as the user operating the remote control device 300. The user identification unit 31 can be configured with a CPU.

[0047] The gaze detection unit 4 detects the gaze of the user identified by the user identification unit 31. The operation target device identification unit 7 identifies an operation target device located in the direction of the gaze detected by the gaze detection unit 4 as an operation target device to be operated by voice, among a plurality of operation target devices registered in the operation target device registration unit 6. The operation after the operation target device identification unit 7 identifies the operation target device is the same as that of the remote control device 100.

[0048] Instead of identifying a person who is in the voice generation direction estimated by the voice detection unit 30 as the user operating the remote control device 300, a modified example of the third embodiment may be configured as follows. The user identification unit 31 identifies, from among the people captured by the camera 1, a person whose mouth moved at the timing when the voice detection unit 30 detected a voice, as the user operating the remote control device 300. In this case, the voice detection unit 30 does not need to have a function of estimating the voice generation direction.

[0049] The present invention is not limited to the first to third embodiments described above, and various modifications are possible without departing from the gist of the present invention. [Explanation of symbols]

[0050] 1 Camera 2. Registered face image recognition unit (user identification unit) 3. Face image registration unit (user identification unit) 4. Gaze detection unit 5. Target device position calculation unit 6. Target device registration section 7. Device identification section 8 Voice response control unit 9. Remote control signal generator 10 Microphone 11 Voice Recognition Unit 12 Voice synthesis control unit 13 Voice synthesis section 14 Speaker 15 Control section 16 Registration control section 17 Voice registration unit (user identification unit) 18 Registration voice recognition unit (user identification unit) 30 Voice detection section 31 User identification part 60 users 100,200,300 Remote control device Tv1, Tv2 Television

Claims

1. A face image registration unit where the face image of the user operating the target device is registered, The user's voice is registered in association with the user's facial image registered in the aforementioned facial image registration unit. The voice registration unit, Based on the voice registered in the aforementioned voice registration unit, the microphone emitted the voice it picked up. A registration voice recognition unit that identifies the user, Based on the facial images registered in the aforementioned facial image registration unit, a person photographed by the camera The facial image registration unit recognizes whether or not the user is registered, and the registration sound A registered face image recognition unit identifies the face image of the user identified by the voice recognition unit, A user identification unit equipped with, A gaze detection unit that detects the gaze of the user identified by the user identification unit, Among the multiple devices to be operated, the operating device located in the direction of the line of sight detected by the eye-line detection unit An operating target device identification unit that identifies the target device as an operating target device to be controlled, According to the sound picked up by the microphone, the target device identification unit is identified. A voice response control unit that controls the target device, A remote control device equipped with the following features.

2. It further includes a voice detection unit that estimates the direction of voice emission, The user identification unit identifies the user based on the direction of speech estimated by the voice detection unit. Determine The remote control device according to claim 1.

3. In the registration mode for registering the direction in which the multiple target devices are located, the user, While looking at one of the multiple target devices, the eye-tracking detection The position of the target device is calculated based on the line of sight detected by the output unit. Calculation unit, The operation unit registers the location information of the target device calculated by the target device location calculation unit. Elephant equipment registration department, The remote control device according to claim 1 or 2, further comprising:

4. The user identification unit identifies the user who made the sound picked up by the microphone. The gaze detection unit detects the direction of the user's gaze identified by the user identification unit, The device identification unit identifies multiple devices registered in the device registration unit. The control target device is located in the direction of the line of sight detected by the eye-line detection unit. Identify the device to be operated on, The voice response control unit identifies the target device according to the sound picked up by the microphone. Controlling the target equipment identified by the department Remote control method.

Images