Remote Control Method

Abstract

To provide a remote control technology for remarkably improving user's convenience by eliminating complexity of searching for a remote control.SOLUTION: A remote control method in an information processing terminal for remotely controlling an apparatus to be controlled extracts information for identifying an apparatus to be controlled and information for identifying desired control operation from description information of a presentation surface included in a circumferential image of the information processing terminal, selecting the apparatus to be controlled to be a remote control object on the basis of the information for identifying the apparatus to be controlled, extracts a control command corresponding to the information for identifying the desired control operation, generates a virtual remote control including a reception object for corresponding to the control command and receiving a user's operation instruction, and receives a user's operation instruction corresponding to the reception object to transmit a control command corresponding to the reception object to the apparatus to be controlled.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a remote control technique for performing remote control using a portable information terminal. [Background technology]

[0002] As a portable information terminal, there is a head-mounted display (hereinafter referred to as "HMD") that displays images of real space superimposed on the display screen of the glasses, and computer-generated images of augmented reality (AR) (AR objects such as avatars).

[0003] Meanwhile, the number of devices operated with a remote controller (hereinafter referred to as "remote control") is increasing. In response to this, learning remote controls (which store and support various remote control functions) that integrate various remote control functions, and smart remote controls (which manage device operations via Internet communication such as Wi-Fi or wireless LAN) are becoming common. For example, Patent Document 1 discloses a remote control system in which a directional general-purpose remote control is aimed at a target remotely controlled device to establish a wireless link, and the general-purpose remote control receives an operation application transmitted from the remotely controlled device and operates the remotely controlled device. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2012 / 132171 Summary of the Invention [Problem to be solved by the invention]

[0005] According to the technology disclosed in Patent Document 1, a user can select a device by simply pointing a universal remote control at the device they want to remotely control, and then use the universal remote control to remotely control the selected device. However, not only universal remote controls, but also off-the-shelf remote controls do not necessarily have user-friendly interfaces, as they may have function buttons for functions that are not normally used, or the layout of the function buttons may differ from the user's preferences. Furthermore, even universal remote controls do not eliminate the hassle of searching for the remote control.

[0006] The present invention has been made in view of the above circumstances, and has as its object to provide a remote control technique that eliminates the hassle of searching for a remote control and significantly improves user convenience. [Means for solving the problem]

[0007] The present invention is a remote control method for an information processing terminal that remotely controls a controlled device, the method including: an image acquisition step of acquiring a surrounding image that is an image of the surroundings of the information processing terminal; an information extraction step of extracting information that identifies the controlled device and information that identifies a desired control operation from written information on a presentation surface included in the surrounding image acquired in the image acquisition step; a device selection step of selecting the controlled device to be a target of the remote control based on the information that identifies the controlled device extracted in the information extraction step; a control command extraction step of extracting a control command that corresponds to the information that identifies the desired control operation extracted in the information extraction step for the controlled device selected in the device selection step; and a reception object that corresponds to the control command extracted in the control command extraction step and receives a user's operation instruction. Contains a virtual remote control generation step of generating a virtual remote control; a virtual remote controller display step of displaying the reception object included in the virtual remote controller generated in the virtual remote controller generation step in an area on the presentation surface where information specifying the desired control operation is described; The method is characterized by comprising an operation reception step of receiving an operation instruction from the user for the reception object, and a command transmission step of transmitting the control command corresponding to the reception object for which the operation instruction from the user has been received in the operation reception step to the controlled device selected in the device selection step. [Effects of the Invention]

[0008] The present invention provides a remote control technology that eliminates the hassle of searching for a remote control and significantly improves user convenience. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is an explanatory diagram for explaining an overview of an embodiment of the present invention. [Figure 2] 1 is a system configuration diagram of a remote control system according to an embodiment of the present invention. [Figure 3] 1 is a hardware configuration diagram of an HMD according to an embodiment of the present invention. [Figure 4] FIG. 2 is a functional block diagram of an HMD according to an embodiment of the present invention. [Figure 5] FIG. 10 is an explanatory diagram illustrating an example of a virtual remote control image according to an embodiment of the present invention. [Figure 6] FIG. 4 is an explanatory diagram for explaining an example of an analysis table according to the embodiment of the present invention. [Figure 7] FIG. 1A is an explanatory diagram illustrating an example of a control data table according to an embodiment of the present invention, and FIG. 1B is an explanatory diagram illustrating an example of a remote control table according to an embodiment of the present invention. [Figure 8] FIG. 10 is an explanatory diagram illustrating an example of a gesture action table according to the embodiment of the present invention. [Figure 9] 10 is a flowchart of a remote control process according to an embodiment of the present invention. [Figure 10] 10 is a flowchart of a specific information acquisition process according to an embodiment of the present invention. [Figure 11] 10 is a flowchart of a control data acquisition process according to an embodiment of the present invention. [Figure 12] 10A is an explanatory diagram illustrating an example of a presentation surface according to a modified example of the present invention, and FIGS. 10B and 10C are explanatory diagrams illustrating examples of virtual remote control images according to modified examples of the present invention. [Figure 13]10A is an explanatory diagram illustrating an example of a presentation surface according to a modified example of the present invention, and FIG. 10B is an explanatory diagram illustrating an example of a virtual remote control image according to a modified example of the present invention. [Figure 14] FIG. 10 is a system configuration diagram of a remote control system according to a modified example of the present invention. [Figure 15] FIG. 10 is an explanatory diagram for explaining an overview of a modified example of the present invention. [Figure 16] FIG. 10 is a hardware configuration diagram of a smartphone according to a modified example of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] An embodiment of the present invention will be described with reference to the drawings. Prior to describing the configuration of the embodiment, an outline of the embodiment will be described. Note that the same reference numerals will be used throughout this specification and the drawings to designate the same components.

[0012] This embodiment is designed with elderly users, who tend to lose their remote controllers (remote control devices), in mind. Not only do elderly users lose their remote controls easily, but they also often find the numerous buttons on ready-made remote controls cumbersome. To solve this problem, this embodiment allows users to simply write down the minimum necessary control operations by hand on a suitable piece of paper at any time, and then use the remote control as if it were a real remote control.

[0013] 1 is a diagram for explaining an overview of this embodiment. In this embodiment, a user 901 wearing an HMD (head mounted display) 100 prepares a piece of paper or the like on which the name of a device (controlled device) to be operated and a desired control operation (desired control operation) are written. Hereinafter, the paper or the like on which the information is written is referred to as a presentation surface 300, and the written information is referred to as written information 310.

[0014] The HMD 100 then generates the remote control functions of the controlled device as a virtual remote control image 400 from the written information 310 on the presentation surface 300 viewed by the user 901 via the display 131. The HMD 100 then accepts the operation of the user 901 on the presentation surface 300 as an operation on the virtual remote control image 400, and operates the controlled device from the HMD 100.

[0015] The virtual remote control image 400 is generated by the HMD 100 by reading the written information 310 on the presentation surface 300 and associating it with a control command for the controlled device. The generated virtual remote control image 400 may be displayed on the display 131 of the HMD 100. For example, a desired control operation written on the presentation surface 300 may be recognized and displayed as a function button (reception object) on the display 131 of the HMD 100. Then, the user 901 performs an operation such as a gesture to give an instruction to the function button, thereby controlling the controlled device.

[0016] The presentation surface 300 may be, for example, a sheet-like member such as the above-mentioned paper. The written information 310 may be handwritten. However, the presentation surface 300 and the written information 310 are not limited to this.

[0017] As a result, according to this embodiment, the user 901 can operate the controlled device as if using the presentation surface 300 that he or she has prepared as a remote control.

[0018] [System Configuration] First, the system configuration of the remote control system 500 of this embodiment that realizes the above will be described below. Fig. 2 is a system configuration diagram of the remote control system 500 of this embodiment.

[0019] As shown in the figure, the remote control system 500 of this embodiment includes an HMD 100 , a controlled device 510 , a wireless router 520 , and a network server 550 .

[0020] The wireless router 520 is connected to the network server 550 via the network 530. The HMD 100 is connected to the network server 550 via the wireless router 520.

[0021] The controlled device 510 is connected to the wireless router 520 via a wired LAN or a wireless LAN. The HMD 100 transmits a control command to the controlled device 510 via the wireless router 520.

[0022] With the above configuration, in this embodiment, the user 901 wearing the HMD 100 can remotely control the controlled device 510.

[0023] The network server 550 is an external server that stores control data for controlling the controlled device 510. The network server 550 transmits the requested control data for the controlled device 510 from the HMD 100 via the wireless router 520 and the network 530 to the HMD 100 that is the sender of the data via the network 530 and the wireless router 520.

[0024] Note that, although one network server 550 is described here as a representative, the number of network servers is not limited to this. For example, the manufacturer of each controlled device 510 may provide a network server 550. Also, the network server 550 is not always necessary. After the HMD 100 acquires the necessary information from the network server 550, the controlled device 510 is controlled through communication between the HMD 100, the wireless router 520, and the controlled device 510.

[0025] The controlled device 510 is a device that the user 901 wearing the HMD 100 wants to control with a remote control. Examples of the controlled device 510 include home appliances such as air conditioners, lighting fixtures, TVs, and electric fans, as well as gas water heaters and floor heating systems to which a home energy management system, HEMS (Home Energy Management System), can be applied. In the following, this embodiment will be described taking an air conditioner as an example of the controlled device 510. Note that the air conditioner is assumed to be a home air conditioner (hereinafter referred to as the common name "air conditioner").

[0026] The HMD 100 is a device worn on the head of a user 901, and displays processed information on a display 131. The HMD 100 of this embodiment has a shape of glasses or goggles, and includes a display 131 at the lens position of the glasses. The display 131 may be a transmissive type or a non-transmissive type.

[0027] In the HMD 100, the user 901 can observe the situation in real space via the display 131. In addition, an augmented reality (AR) object is displayed on the display 131. Therefore, the user 901 can simultaneously view both the AR object displayed on the display 131 and the situation in real space.

[0028] [HMD] Next, the hardware configuration and functional blocks of the HMD 100 of this embodiment will be described with reference to the drawings.

[0029] [Hardware configuration] The hardware configuration of the HMD 100 of this embodiment that realizes the above processing will be described below. Figure 3 is a block diagram showing an example of the configuration of the HMD 100 of this embodiment. In these figures, the same components as those shown in Figures 1 and 2 are assigned the same reference numerals.

[0030] As shown in this figure, the HMD 100 of this embodiment includes a main processor 111, a system bus 112, a memory device 110, an operation reception device 120, an image processing device 130, an audio processing device 140, a sensor 150, a communication device 160, an extension interface (I / F) unit 105, and a timer 106.

[0031] The main processor 111 is a main control unit that controls the entire HMD 100 in accordance with a predetermined program. The main processor 111 is realized by a CPU (Central Processor Unit) or a microprocessor unit (MPU). The main processor 111 performs overall operation control processing of the HMD 100 by executing programs such as an operating system (OS) and various operation control applications stored in the storage device 110. The main processor 111 also controls the startup operations of various applications. The main processor 111 performs processing in accordance with a clock signal measured and output by the timer 106.

[0032] The system bus 112 is a data communication path for transmitting and receiving data between the main processor 111 and each unit in the HMD 100 .

[0033] The storage device 110 includes a RAM 113, a ROM 114, and a flash memory 115. The storage device 110 stores programs such as an operating system and various operation control applications for music, images, documents, etc. It also stores information data such as base data required for basic operations by the operating system and file data launched by various applications. For example, when a music application installed in the HMD 100 is launched and music file data is selected, the selected music file data is played on the HMD 100, allowing the user to listen to the desired music.

[0034] The RAM 113 is a program area for executing basic operation programs and other application programs. The RAM 113 also serves as a temporary storage area for temporarily storing data as needed when various application programs are executed. The RAM 113 may be integrated with the main processor 111.

[0035] The ROM 114 and flash memory 115 store various programs for realizing the functions of the HMD 100, operation setting values, sensor information including detection values ​​from a sensor 150 (described later), and various display data such as virtual objects and content.

[0036] Flash memory 115 stores operation programs downloaded from network 530 and various data created by the operation programs. Each operation program stored in flash memory 115 can be updated and its functions expanded by downloading it from each server device on network 530.

[0037] Furthermore, flash memory 115 can store content such as moving images, still images, and audio downloaded from network 530. It also stores data such as moving images and still images captured by in-camera 134 or out-camera 133.

[0038] The ROM 114 and flash memory 115 are so-called non-volatile storages that retain stored information even when power is not supplied from an external source to the HMD 100. Hereinafter, unless there is a need to distinguish between the ROM 114 and the flash memory 115, they will be collectively referred to as the internal memory storage. The main processor 111 loads new application programs stored in the internal memory storage into the RAM 113 and executes them, thereby enabling the HMD 100 to realize a variety of functions.

[0039] The built-in memory storage needs to retain stored information even when power is not supplied to the HMD 100. Therefore, devices such as flash ROM, SSD (Solid State Drive), and HDD (Hard Disc Drive) are used as the built-in memory storage.

[0040] The operation reception device 120 receives input of operation instructions for the HMD 100. In this embodiment, the HMD 100 includes a button switch 121 and a touch panel 122. The button switch 121 includes a power key, a volume key, a home key, etc. The touch panel 122 receives operation instructions input via a touchpad. Note that the HMD 100 of this embodiment does not necessarily need to include all of these operation reception devices 120. Furthermore, operations for the HMD 100 may be received via a separate information processing terminal device connected by wired or wireless communication.

[0041] The operation reception device 120 may be provided in a position and form within the HMD 100 that allows the user 901 to easily perform input operations. The operation reception device 120 may be separated from the main body of the HMD 100 and connected by wire or wirelessly. Alternatively, the operation reception device 120 may utilize the line of sight of the user 901. For example, an input operation screen is displayed on the display 131, and input operation information is captured based on the position on the input operation screen at which the user 901's line of sight is directed, detected by an in-camera 134 (described later). A pointer may be displayed on the input operation screen, and the input operation information may be captured by operating the pointer. Alternatively, the user 901 may utter a voice indicating an input operation, and the sound may be collected by a microphone 143 (described later), and the input operation information may be captured.

[0042] The image processing device 130 is an image (video) processor, and includes a display 131, an outer camera 133, and an inner camera 134.

[0043] The display 131 is a display device (display) such as a liquid crystal panel, and presents image data processed by the image signal processing unit to the user of the HMD 100. The display 131 includes a display for the left eye and a display for the right eye. The display 131 may be a transmissive (optically transmissive) display or a non-transmissive (video transmissive) display.

[0044] The optically transmissive display includes a projection unit that projects various information such as playback information from the activated application and notification information for the user 901, and a transparent half mirror that forms an image of the projected information in front of the user's eyes. The video transmissive display also includes a liquid crystal panel or the like that displays various information together with real-space objects in front of the user that are photographed by the outer camera 133. Through the display 131, the user 901 can visually recognize not only the image in front of the user's eyes within the field of view, but also playback information such as music, images, and documents from the activated application.

[0045] The image signal processing unit is an image (video) signal processor that processes images input from outer camera 133 and inner camera 134. The image signal processing unit also superimposes an object created by main processor 111 or the like on the input image and outputs the superimposed image to display 131. The image signal processing unit may be realized by main processor 111, or may be realized by a processor dedicated to images that is provided separately from main processor 111.

[0046] In this embodiment, an image imitating a remote control screen or the like is displayed on the display 131 as an AR object.

[0047] The outer camera 133 and the inner camera 134 are cameras that input image data of the surroundings and objects by converting light input from the lens into an electrical signal using electronic devices such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0048] The outer camera 133 captures an image of the surroundings of the HMD 100. The outer camera 133 is installed in front of the HMD 100 and captures the forward visual field. The HMD 100 is classified into an optical transmission type and a video transmission type depending on the type of display 131, but both types capture and capture the visual field in front of the user's eyes, captured by the outer camera 133.

[0049] The in-camera 134 captures an image of a different area from that of the out-camera 133. For example, it captures an image of the eyes of the user 901. In this embodiment, the in-camera 134 functions as a gaze detection sensor together with a processing unit that performs gaze identification processing.

[0050] The gaze detection sensor includes a right eye gaze detection unit and a left eye gaze detection unit, which detect the gaze of the right eye and the left eye, respectively. The gaze detection process can utilize well-known techniques commonly used for eye tracking. For example, in a method using corneal reflex, an infrared LED (Light Emitting Diode) is irradiated onto the face, an image is taken with an infrared camera, and the position on the cornea of ​​the light reflected by the infrared LED (corneal reflex) is used as the reference point, and the gaze is detected based on the position of the pupil relative to the position of the corneal reflex. In this case, an infrared camera and an infrared LED are provided.

[0051] The audio processing device 140 is an audio processor that processes audio, and includes a speaker 141 and a microphone 143.

[0052] The speaker 141 outputs the audio signal processed by the audio signal processing unit to the outside. The audio signal processing unit is an audio signal processor. The HMD 100 uses, for example, headphones.

[0053] The microphone 143 converts the user's voice and the like into audio data and inputs it. For example, an external sound microphone and a vocalization sound microphone may be provided. These microphones collect external sounds and the vocalizations of the user 901, respectively. Headphones may also be connected. The headphones are worn on the ears of the user 901 to listen to the voices directed to the user 901.

[0054] The sensor 150 is a group of sensors for detecting the state of the HMD 100. In this embodiment, the sensor 150 includes a GPS (Global Positioning System) receiving unit 151, a gyro sensor 152, a geomagnetic sensor 153, an acceleration sensor 154, and a depth sensor 155. These sensors detect the position, movement, tilt, direction, etc. of the HMD 100. The depth sensor 155 also acquires distance information from the HMD 100 to an object. Note that other sensors may also be included.

[0055] The acceleration sensor 154 is a sensor that detects acceleration, which is a change in speed per unit time, and can detect movement, vibration, shock, etc. The gyro sensor 152 is a sensor that detects angular velocity in the rotation direction, and can detect the vertical, horizontal, and diagonal posture states. The acceleration sensor 154 and gyro sensor 152 installed in the HMD 100 can be used to detect the movement of the HMD 100.

[0056] The geomagnetic sensor 153 is a sensor that detects the magnetic force of the earth and detects the direction in which the HMD 100 is facing. A three-axis type that detects geomagnetism in the up-down direction in addition to the front-back and left-right directions may be used, and the movement of the HMD 100 may be detected by capturing changes in the geomagnetism in response to the movement of the HMD 100.

[0057] The depth sensor 155 is a sensor that can capture the three-dimensional shape of an object such as a person or an object. It can be a LiDAR (Light Detection and Ranging) sensor that irradiates an object with infrared or other laser light, measures the reflected scattered light, and analyzes and detects the distance to a distant object and the state of the object, a TOF (Time Of Flight) sensor that measures distance by irradiating a subject with pulsed light and measuring the reflection time for each pixel, or a millimeter wave radar that emits millimeter wave radio waves, captures the reflected waves, and detects the distance to the object and the state of the object.

[0058] The communication device 160 is a communication interface that performs wireless communication with other nearby information terminals via short-distance wireless communication, wireless LAN, or base station communication. During wireless communication, data is transmitted and received via a transmitting / receiving antenna.

[0059] The communication device 160 includes a LAN (Local Area Network) communication interface (I / F) 161, a telephone network communication I / F 162, a BT (Bluetooth (registered trademark)) communication I / F 163, and an infrared communication I / F 164.

[0060] The LAN communication I / F 161 is connected to a network 530 such as the Internet via a wireless access point or the like, and transmits and receives data to and from each server on the network 530. The connection to the wireless access point or the like may be via Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0061] The telephone network communication I / F 162 is connected to a communication network using a mobile telephone communication network, and transmits and receives information to and from a server on the communication network. The communication methods used include, for example, third-generation mobile communication systems (hereinafter referred to as "3G") such as the GSM (registered trademark) (Global System for Mobile Communications) method, the W-CDMA (Wideband Code Division Multiple Access) method, the CDMA2000 method, and the UMTS (Universal Mobile Telecommunications System) method, as well as communication methods called the LTE (Long Term Evolution) method, the fourth generation (4G), and the fifth generation (5G).

[0062] The Bluetooth communication I / F 163 is an interface for communicating with an external device according to the Bluetooth standard. The infrared communication I / F 164 is an interface for communicating with an external device by infrared rays. For example, IrDA (Infrared Data Association, registered trademark) is used.

[0063] Note that short-range wireless communication may be performed using Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), or the like. Alternatively, wireless LAN such as Wi-Fi (registered trademark) may be used. Furthermore, an ultra-wideband (UWB) system may also be used.

[0064] The LAN communication I / F 161, the telephone network communication I / F 162, the BT communication I / F 163, and the infrared communication I / F 164 each include an encoding circuit, a decoding circuit, an antenna, and the like.

[0065] The communication device 160 may use other methods, such as optical communication or acoustic communication, as a means of wireless communication. In that case, instead of the transmitting and receiving antennas, a light emitting / receiving unit and an acoustic wave output / input unit are used, respectively.

[0066] The expansion interface unit 105 is a group of interfaces for expanding the functions of the HMD 100, and in this embodiment includes a charging terminal, a video / audio interface, a USB (Universal Serial Bus) interface, a memory interface, etc. The video / audio interface inputs video signals / audio signals from external video / audio output devices, outputs video signals / audio signals to external video / audio input devices, etc. The USB interface connects to USB devices. The memory interface connects to memory cards and other memory media to send and receive data.

[0067] 3 includes components that are not essential to this embodiment, but the effects of this embodiment are not impaired even if these components are not provided. Furthermore, components not shown, such as a digital broadcast reception function and an electronic money payment function, may be further added.

[0068] [HMD functional blocks] Next, a description will be given of the functional configuration of the HMD 100 related to this embodiment. Fig. 4 is a functional block diagram of the HMD 100 of this embodiment. Each function shown in this diagram is realized by the main processor 111 loading a program stored in the internal memory storage into the RAM 113 and executing it.

[0069] 4, the HMD 100 includes a main control unit 210, a virtual remote control generation unit 220, an operation reception unit 230, a command transmission unit 240, and a data storage unit 250. The data storage unit 250 stores an analysis table 251, a control data table 252, a remote control table 253, a gesture action table 254, and processing data 255. The data storage unit 250 is constructed in the internal memory storage or the RAM 113.

[0070] The main control unit 210 controls the overall operation of the HMD 100 .

[0071] The virtual remote control generation unit 220 generates a virtual remote control for the controlled device 510 from the written information 310 on the presentation surface 300. The virtual remote control has a function that allows the user 901 of the HMD 100 to use the written information 310 on the presentation surface 300 as if it were a remote control for the controlled device 510. The virtual remote control generation unit 220 of this embodiment generates a virtual remote control image 400 that presents function buttons of the remote control by superimposing them on the corresponding written information 310 on the presentation surface 300. An operation reception unit 230 (described later) receives an operation instruction from the user 901 on a function button on this virtual remote control image 400, and a command transmission unit 240 transmits a control command to the controlled device 510.

[0072] The virtual remote control generation unit 220 of this embodiment includes an image acquisition unit 221 , a specific information acquisition unit 222 , a control data acquisition unit 223 , an association unit 224 , and a virtual image generation unit 225 .

[0073] The image acquisition unit 221 acquires an image of the presentation surface 300 that serves as the base of the virtual remote control. In this embodiment, the outer camera 133 captures an image of the real space to acquire a surrounding image. The captured real space includes written information 310 on the presentation surface 300.

[0074] The specific information acquisition unit 222 performs a specific information acquisition process. The specific information acquisition process is a process of analyzing the surrounding image acquired by the image acquisition unit 221 and identifying the controlled device, which is the device that the user 901 wants to operate, and the desired control operation, which is the control operation that the user wants to operate. In this embodiment, the written information 310 on the presentation surface 300 is analyzed to identify the controlled device and the desired control operation. The analysis table 251 is used for the identification. The identification results are output as controlled device identification information and desired control operation identification information, respectively. The analysis table 251 and the specific information acquisition process will be described later.

[0075] The control data acquisition unit 223 performs a control data acquisition process. The control data acquisition process is a process for acquiring control data for the controlled device 510 identified by the identification information acquisition unit 222. The control data acquisition unit 223 first determines whether or not the control data for the controlled device 510 is stored in the control data table 252 of its own device (HMD 100). If not stored, the control data acquisition unit 223 checks whether or not there is a connectable controlled device 510 that is not stored in the control data table 252. If there is a controlled device 510 that is not stored in the control data table 252, the control data acquisition unit 223 acquires information about the controlled device 510, such as the device name and model number, from the controlled device 510 and acquires control data from the network server 550. The control data acquired from the network server 550 is stored in the control data table 252 in association with identification information (ID) assigned to each controlled device 510. The control data acquisition process and the control data table 252 will be described later.

[0076] The control data includes possible control operations for each controlled device 510 and control commands for each control operation.

[0077] The association unit 224 performs an association process to associate the desired control operation identified by the identification information acquisition unit 222 with the control command included in the control data. The association result is stored in a remote control table 253. This remote control table 253 is the control data of the virtual remote control desired by the user 901. The association process and the remote control table 253 will be described in detail later.

[0078] The virtual image generation unit 225 generates a virtual remote control image 400 that functions as a virtual remote control, and displays it on the display 131. The display position is a position that coincides with the presentation surface 300 in the line of sight of the user 901. The virtual image generation unit 225 generates the virtual remote control image 400 using the remote control table 253.

[0079] An example of the generated virtual remote controller image 400 is shown in Fig. 5. As shown in this figure, the virtual remote controller image 400 includes a controlled device information display area 410 and a desired control operation display area 420.

[0080] In the controlled device information display area 410, a controlled device name 411 is displayed as information for identifying the controlled device identified by the identification information acquisition unit 222. In the example shown in the figure, "air conditioner" is displayed.

[0081] The desired control operation display area 420 displays reception objects 421 and 422 of function buttons that receive instructions from the user 901 for the desired control operation identified by the identification information acquisition unit 222. The reception objects 421 and 422 are created for each desired control operation. The display positions of the reception objects 421 and 422 are positions on the presentation surface 300 where the information corresponding to the desired control operation is written in the written information 310.

[0082] The operation reception unit 230 receives an operation instruction from the user 901 to the virtual remote controller (virtual remote controller image 400). In this embodiment, for example, when an operation instruction to the reception objects 421 and 422 is detected, the operation instruction is received as an operation instruction for a desired control operation corresponding to the reception objects 421 and 422.

[0083] Operation instructions are given, for example, by gesture movements or eye gaze movements. The operation reception unit 230 analyzes these operations by referring to the gesture movement table 254, etc., and detects the operation instructions. If a predetermined movement is detected, it is determined that the reception object 421, 422 has been selected. Then, the desired control operation associated with the reception object 421, 422 is identified.

[0084] The operation receiving unit 230 detects the gesture operation of the user 901 from the sensor 150 provided in the HMD 100 or hand movements extracted from an image captured by the outer camera 133. Also, the gaze operation is detected using the function of the gaze detection sensor.

[0085] The command transmitting unit 240 transmits a control command associated with the specified desired control operation to the controlled device. In this embodiment, the control command is transmitted to the controlled device 510 via the communication device 160 and the wireless router 520. The control command is obtained from the remote control table 253.

[0086] [Analysis Table] Here, we will explain an example of the analysis table 251. The analysis table 251 is used when the specific information acquisition unit 222 analyzes the written information 310. This is a table in which analysis results are registered in association with character strings, figures, colors, etc. (hereinafter referred to as character strings, etc.) that may be described in the written information 310.

[0087] 6 is a diagram showing an example of the analysis table 251 of this embodiment. As shown in the figure, the analysis table 251 registers, for each description 251a, an analysis content 251b and a description type 251c in association with each other.

[0088] The written content 251a stores, for example, candidates for character strings or the like that the user 901 may write on the presentation surface 300 as written information 310. For example, names of controlled devices 510 such as air conditioners, coolers, lighting fixtures, lights, lamps, televisions, etc., and names of desired control operations such as cooling, heating, dehumidifying, fanning, stopping, turning on, turning off, etc. are stored. Here, various different names and descriptions are stored for the names of the controlled devices 510 and the names of the desired control operations. These are character strings or the like that the user 901 may write.

[0089] Analysis content 251b stores analysis content of the name of controlled device 510 or the name of desired control operation written in description content 251a. Analysis content is information that uniquely identifies character strings or the like that may be written by many users 901.

[0090] In this embodiment, for example, if the description 251a is the name of the controlled device 510, the device name that identifies the controlled device 510 is stored. Specifically, if the description 251a is "air conditioner," "cooler," or the like, "air conditioner device" is stored. Also, if the description 251a is "lighting fixture," "light," or "lamp," "lighting device" is stored.

[0091] Furthermore, if the content 251a is the name of a desired control operation, information specifying the desired control operation is stored. Specifically, if the content 251a is "cooling" or the like, "cooling function on" is stored, and if it is "stop" or the like, "stop" is stored.

[0092] The description type 251c stores the type of what the description content 251a represents. In this embodiment, either the controlled device 510 or the desired control operation is registered.

[0093] The specific information acquisition unit 222 analyzes the written information 310 on the presentation surface 300 and extracts character strings and the like. Then, it searches for and detects information matching the extracted character strings and the like in the written content 251a of the analysis table 251. Then, it extracts the analysis content 251b and the description type 251c registered correspondingly. Then, if the description type 251c indicates a controlled device, it outputs the analysis content 251b as controlled device identification information. On the other hand, if the description type 251c indicates a desired control operation, it outputs the analysis content 251b as desired control operation identification information.

[0094] Here, a specific example will be described in which the specific information acquisition unit 222 analyzes the written information 310, using the analysis table 251 as an example. The specific information acquisition unit 222 collates the content written as written information 310 on the presentation surface 300 with the written content 251a, and sets the analysis content 251b and the description type 251c of the data that matches as the analysis results.

[0095] For example, if the description information 310 contains character strings such as "air conditioner" or "cooler," the analysis results for the analysis content 251b and description type 251c are "air conditioner device" and "controlled device," respectively.

[0096] Furthermore, if the written information 310 contains "ray", "ray", "air conditioning", "cooling" or the like, the analysis results will be "air conditioning function on" and "desired control operation".

[0097] If the written information 310 is "Teishi," "Tei," "Stop," "Stop," or the like, the analysis results are "Stop" and "Desired control operation."

[0098] If "Danbou", "Dan", "heating", "warm" or the like is written as the written information 310, "Danbou function on" and "desired control operation" are taken as the analysis results.

[0099] When "dehumidifying function on", "dehumidifying function on", "dehumidifying function on", "desired control operation", etc. are written as the written information 310, the analysis results are "dehumidifying function on", "desired control operation".

[0100] If the written information 310 contains "electricity," "electrical," "lighting," "showmei," "light," etc., the analysis results will be "lighting equipment" and "controlled equipment."

[0101] The analysis table 251 is an example, and explanations of other character strings not listed and the analysis contents for those character strings will be omitted here. Also, each character string is merely an example. As described above, the written information 310 can also use figures (illustrations), pictograms (snow marks, flame marks, etc.), symbols, etc. in addition to character strings.

[0102] If the information described in the description information 310 is not registered in the analysis table 251, the specific information acquisition unit 222 newly registers the result of analyzing the description information 310 (such as a character string) in the description content 251a. In this case, the corresponding analysis content 251b and description type 251c are registered by the user 901.

[0103] [Control Data Table] The control data table 252 stores control data for the controlled device 510. Fig. 7(a) is a diagram showing an example of the control data table 252 of this embodiment. As shown in the figure, the control data table 252 stores, in association with a controlled device name 252a, its identification information (ID) 252b, control data 252c for the controlled device 510, and detailed information 252d for the controlled device 510.

[0104] The ID 252b may be any information that uniquely determines the control data of the controlled device 510. For example, a model number set by the manufacturer may be used. The control data 252c is a set of control commands for each possible control operation of the controlled device. The detailed information 252d is, for example, a name, a manufacturer name, a physical address, etc. The detailed information 252d is used for connecting the HMD 100 and the controlled device 510, etc. Note that the detailed information 252d may be stored as a sub-table linked to the ID 252b, rather than in the same control data table 252.

[0105] The above information about the controlled devices 510 that the user 901 has previously connected to and the controlled devices 510 that the user 901 may use is registered in advance in the control data table 252. Alternatively, the control data acquisition unit 223 may acquire the information from the network server 550 and register it.

[0106] [Remote control table] Next, a description will be given of the remote control table 253. The remote control table 253 is a table that stores information for identifying the controlled device 510 targeted by the virtual remote control and a control command for each desired control operation.

[0107] 7(b) is a diagram showing an example of the remote control table 253 of this embodiment. As shown in this figure, the remote control table 253 stores a description type 253a, analysis content 253b, identification information (ID) 253c, control command 253d, area 253e, and display data 253f. The remote control table 253 may also store a creation date and time 253g.

[0108] The description type 253a is the type (controlled device or desired control operation) of the description information 310. In this embodiment, the description type 251c obtained as a result of analysis by the specific information acquisition unit 222 using the analysis table 251 is stored.

[0109] The analysis content 253b is the result of the specific information acquisition unit 222 analyzing the description information 310. In this embodiment, the analysis content 251b of the analysis table 251 is stored.

[0110] When the description type 253a is a controlled device, an ID that uniquely identifies the controlled device is stored in the ID 253c. The ID of the controlled device 510 is acquired by the identification information acquisition unit 222 from the controlled device itself or a network, etc., based on the analysis content 251b of the controlled device.

[0111] When the description type 253a is a desired control operation, the control command 253d is a control command corresponding to the control operation, and is acquired by the association unit 224 from the control data table 252 and stored.

[0112] The display data 253f is data to be displayed on the virtual remote control image 400. The area 253e is information for specifying the display area of ​​the virtual remote control image 400. In this embodiment, the area 253e stores the results of analysis performed by the specification information acquisition unit 222. Note that the area 253e stores, for example, information specified by the pixel position of the image acquired by the HMD 100.

[0113] The virtual image generating unit 225 generates reception objects for a group of function buttons by regarding the handwritten character strings written on the presentation surface 300 as function buttons.

[0114] For example, when generating the virtual remote control image 400, the virtual image generation unit 225 displays the information registered in the display data 253f of the data in which the description type 253a is registered as "controlled device" in the controlled device information display area 410. In the case of Fig. 7(b), the information is "air conditioner."

[0115] In addition, the virtual image generation unit 225 displays display data 253f of all data for which "desired control operation" is registered in the description type 253a in the desired control operation display area 420 of the virtual remote control image 400. In the case of Fig. 7(b), these are "cooling" and "stop."

[0116] The main control unit 210 erases the display of the virtual remote control image 400 after a predetermined period has elapsed or upon receiving an instruction from the user 901. At this time, the remote control table 253 is also erased. This makes it possible to avoid unnecessary memory consumption. The predetermined period may be, for example, one minute, one hour, the entire registered day, one week, etc.

[0117] Furthermore, after the display of the virtual remote control image 400 is erased, the remote control table 253 may be erased after a separately determined fixed time has elapsed. In this case, even if the display of the virtual remote control image 400 has been erased, the virtual remote control image 400 is displayed again and operations are accepted when the user 901 looks at the controlled device 510, as long as the remote control table 253 has not yet been erased. Note that the remote control table 253 may be configured to be erased when the virtual remote control generation process for the next controlled device 510 is started.

[0118] [Gesture Action Table] Next, a description will be given of an example of the gesture action table 254. The gesture action table 254 is a table in which operation instructions are registered in association with each gesture of the user 901. The gesture action table 254 is used when identifying an operation instruction from the gesture action of the user 901 detected by the operation receiving unit 230.

[0119] 8 is a diagram showing an example of the gesture action table 254 of this embodiment. As shown in this diagram, the gesture action table 254 includes a gesture action 254a and an operation instruction 254b.

[0120] The gesture action 254a is a gesture of the user 901 that the operation receiving unit 230 may detect.

[0121] The operation instruction 254b stores the content of the operation instruction corresponding to each gesture. The content of the operation instruction is determined in advance. The user 901 may register a desired operation instruction.

[0122] For example, when the operation receiving unit 230 detects, as a gesture action of the user 901, an action of touching a function button that is a reception object 421, 422, the operation receiving unit 230 determines that this is an instruction to execute the corresponding function (here, a control operation), and acquires a control command associated with the execution instruction from the remote control table 253. The command sending unit 240 sends the control command to the controlled device 510.

[0123] Additionally, as shown in the figure, for example, when the operation receiving unit 230 detects a gesture of holding a presented function button and sliding it within the display area of ​​the group of function buttons, the operation receiving unit 230 determines that this is an instruction to move the function button. Furthermore, when the operation receiving unit 230 detects a gesture of holding a presented function button and sliding it out of the function button display area, the operation receiving unit 230 determines that this is an instruction to erase the function button. Furthermore, when the operation receiving unit 230 detects a gesture of drawing an arc that surrounds the entire group of presented function buttons, the operation receiving unit 230 determines that this is an instruction to select the function button surrounded by the arc. Furthermore, when the operation receiving unit 230 detects a gesture of the user 901 moving their hand sideways, the operation receiving unit 230 determines that this is an instruction to end the function button display.

[0124] Each gesture operation is merely an example, and naturally, other gesture operations can also be valid. For example, as described above, the line of sight of the user 901 may be detected by the in-camera 134, and a function button may be selected.

[0125] By providing this gesture action table 254, the user 901 can freely set the layout of the function buttons.

[0126] Other data required for processing, data generated during processing, and data generated as a result of processing are stored in processing data 255.

[0127] Using these functions and data, the user 901 performs the remote control process described with reference to Fig. 1. That is, the user 901 wearing the HMD 100 can obtain an operating environment as if the presentation surface 300 were a remote control, simply by viewing the presentation surface 300 via the display 131 of the HMD 100.

[0128] [Remote Control Processing] Next, the flow of remote control processing including virtual remote control generation processing by the HMD 100 of this embodiment will be described. Fig. 9 shows the processing flow of the remote control processing of this embodiment. This processing is started in response to an instruction from the user 901 or when the HMD 100 detects the presentation surface 300. Note that here, an example will be described in which data is deleted after a predetermined period of time has passed since the virtual remote control image 400 was created.

[0129] First, the image acquisition unit 221 acquires a surrounding image (step S1101).

[0130] The specific information acquisition unit 222 performs a specific information acquisition process to acquire specific information (controlled device specific information and desired control operation specific information) from the surrounding image (step S1102).

[0131] The control data acquisition unit 223 performs a control data acquisition process to acquire the control data of the controlled device 510 based on the controlled device identification information (step S1103). If the control data is stored in the storage device 110 of the controlled device, the control data is acquired from the storage device 110; if not, the control data is acquired from the network server 550 via the communication device 160.

[0132] The control data acquisition unit 223 determines whether or not control information relating to the device identified as the controlled device in step S1103 has been acquired (step S1104).

[0133] If control data can be obtained from either the device itself or the network server 550 (S1104; Yes), the association unit 224 performs association processing to complete the remote control table 253 (step S1105). Here, a control command is associated with a desired control operation.

[0134] The virtual image generation unit 225 references the remote control table 253 and generates a virtual remote control image (step S1106). Here, the virtual image generation unit 225 creates display data including reception objects 421, 422 of the function buttons associated with the control commands and a display of the name of the controlled device 510. The display data to be created is generated so that it is respectively arranged in the corresponding areas of the surrounding image acquired by the image acquisition unit 221. For example, as shown in FIG. 1, the display data is generated so that the name of the controlled device is displayed in the area on the presentation surface 300 where "air conditioner" is written, a function button for receiving an instruction to "cool" is displayed in the area where "cooling" is written, and a function button for receiving an instruction to "stop" is displayed in the area where "stop" is written.

[0135] If the display 131 is a transparent display, the virtual remote control image does not need to be displayed. However, area information of the function buttons (reception objects 421, 422) is retained, and instructions from the user 901 are accepted. For example, the operation acceptance unit 230 accepts operation instructions from the user 901 by retaining information on the arrangement positions of the reception objects 421, 422 and the corresponding control operations.

[0136] Here, main control unit 210 causes timer 106 to start counting (step S1107). Then, it is determined whether or not timer value tm has exceeded a predetermined period Th (tm≦Th) (step S1108), and if it has exceeded period Th (tm>Th), main control unit 210 erases remote control table 253 (step S1111) and ends the process.

[0137] On the other hand, if it is within the period (tm≦Th), when the operation receiving unit 230 receives an operation instruction for the desired control operation from the user 901 (step S1109), the operation receiving unit 230 extracts a control command for the received control operation from the remote control table 253.

[0138] Command transmitting section 240 transmits the extracted control command to controlled device 510 via communication device 160 (step S1110), and the process returns to step S1108.

[0139] If the control data could not be acquired in step S1104 (S1104; No), main control unit 210 performs a display process (NG display process) to indicate that the data could not be acquired (step S1112) and ends the process. In the NG display process, for example, a message indicating that the control data for specified controlled device 510 could not be acquired is displayed on display 131, or a message indicating that a virtual remote control cannot be created is displayed on display 131.

[0140] [Specific information acquisition process] Next, a description will be given of the flow of the specific information acquisition process performed by the specific information acquisition unit 222. Fig. 10 shows the processing flow of the specific information acquisition process of this embodiment. This process analyzes the surrounding image and completes the description type 253a, analysis content 253b, and ID 253c of the remote control table 253.

[0141] The specific information acquisition unit 222 first performs a presentation surface shape recognition process (step S1201). This presentation surface shape recognition process is a process for recognizing the shape of the presentation surface 300. Specifically, the surrounding image captured by the outer camera 133 is analyzed to detect a predetermined shape of the presentation surface 300. In this embodiment, an outer frame 305 is detected. The area inside the detected outer frame 305 is set as a scanning area for detecting the name of the controlled device 510 and information on the desired control operation. As described above, the name of the controlled device and the desired control operation are written on the presentation surface 300 as written information 310.

[0142] The shape of the outer frame 305 is, for example, a rectangle. However, the shape is not limited to this. The shape does not have to be an exact figure. For example, when a rectangle is handwritten, the lines may be shaky or the corners may be slightly rounded, so the rectangle is determined based on how rectangular it appears. Furthermore, the presentation surface 300 may be determined based on marks and characters drawn within the rectangular outer frame 305. For example, the presentation surface 300 may be determined based on the characteristics of the mark, such as "remote control," a mark with the character "R" circled, or a user's favorite mark.

[0143] Next, the specific information acquisition unit 222 determines whether or not there is written information 310 that may correspond to controlled device information or desired control operation information in the scanning area of ​​the presentation surface 300 (step S1202). The presence or absence of information is determined by, for example, binarizing the scanning area and performing a simple image analysis process such as determining whether or not there is a continuous black area of ​​a predetermined size or larger.

[0144] Here, if it is determined that there is no written information 310 (S1202; No), an NG display process is performed (step S1211), and the process ends. In the NG display process, for example, a message indicating that the written information 310 cannot be detected is output. Furthermore, when determining whether or not the surface is the presentation surface 300, the degree of likelihood of the surface being the presentation surface 300 may be calculated, and if the degree is equal to or greater than a certain threshold, a message may be displayed.

[0145] If it is determined that the described information 310 is present (S1202; Yes), the specific information acquisition unit 222 performs a presentation surface analysis process to analyze the scanning area in detail (step S1203). In the presentation surface analysis process, the described information 310 is analyzed to extract controlled device information candidates and desired control operation information candidates.

[0146] Controlled device information candidates and desired control operation information candidates are extracted by comparing the analysis results of description information 310 with description contents 251a registered in analysis table 251. For example, in the example of Fig. 1, "air conditioner" is extracted as a controlled device information candidate, and "cooling" and "stop" are extracted as desired control operation candidates. At this time, analysis contents 251b corresponding to each of the controlled device information candidate and the desired control operation candidate are determined.

[0147] Next, the specific information acquisition unit 222 performs ID acquisition processing to acquire identification information (ID) of the controlled device 510 from the controlled device information candidates (step S1204). Here, the ID is information that uniquely identifies the controlled device 510, such as a model number. For example, the ID may be detailed device information such as the manufacturer name, device type, physical address, etc. of the controlled device 510.

[0148] The specific information acquisition unit 222, for example, compares the controlled device information candidates with the analysis table 251 to select the type of device. The type of device is, for example, information registered in the analysis content 251b of the analysis table 251. Then, the identification information (ID) 252b of all records in which the selected device type is registered as the controlled device name 252a may be acquired from the control data table 252, and a connection establishment request may be transmitted to the surrounding area using the detailed information 252d corresponding to the acquired identification information (ID) 252b, and it may be determined whether or not a response is obtained. As a result, responses are received only from devices of the corresponding type.

[0149] The ID acquisition process makes it possible to determine whether or not there is a controlled device 510 in the vicinity to which a control command can be transmitted using the virtual remote controller. That is, the specific information acquisition unit 222 determines whether or not there is a response within a predetermined period of time (step S1205).

[0150] If a response is received within the predetermined period (S1205; Yes), the identification information acquisition unit 222 outputs the extracted and identified analysis content 251b, its description type 251c, and ID 252b to the control data acquisition unit 223. The identification information acquisition unit 222 further registers the extracted and identified information in the remote control table 253 (step S1206) and terminates the process. The information to be registered here is the description type 253a, analysis content 253b, ID 253c, and area 253e and display data 253f obtained in the presentation surface analysis process.

[0151] On the other hand, if no response is received (S1205; No), the process proceeds to step S1211, where NG display processing is performed, and the process ends. In this case, the NG display processing outputs a message indicating that there is no controlled device 510 in the vicinity to which an operation command can be sent using the virtual remote controller.

[0152] Note that a selection rule is determined in advance when responses are received from multiple controlled devices 510. The rule may be, for example, to select the device that sent the response earliest, to select the device in the line of sight of user 901, to present the selection to user 901, or to accept the selection.

[0153] Furthermore, if no response is received from the desired device, the specific information acquisition unit 222 may search for a connectable controlled device 510. At this time, if a new controlled device 510 that is not registered in the control data table 252 is detected, detailed information of the new controlled device 510 is acquired and registered in the control data table 252. Then, it is checked whether there is any device among the new controlled devices 510 that corresponds to the controlled device information candidates.

[0154] Furthermore, for example, if the presence of a new controlled device 510 is clear, the user 901 may acquire ID 252b by photographing the manufacturer name and model number of the controlled device 510 with the outer camera 133. Alternatively, the user 901 may photograph the external shape of the controlled device 510 and send an inquiry to the network server 550 along with the photographed image. Alternatively, the user 901 may acquire ID 252b from the homepage of the manufacturer or the like. In this case, the user 901 may handwrite some of the detailed information on the presentation surface 300 and acquire it as photographed data. Furthermore, the user 901 may vocally utter some of the detailed information and acquire it with the voice processing device 140.

[0155] In this case as well, an ID is assigned to each controlled device 510 registered in the control data table 252 at the time of registration.

[0156] It should be noted that the user 901 does not need to accurately write the name of the controlled device 510 on the presentation surface 300 as the information about the controlled device 510. For example, other characters such as "air conditioner" or "cooler" or a picture that can be identified as an air conditioner may be used. In this embodiment, various expected expressions that the user 901 may write are registered in advance in the analysis table 251. Furthermore, AI (artificial intelligence) may be used to analyze the content that the user 901 has written on the presentation surface 300 as the written information 310.

[0157] [Control data acquisition process] Next, a description will be given of the control data acquisition process performed by the control data acquisition unit 223. Fig. 11 shows the processing flow of the control data acquisition process of this embodiment. This process is a process for acquiring control data of the controlled device 510 identified in the identification information acquisition process.

[0158] The control data acquisition unit 223 searches the control data table 252 to determine whether or not control data for the controlled device 510 is registered based on the controlled device identification information including the ID of the controlled device 510 identified by the identification information acquisition unit 222 (step S1301). Here, the control data table 252 is searched based on the ID to determine whether or not data having a matching ID 252b is registered.

[0159] If matching data is found and control data is registered (S1301; Yes), the control data acquisition unit 223 performs a control command extraction process (step S1302). Specifically, the control data of the matching data is referenced, and a control command associated with the analysis content of the desired control operation is extracted for each desired control operation. Note that, in the specific information acquisition process, if the specific information acquisition unit 242 acquires the ID of the controlled device 510 from the ID control data table 252, the control command may be extracted from the control data 252c associated with the acquired ID 252b without searching the control data table 252 again.

[0160] Then, the control data acquisition unit 223 registers each of the extracted control commands in the remote control table 253 in association with the desired control operation (step S1303), and ends the process.

[0161] If there is no matching data (S1301; No), the control data acquisition unit 223 requests the control data corresponding to the ID from the network server 550 (step S1304), and then determines whether or not the return control data has been received (step S1305).

[0162] If return control data is received (S1305; Yes), the control data acquisition unit 223 associates the received control data with the controlled device name and ID, and registers the data in the control data table 252 (step S1306), and proceeds to step S1302.

[0163] In step S1305, if a response is received indicating that the control data is not available despite the request to the network server 550, or if there is no response within a predetermined period of time (S1305; No), the control data acquisition unit 223 performs NG display processing (step S1311) and ends the processing. Here, a message indicating that the control data of the controlled device 510 cannot be obtained is output.

[0164] As described above, the HMD 100 of this embodiment includes the communication device 160, and transmits a control command to the controlled device 510 via the communication device 160 to remotely control the controlled device 510. The HMD 100 also includes a virtual remote control generation unit 220 that identifies the controlled device 510 and a desired control operation that is a desired control operation from a surrounding image that is an image of the surroundings, and generates reception objects 421 and 422 that receive an operation instruction from the user 901 for the identified desired control operation, an operation reception unit 230 that receives the operation instruction from the user 901 via the reception objects 421 and 422, and a command transmission unit 240 that transmits a control command corresponding to the operation instruction received by the operation reception unit 230 to the controlled device 510.

[0165] The surrounding image also includes a presentation surface 300 having a notation indicating the desired control operation. The virtual remote control generation unit 220 includes: an identification information acquisition unit 222 that acquires, from the surrounding image, controlled device identification information that identifies the controlled device 510 and desired control operation identification information that identifies the desired control operation; a control data acquisition unit 223 that acquires control data for the controlled device using the controlled device identification information; an association unit 224 that associates a control command for controlling the controlled device 510, which is included in the control data, with the desired control operation; and a virtual image generation unit 225 that generates reception objects 421, 422 and generates a virtual remote control image 400 in which the reception objects 421, 422 are arranged in an area where the notation indicating the desired control operation is displayed when the surrounding image is displayed on the display 131.

[0166] As described above, according to this embodiment, the surface (presentation surface 300) for presenting remote control function buttons is imaged by the outer camera 133 mounted on the HMD 100, the controlled device 510 to be operated and its function (desired control operation) are identified from the figures and characters drawn on the presentation surface 300, and the function button of the remote control for controlling the controlled device 510 is presented as an AR object (function button image, etc.) at the location where the function is drawn. At this time, the AR object and characters, images, etc. printed or written on the presentation surface 300 are presented superimposed on the presentation surface 300.

[0167] Furthermore, the HMD 100 receives operations such as selection, movement, and deletion on the function buttons by the user 901 through gestures or the like made by the user 901 on the function buttons, and controls the controlled device.

[0168] According to this embodiment, even if user 901 loses the remote control, a virtual remote control can be generated at any time, and this virtual remote control can display only the functions required by user 901 as function buttons. Furthermore, presentation surface 300, which is the basis of the virtual remote control, can be drawn on paper or the like, and can be pasted on a refrigerator or bound together like a notebook and reused as needed. This allows even forgetful user 901 to easily control controlled device 510 without having to search for the remote control.

[0169] That is, according to this embodiment, the user 901 can select a function that the user 901 desires, and can freely set and generate a virtual remote controller on which function buttons are arranged to realize the selected function.

[0170] According to this embodiment, even if the user 901 does not have a remote controller at hand, he / she can write the necessary operations by hand on a piece of paper or the like, and this will function as a remote controller. This means that the user 901 does not need to search for a remote controller. Also, since only the necessary functions can be created as function buttons, there is no hassle. Furthermore, the layout of the buttons can be chosen as desired, making it easy to use.

[0171] As described above, according to this embodiment, the hassle of searching for a remote control when remotely controlling a device is eliminated, and convenience for the user 901 is significantly improved.

[0172] Furthermore, in this embodiment, for the controlled device 510 for which a virtual remote control has been generated at least once, the control data table 252 is stored in the data storage unit of the HMD 100. By using this stored control data, control of the controlled device 510 can be performed from the next time onwards without going through the network 530. As a result, problems in the network 530 (such as delays due to network line congestion or line disconnections) can be avoided.

[0173] On the other hand, in this embodiment, the remote control table 253 is deleted when it is no longer in use, and is created anew the next time it is needed. The user 901 does not need to search for the virtual remote control in the data storage unit 250 of the HMD 100. Also, unnecessary memory consumption can be reduced.

[0174] <Variation 1> In the above embodiment, if the desired control operation written by the user 901 on the presentation surface 300 includes a function that is not available in the controlled device, the absence of that function may be clearly indicated.

[0175] An example of a virtual remote control image in this case is shown in Fig. 12(b) Here, an example is shown in which the user 901 writes on the presentation surface 300 as shown in Fig. 12(a).

[0176] As shown in FIG. 12( a ), the user 901 enters “air conditioner” as the controlled device name 311 , and enters “rayon,” “tei,” and “danbou” as the desired control operation 312 .

[0177] If the controlled device 510 acquired by the specific information acquisition unit 222 does not have a heating function, the control data acquired by the control data acquisition unit 223 does not contain a control command corresponding to heating. The control data acquisition unit 223 notifies the virtual image generation unit 225 of this.

[0178] For the control operations of "Ray Bow" and "Teishi," the virtual image generation unit 225 generates and displays function buttons in association with the control commands using the method of the above embodiment. However, for "Danbou," in response to a notification from the control data acquisition unit 223, an unavailable display 431 is displayed indicating that the character cannot be used.

[0179] 12(b), since the "Danbou" function is not available, an AR object with an X mark is created as a "disabled" display 431 on the virtual remote control image 400, and is displayed superimposed on the "Danbou" reception object 423. This allows the user 901 to intuitively understand that the "Danbou" function is unavailable.

[0180] Similarly, in the above embodiment, if the control data for the controlled device 510 does not exist, a message to that effect is output, but this is not limiting. For example, as shown in Fig. 12(c), an "unavailable" message 431 may be displayed above the controlled device name 411 on the virtual remote controller to notify the user 901.

[0181] In this modification, an AR object with an X mark is superimposed and presented, but the "unavailable" display 431 is not limited to this. For example, the AR object may be a character string such as "There is no corresponding heating function" or "There is no operable air conditioner." Furthermore, the character string may be output as sound from the speaker 141.

[0182] This modification can reduce unnecessary operations by the user.

[0183] <Variation 2> In the above embodiment, the controlled device 510 has been described as an air conditioner, but the controlled device 510 is not limited to an air conditioner and may be, for example, a lighting fixture.

[0184] 13(a) shows an example of a presentation surface 301 created by the user 901 when the controlled device 510 is a lighting fixture, and Fig. 13(b) shows an example of a virtual remote control image 401 created by the HMD 100 by analyzing the presentation surface 301. As shown in this figure, from the description information 310, the virtual remote control image 401 is generated, displaying "lighting" as the controlled device name 411 and "on" and "off" as the reception objects 421 and 422, respectively.

[0185] <Variation 3> Furthermore, in the above embodiment, the control command is transmitted from the HMD 100 to the controlled device 510 via the network 530. However, the transmission of the control command from the HMD 100 to the controlled device 510 is not limited to this.

[0186] 14, a control command may be transmitted directly from the HMD 100 to the controlled device 510 using a short-range communication I / F such as infrared communication or Bluetooth. In this case, the command transmission unit 240 transmits a control command corresponding to a control operation selected by the user 901 via the Bluetooth communication I / F 163 and the infrared communication I / F 164.

[0187] According to this modification, the controlled device 510 can be controlled by the HMD 100 using the short-range communication I / F without going through the network 530. As a result, there is no need to consider problems in the network 530 (such as delays due to network line congestion or line disconnections).

[0188] The user 901 may be allowed to select which communication means to use to transmit a control command from the HMD 100 to the controlled device 510.

[0189] <Variation 4> In the above embodiment, the control data of the controlled device is registered in advance in the HMD 100 or acquired from the network server 550. However, the control data may be acquired from the controlled device 510 itself. In this case, the control data acquisition unit 223 accesses the controlled device 510 via any one of the LAN communication I / F 161, the BT communication I / F 163, and the infrared communication I / F 164, and acquires the control data from the controlled device 510.

[0190] In particular, when control information is acquired via the infrared communication I / F 164, an infrared pattern (data format) generated from a general remote control device for each control operation may be acquired instead of a control command.

[0191] Then, command transmitting unit 240 transmits an infrared pattern associated with the operation of user 901 to controlled device 510. This makes it possible to output an infrared pattern that is substantially the same as the infrared pattern generated by a normal remote control, thereby enabling instructions to be transmitted with high precision.

[0192] There are several infrared data formats that can be used with remote controls, such as the NEC format, the Japan Electric Appliances Association format, and the Sony format. Needless to say, in this modification, the infrared data format adopted by controlled device 510 is used.

[0193] Furthermore, pairing must be performed in advance when communicating by Bluetooth via the BT communication I / F 163. If pairing fails, the user 901 is notified of this.

[0194] <Variation 5> In the above embodiment, the HMD 100 has been described as an example of the information processing terminal used by the user 901, but the information processing terminal to be used is not limited to this. For example, a portable information terminal such as a smartphone (hereinafter referred to as a smart phone) or a tablet terminal may be used.

[0195] Figure 15 shows an overview of the virtual remote control processing and a display example when using a smartphone.

[0196] As shown in this figure, a user 901 holds up a smartphone 101 between himself and the presentation surface 300, captures an image of the area of ​​the presentation surface 300 that includes written information 310, and obtains a surrounding image.

[0197] The smartphone 101 analyzes the surrounding image, identifies the controlled device 510 and the desired control operation, and associates a control command or an infrared format with it. Then, it creates a virtual remote control image 400 and displays it on the display 131 of the smartphone 101. Then, upon receiving an operation instruction from the user 901 via the virtual remote control image 400, the smartphone 101 transmits a control command (infrared format) corresponding to the operation instruction to the controlled device 510.

[0198] 15, the generated virtual remote control image 400 includes a controlled device information display area 410 and a desired control operation display area 420. In the example of this figure, reception objects 421 and 422 are displayed: "Cooling" for receiving an instruction to start cooling, and "Stop" for receiving an instruction to stop cooling.

[0199] The hardware configuration of the smartphone 101 that realizes this is shown in Fig. 15. The hardware configuration of the smartphone 101 basically has the same configuration as the hardware of the HMD 100. However, the smartphone 101 is not worn and used like the HMD 100, but is held in the hand of the user 901. For this reason, the operation reception device 120 is partially different.

[0200] The smartphone 101 includes an operation reception device 120 that includes not only a button switch 121 and a touch panel 122 but also an operation key 123 .

[0201] The functional configuration of the smartphone 101 for realizing the above embodiment is basically the same as that of the HMD 100. However, the operation reception unit 230 of the smartphone 101 does not need to detect and analyze gesture movements to identify the user 901. The intention of the user 901 is received by detecting a touch operation on the function reception button display area of ​​the touch panel 122.

[0202] This modification can be realized not only on smartphones and tablets, but also on any device with an equivalent or similar hardware and software configuration, such as a laptop PC or tablet PC.

[0203] <Variation 6> In the above embodiment, the presentation surface 300 has been described as being, for example, a sheet of paper. However, the presentation surface 300 is not limited to this. For example, it may be a whiteboard or the like. As long as the user 901 can freely write on it and take a photograph, the material and shape are not important.

[0204] Presentation surface 300 may also be provided by the manufacturer of controlled device 510. For example, presentation surface 300 may be a description of a remote control in a manual. Furthermore, presentation surface 300 may be the remote control itself that is provided in controlled device 510.

[0205] If the description information 310 is prepared in advance, that is, if a manual or a remote control itself that was not created by the user 901 is used on the presentation surface 300, when creating a virtual remote control, it is not necessary to generate all of the reception objects for the function buttons.

[0206] In this case, the identification information acquisition unit 222 analyzes the image on the presentation surface 300, identifies the control operations, and then presents them to the user 901, and accepts the selection of the control operations to be displayed on the virtual remote control image 400. Then, the control operation whose selection has been accepted is identified as the desired control operation.

[0207] As a result, even if ready-made information is used as the presentation surface 300, only the functions desired by the user 901 can be displayed on the virtual remote controller.

[0208] <Variation 7> In the above embodiment, the identification information acquisition unit 222 analyzes the written information 310 to identify characters. At this time, the analysis may use an existing character analysis function such as an OCR (Optical Character Recognition) function. However, the present invention is not limited to this. For example, the user 901 may register handwritten characters in advance, and the registered characters may be analyzed using a method such as pattern matching by referring to the registration.

[0209] Also, only when the characters handwritten by the user 901 cannot be analyzed by the OCR function or the like, the system may be configured to have the user 901 input digital data, and the handwritten characters and the digital data are associated and registered. In this case, the next time presentation surface information is acquired, pattern matching processing is performed using the registered handwritten characters as a pattern to identify the characters.

[0210] <Variation 8> In the above embodiment, the description information 310 is basically a character string. However, the description information 310 is not limited to a character string. It may be a figure, a color, or the like. In this case, the figure, color, or the like is registered in advance in the description content 251a in the analysis table 251.

[0211] The registration may be performed by the user 901. The user 901 registers in advance his / her favorite shapes and colors as those representing a specific controlled device 510 or a specific desired control operation. This allows the user 901 to generate a virtual remote controller without even writing any characters.

[0212] Furthermore, instead of the description information 310, voice or the like may be used. In this case, the voice is registered in the analysis table 251. That is, instead of the description content 251a, or in addition, voice data is registered. Then, analysis content 251b and description type 251c are registered in association with the voice data.

[0213] <Variation 9> Furthermore, in the above embodiment, when the virtual remote control image is displayed, the display of the virtual remote control image 400 is erased after a predetermined time has elapsed. However, this is not limiting. For example, the display of the virtual remote control image 400 may be erased in response to an explicit instruction from the user 901.

[0214] Furthermore, not only the display of the virtual remote control image 400 but also the remote control table 253 that is the basis for generating the virtual remote control image 400 may be configured to be erased in synchronization with the timing at which the display of the virtual remote control image 400 is erased.

[0215] <Modification 10> If information on the layout of the function buttons of the controlled device 510 (layout of the function buttons of the remote control) can be obtained from the manufacturer's homepage or the like via the network 530, the layout information of the function buttons of the controlled device 510 may be further stored in the data storage unit 250. Then, the virtual image generation unit 225 may generate the virtual remote control image 400 according to the layout of the function buttons of the controlled device 510, rather than the layout of the presentation surface 300.

[0216] Note that the configurations that realize the technology of the present invention are not limited to the above-described embodiments and modifications, and various modifications are possible. For example, it is possible to replace part of the configuration of one embodiment or modification with the configuration of another example. It is also possible to add the configuration of another example to the configuration of one embodiment or modification. All of these fall within the scope of the present invention. Furthermore, numerical values, messages, etc. that appear in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different ones are used.

[0217] The functions of the present invention described above may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. Alternatively, they may be realized by software, with a microprocessor unit or the like interpreting and executing a program that realizes each function. Hardware and software may be used together. The software may be stored in the internal memory storage of the HMD 100 before shipping. After shipping, the software may be obtained from various server devices on the Internet. Alternatively, the software may be provided on a memory card, optical disc, or the like.

[0218] Furthermore, the control lines and information lines shown in the diagram are those considered necessary for explanation, and do not necessarily represent all the control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0219] 100: HMD, 101: smartphone, 105: expansion interface unit, 106: timer, 110: storage device, 111: main processor, 112: system bus, 113: RAM, 114: ROM, 115: flash memory, 120: operation reception device, 121: button switch, 122: touch panel, 123: operation key, 130: image processing device, 131: display, 133: outer camera, 134: inner camera, 140: audio processing device, 141: speaker, 143: microphone, 150: sensor, 151: receiving unit, 152: gyro sensor, 153: geomagnetic sensor, 154: acceleration sensor, 155: depth sensor, 160: communication device, 161: LAN communication I / F, 162: telephone network communication I / F, 163: BT communication I / F, 164: infrared communication I / F, 210: Main control unit, 220: Virtual remote control generation unit, 221: Image acquisition unit, 222: Specific information acquisition unit, 223: Control data acquisition unit, 224: Corresponding unit, 225: Virtual image generation unit, 230: Operation reception unit, 240: Command transmission unit, 250: Data storage unit, 251: analysis table, 251a: description content, 251b: analysis content, 251c: description type, 252: control data table, 252a: controlled device name, 252b: ID, 252c: control data, 252d: detailed information, 253: remote control table, 253a: description type, 253b: analysis content, 253c: ID, 253d: control command, 253e: area, 253f: display data, 253g: creation date and time, 254: gesture action table, 254a: gesture action, 254b: operation instruction, 255: processing data, 300: Presentation surface, 301: Presentation surface, 305: Outer frame, 310: Description information, 311: Controlled device name, 312: Desired control operation, 400: Virtual remote control image, 401: Virtual remote control image, 410: Controlled device information display area, 411: Controlled device name, 420: Desired control operation display area, 421: Reception object, 422: Reception object, 423: Reception object, 431: Unavailable display, 500: Remote control system, 510: Controlled device, 520: Wireless router, 530: Network, 550: Network server, 901: User

Claims

1. A remote control method in an information processing terminal that remotely controls a controlled device, comprising: an image acquisition step of acquiring a surrounding image that is an image of the surroundings of the information processing terminal; an information extraction step of extracting information specifying a controlled device and information specifying a desired control operation from description information on a presentation surface included in the surrounding image acquired in the image acquisition step; a device selection step of selecting the controlled device to be a target of the remote control based on the information identifying the controlled device extracted in the information extraction step; a control command extraction step of extracting a control command corresponding to information specifying the desired control operation extracted in the information extraction step for the controlled device selected in the device selection step; a virtual remote controller generating step of generating a virtual remote controller including a reception object that receives a user's operation instruction corresponding to the control command extracted in the control command extracting step; a virtual remote controller display step of displaying the reception object included in the virtual remote controller generated in the virtual remote controller generation step in an area on the presentation surface where information specifying the desired control operation is described; an operation reception step of receiving an operation instruction from the user with respect to the reception object; a command transmitting step of transmitting the control command corresponding to the receiving object that has received the user's operation instruction in the operation receiving step to the controlled device selected in the device selecting step; A remote control method comprising:

2. 2. The remote control method according to claim 1, The selection of the controlled device in the device selection step includes: A remote control method characterized by using information specifying the controlled device, which has character information or graphic information, to refer to a predetermined table and obtain identification information that can identify the controlled device.

3. 2. The remote control method according to claim 1, The extraction of the control command in the control command extraction step includes: A remote control method characterized by extracting a predetermined control command corresponding to information specifying the desired control operation extracted in the information extraction step from a plurality of control commands corresponding to the controlled device selected in the device selection step.

4. A remote control method according to any one of claims 1 to 3, a step of displaying the reception object in the virtual remote control display step, the reception object being erased after a predetermined period of time has elapsed since the reception object was displayed;

5. A remote control method according to any one of claims 1 to 3, A remote control method characterized in that, if the desired control operation is a control operation that cannot be accepted by the controlled device, an unacceptable display is generated in the virtual remote control generation step, and the unacceptable display is displayed by superimposing it on the accepted object in the virtual remote control display step.

Images