Information processing device, information processing method, and program

Abstract

To provide an information processing device, information processing method, and program that enable remote operation of functions of a hardware key on an electronic device.SOLUTION: An information processing device includes: a receiving unit configured to receive an image of an indication object from an imaging device that captures the indication object; a detection unit configured to detect an indication position pointed to by the indication object from the image received by the receiving unit; and a first determination unit configured to determine whether the indication position detected by the detection unit is within a region of a hardware key installed on an electronic device. The information processing device executes a function corresponding to a button on the hardware key, provided that the first determination unit determines that the indication position is within the region of the hardware key.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] While conventional pointing technology allows for pointing and operation on the screen, it has the problem of being unable to operate off-screen hardware keys (hereafter referred to as "hard keys") from a distance. While a method for operating hard keys from a distance is possible using a device such as a remote control, if the operating device is not nearby, the user must go and retrieve the device, which makes it impossible to operate the device while sitting down.

[0003] In order to enable the operation of the functions of such hard keys of electronic devices, a configuration has been disclosed that enables the operation of an electronic terminal by a touch operation on a display magnification device (for example, Patent Document 1). Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional technology has a problem in that it is not possible to operate the functions of the hard keys of the electronic device from a remote location.

[0005] The present invention has been made in view of the above, and has an object to provide an information processing device, an information processing method, and a program that enable remote operation of the functions of the hardware keys of an electronic device. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the present invention provides an information processing device comprising: a receiving unit that receives an image of a pointing object from an imaging device that images the pointing object; a detection unit that detects a pointing position to which the pointing object is pointing from the image received by the receiving unit; and a first judgment unit that judges whether the pointing position detected by the detection unit is within the area of ​​a hardware key installed on an electronic device, and the information processing device executes a function corresponding to a button of the hardware key, on the condition that at least the first judgment unit judges that the pointing position is within the area of ​​the hardware key. [Effects of the Invention]

[0007] According to the present invention, it is possible to remotely operate the functions of the hardware keys of an electronic device. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of an information processing system according to the first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the hardware configuration of the IWB according to the first embodiment. [Figure 3] FIG. 3 is an example of an image captured by the imaging device on the right side. [Figure 4] FIG. 4 is an example of an image captured by the imaging device on the left. [Figure 5] FIG. 5 is a front view of the display of the IWB according to the first embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of the configuration of functional blocks of the IWB according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing an example of an area definition table of the IWB according to the first embodiment. [Figure 8] FIG. 8 is a diagram showing an example in which a pointer is displayed at a designated position on the IWB according to the first embodiment. [Figure 9]FIG. 9 is a diagram showing an example in which a software menu is displayed on the display when the pointing position is in the area of ​​a hard key in the IWB according to the first embodiment. [Figure 10] FIG. 10 is a diagram showing another example of the area definition table of the IWB according to the first embodiment. [Figure 11] FIG. 11 is a diagram showing another example in which a software menu is displayed on the display when the pointing position is in the area of ​​a hard key in the IWB according to the first embodiment. [Figure 12] FIG. 12 is a diagram showing another example of the area definition table of the IWB according to the first embodiment. [Figure 13] FIG. 13 is a flowchart showing an example of the flow of a software menu display operation of the IWB according to the first embodiment. [Figure 14] FIG. 14 is a flowchart showing an example of the flow of the execution operation of the software menu of the IWB according to the first embodiment. [Figure 15] FIG. 15 is a flowchart showing an example of the flow of the operation of hiding the software menu of the IWB according to the first embodiment. [Figure 16] FIG. 16 is a diagram illustrating an example of the configuration of functional blocks of an IWB according to the second embodiment. [Figure 17] FIG. 17 is a flowchart showing an example of the flow of a software menu display operation of the IWB according to the second embodiment. [Figure 18] FIG. 18 is a flowchart showing an example of the flow of the execution operation of the software menu of the IWB according to the second embodiment. [Figure 19] FIG. 19 is a diagram illustrating an example of the hardware configuration of an IWB according to the third embodiment. [Figure 20] FIG. 20 is a diagram illustrating an example of the configuration of functional blocks of an IWB according to the third embodiment. [Figure 21] FIG. 21 is a flowchart showing an example of the flow of a software menu display operation of the IWB according to the third embodiment. [Figure 22] FIG. 22 is a flowchart showing an example of the flow of the execution operation of the software menu of the IWB according to the third embodiment. [Figure 23] FIG. 23 is a flowchart showing an example of the flow of the operation of executing the functions of the hard keys of the IWB according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, with reference to the drawings, embodiments of an information processing device, an information processing method, and a program according to the present invention will be described in detail. Furthermore, the present invention is not limited to the following embodiments, and the components in the following embodiments include those that would be easily conceived by a person skilled in the art, those that are substantially the same, and those that are within the scope of what is called equivalents. Furthermore, various omissions, substitutions, modifications, and combinations of the components can be made without departing from the spirit of the following embodiments.

[0010] [First embodiment] (Overall configuration of information processing system) 1 is a diagram showing an example of the overall configuration of an information processing system according to the first embodiment. The overall configuration of the information processing system 1 according to this embodiment will be described with reference to FIG.

[0011] As shown in FIG. 1, the information processing system 1 includes an IWB (Interactive Whiteboard) 10, a camera 20a, and a camera 20b.

[0012] The IWB 10 is an electronic device (an example of an information processing device) that can display images and write figures, characters, etc. on a display using an electronic pen. The IWB 10 detects a position (pointing position) pointed to by a pointing object 30, such as a user's hand or a pointer stick, based on captured images received from cameras 20a and 20b.

[0013] The camera 20a is an imaging device that is installed on the left side of the user when looking in the direction from the user toward the IWB 10 and is connected to the IWB 10. The camera 20a captures an image with an angle of view that includes the display of the IWB 10 and the pointing object 30, and transmits the captured image to the IWB 10.

[0014] Camera 20b is an imaging device that is installed on the right side of the user when looking in the direction from the user toward the IWB 10 and is connected to the IWB 10. Camera 20b captures an image with an angle of view that includes the display of the IWB 10 and the pointing object 30, and transmits the captured image to the IWB 10.

[0015] (IWB hardware configuration) 2 is a diagram showing an example of the hardware configuration of the IWB according to the first embodiment. The hardware configuration of the IWB 10 according to this embodiment will be described with reference to FIG.

[0016] As shown in FIG. 2, the IWB 10 of this embodiment includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an SSD (Solid State Drive) 204, a network I / F 205, an external device connection I / F 206, a capture device 211, a GPU (Graphics Processing Unit) 212, a display controller 213, a hard key controller 214, a sensor controller 215, a contact sensor 216, an electronic pen controller 217, a near-field communication circuit 219, a display 220, and hard keys 221.

[0017] The CPU 201 is a computing device that controls the overall operation of the IWB 10. The ROM 202 is a non-volatile storage device that stores programs used to drive the CPU 201, such as an IPL (Initial Program Loader). The RAM 203 is a volatile storage device that is used as a work area for the CPU 201.

[0018] The SSD 204 is an auxiliary storage device that stores various data such as programs for the IWB 10 .

[0019] The network I / F 205 is an interface for data communication via a network, and is an interface circuit capable of communication in accordance with, for example, Ethernet (registered trademark), TCP (Transmission Control Protocol) / IP (Internet Protocol), and the like.

[0020] The external device connection I / F 206 is an interface circuit for connecting various external devices, such as a USB (Universal Serial Bus) memory 225 and external devices (such as a microphone 223, a speaker 224, a camera 20a, and a camera 20b).

[0021] The capture device 211 is a device circuit that displays video information as a still image or a moving image on the display of an external PC (Personal Computer) 301 .

[0022] The GPU 212 is a computing device that specializes in executing image processing. The display controller 213 is a controller that controls and manages screen display to output images that have been processed by the GPU 212 to the display 220 or the like.

[0023] The hard key controller 214 is a controller that detects user operations on a hard key 221 that is configured with one or more buttons and that is installed on the housing of the IWB 10. In response to the operation on the hard key 221, the hard key controller 214 executes processing requests to a software program and control requests for the display 220 to the display controller 213.

[0024] The sensor controller 215 is a controller that controls the processing of the contact sensor 216. The contact sensor 216 is a sensor that detects contact of the electronic pen 222, the user's finger 302, or the like on the display 220. Specifically, the contact sensor 216 inputs and detects coordinates using an infrared blocking method. This coordinate input and coordinate detection method involves two light receiving and emitting devices installed at both ends of the upper side of the display 220 emitting multiple infrared rays parallel to the display 220, and receiving light that is reflected by a reflecting member installed around the periphery of the display 220 and returns along the same optical path as the light emitted by the light receiving element. The contact sensor 216 outputs the IDs of the infrared rays radiated by the two light receiving and emitting devices that are blocked by an object to the sensor controller 215, and the sensor controller 215 identifies the coordinate position that is the contact position of the object.

[0025] The electronic pen controller 217 is a controller that communicates with the electronic pen 222 to determine whether the tip of the pen or the butt of the pen touches the display 220 .

[0026] The short-distance communication circuit 219 is a communication circuit based on NFC (Near Field Communication), Bluetooth (registered trademark), etc. To the short-distance communication circuit 219, an antenna 219a for wireless communication is connected.

[0027] The display 220 is a display device such as a liquid crystal or organic EL display device that displays various images.

[0028] The hard key 221 is a hardware key configured with buttons for executing various functions such as power, sleep, display adjustment, etc. Note that the hard key 221 is not limited to being a hardware key already provided in the IWB 10, and may be an external hardware key that can be connected to the hard key controller 214.

[0029] Furthermore, the IWB 10 includes a bus 210. The bus 210 is an address bus, a data bus, etc., for electrically connecting the components such as the CPU 201 shown in FIG.

[0030] The contact sensor 216 is not limited to an infrared blocking type, and various detection means may be used, such as a capacitive touch panel that identifies the contact position by detecting a change in capacitance, a resistive film touch panel that identifies the contact position by a change in voltage between two opposing resistive films, or an electromagnetic induction touch panel that identifies the contact position by detecting electromagnetic induction caused by a contacting object coming into contact with the display 220. Furthermore, the electronic pen controller 217 may determine whether or not a part of the electronic pen 222 that is held by the user or another part of the electronic pen has been touched, in addition to the pen tip or pen tail of the electronic pen 222.

[0031] (Calculating the transformation matrix) Fig. 3 is an example of an image captured by the imaging device on the right side. Fig. 4 is an example of an image captured by the imaging device on the left side. Fig. 5 is a front view of the display of the IWB according to the first embodiment. With reference to Figs. 3 to 5, a method for detecting a position pointed to by a pointing object 30 using two cameras (cameras 20a and 20b) in the information processing system 1 according to this embodiment will be described.

[0032] FIG. 3 shows the captured image IMR (hereinafter sometimes referred to as the right captured image) captured by the right camera 20b, and FIG. 4 shows the captured image IML (hereinafter sometimes referred to as the left captured image) captured by the left camera 20a.

[0033] The IWB 10 receives the right captured image from the camera 20b, extracts an object such as a hand from the right captured image, and compares the shape and color of the object with pre-stored shape and color data of a hand with one finger outstretched, thereby determining that the object is a hand with one finger outstretched. This determination is made by machine learning using shape and color data of many hands with one finger outstretched.

[0034] Next, the IWB 10 identifies the base of the finger (point P1 shown in FIG. 3) and the tip (point Q1 shown in FIG. 3). The IWB 10 also compares shape data of the object extracted from the right captured image with pre-stored shape data of the display 220, thereby determining that the object is the display 220. This determination is made by machine learning using shape data of many displays 220.

[0035] In the right captured image (captured image IMR) shown in Fig. 3, points A1, B1, C1, and D1 respectively indicate the upper left corner, lower left corner, upper right corner, and lower right corner of the display 220. In the right captured image shown in Fig. 3, point Ei is the intersection of an extension of line segment P1Q1 and an extension of line segment A1B1. Point Fi is the intersection of an extension of line segment P1Q1 and line segment C1D1.

[0036] Similarly to the above, the IWB 10 receives a left captured image from the camera 20a, determines a hand with one finger out from the left captured image, and further identifies the base (point P2 shown in FIG. 4) and tip (point Q2 shown in FIG. 4) of the finger. Then, similar to the above, the IWB 10 determines the display 220 from the left captured image.

[0037] In the left captured image (captured image IML) shown in Fig. 4, points A2, B2, C2, and D2 respectively indicate the upper left corner, lower left corner, upper right corner, and lower right corner of the display 220. In the left captured image shown in Fig. 4, point Gi is the intersection of an extension of line segment P2Q2 and line segment A2B2. Point Hi is the intersection of an extension of line segment P2Q2 and line segment C2D2.

[0038] 5 shows a front view of the display 220 of the IWB 10. Point T on the display 220 pointed to by the finger corresponds to the intersection of a line connecting points Ei and Fi in the right captured image captured by the right camera 20b and a line connecting points Gi and Hi in the left captured image captured by the left camera 20a. Since the coordinates in the captured image are based on the pixel positions of the captured image, when a pointer or the like is to be displayed at the part pointed to by the finger (pointing position), the coordinates must be converted to coordinates based on the display pixel positions of the display 220.

[0039] 5 indicate the upper left corner, lower left corner, upper right corner, and lower right corner of the display 220. Points Ed and Fd are points of coordinates based on the display pixel positions of the display 220, obtained by converting points Ei and point Fi in the right captured image from coordinates based on pixel positions of the right captured image using a transformation matrix TR for coordinate conversion, which will be described later. Points Gd and Hd are points of coordinates based on the display pixel positions of the display 220, obtained by converting points Gi and point Hi in the left captured image from coordinates based on pixel positions of the left captured image using a transformation matrix TL for coordinate conversion, which will be described later.

[0040] The transformation matrices TR and TL are obtained by the following equations: The coordinates of the upper left, lower left, upper right, and lower right corners of the display 220 based on the pixel positions of the right captured image are respectively A1(a1x, a1y), B1(b1x, b1y), C1(c1x, c1y), and D1(d1x, d1y), and the coordinates of the upper left, lower left, upper right, and lower right corners of the display 220 based on the display pixel positions of the display 220 are respectively A3(a3x, a3y), B3(b3x, b3y), C3(c3x, c3y), and D3(d3x, d3y), and the eight simultaneous equations shown in the following equation (1) are established.

[0041]

number

[0042] From the simultaneous equations shown in the above equation (1), a transformation matrix TR that transforms the coordinates of the right captured image into coordinates based on the display pixel positions of the display 220 is calculated as shown in the following equation (2).

[0043]

number

[0044] Furthermore, the coordinates of the upper left, lower left, upper right, and lower right corners of the display 220 based on the pixel positions of the left captured image are respectively A2(a2x, a2y), B2(b2x, b2y), C2(c2x, c2y), and D2(d2x, d2y), and the coordinates of the upper left, lower left, upper right, and lower right corners of the display 220 based on the display pixel positions of the display 220 are respectively A3(a3x, a3y), B3(b3x, b3y), C3(c3x, c3y), and D3(d3x, d3y), and the eight simultaneous equations shown in the following equation (3) are formulated.

[0045]

number

[0046] From the simultaneous equations shown in the above equation (3), a transformation matrix TL that transforms the coordinates of the left captured image into coordinates based on the display pixel positions of the display 220 is calculated as shown in the following equation (4).

[0047]

number

[0048] Then, using the transformation matrix TR shown in the above equation (2), point Ei(eix, eiy) in the right captured image is transformed into point Ed(edx, edy) expressed in the display position coordinates of the display 220 by the following equations (5) and (6).

[0049]

number

[0050]

number

[0051] Furthermore, the point Fi (fix, fiy) in the right captured image is transformed into a point Fd (fdx, fdy) expressed in the display position coordinates of the display 220 by the following equations (7) and (8) using the transformation matrix TR shown in the above equation (2).

[0052]

number

[0053]

number

[0054] Furthermore, the point Gi(gix,giy) in the left captured image is transformed into a point Gd(gdx,gdy) expressed in the display position coordinates of the display 220 by the following equations (9) and (10) using the transformation matrix TL shown in the above equation (4).

[0055]

number

[0056]

number

[0057] Furthermore, the point Hi(hix, hiy) in the left captured image is transformed into a point Hd(hdx, hdy) expressed in the display position coordinates of the display 220 by the following equations (11) and (12) using the transformation matrix TL shown in the above equation (4).

[0058]

number

[0059]

number

[0060] The display coordinates of point T on display 220 pointed at by the finger are the intersection of line segments EdFd and GdHd. Therefore, IWB 10 can calculate the linear equations of line segments EdFd and GdHd, and calculate the display coordinates of point T using simultaneous equations based on these linear equations. Furthermore, the transformation matrices TR and TL shown in the above equations (2) and (4) only need to be calculated once if the installation positions of cameras 20a and 20b are not changed, and can be stored in RAM 203 or SSD 204 of IWB 10.

[0061] (Configuration and operation of IWB functional blocks) FIG. 6 is a diagram showing an example of the configuration of functional blocks of an IWB according to the first embodiment. FIG. 7 is a diagram showing an example of an area definition table of an IWB according to the first embodiment. FIG. 8 is a diagram showing an example of displaying a pointer at a pointed position on an IWB according to the first embodiment. FIG. 9 is a diagram showing an example of displaying a software menu on a display when the pointed position is in a hard key area on an IWB according to the first embodiment. FIG. 10 is a diagram showing another example of an area definition table of an IWB according to the first embodiment. FIG. 11 is a diagram showing another example of displaying a software menu on a display when the pointed position is in a hard key area on an IWB according to the first embodiment. FIG. 12 is a diagram showing another example of an area definition table of an IWB according to the first embodiment. The configuration and operation of functional blocks of an IWB 10 according to this embodiment will be described with reference to FIGS. 6 to 12.

[0062] As shown in FIG. 6, the IWB 10 includes an image receiving unit 101 (receiving unit), a transformation matrix calculation unit 102, an indication position detection unit 103 (detection unit), a determination unit 104 (first determination unit), a display control unit 105, and a memory unit 106.

[0063] The image receiving unit 101 is a functional unit that receives, via the external device connection I / F 206, captured images (left captured image and right captured image) captured by the cameras 20a and 20g.

[0064] The transformation matrix calculation unit 102 is a functional unit that calculates the above-mentioned transformation matrix. Specifically, the transformation matrix calculation unit 102 first extracts an object on the display 220 from the right-hand captured image received by the image receiving unit 101 using the above-mentioned method, and identifies the coordinates of the above-mentioned points A1, B1, C1, and D1 from the object. Then, the transformation matrix calculation unit 102 calculates the transformation matrix TR using the above-mentioned formulas (1) and (2). Similarly, the transformation matrix calculation unit 102 extracts an object on the display 220 from the left-hand captured image received by the image receiving unit 101 using the above-mentioned method, and identifies the coordinates of the above-mentioned points A2, B2, C2, and D2 from the object. Then, the transformation matrix calculation unit 102 calculates the transformation matrix TR using the above-mentioned formulas (3) and (4). The transformation matrix calculation unit 102 stores the calculated transformation matrices TR and TL in the storage unit 106.

[0065] The calculation of the transformation matrices TR and TL by the transformation matrix calculation unit 102 only needs to be performed once unless the installation positions of the cameras 20a, 20b and IWB 10 (display 220) are moved, and subsequent processing can be performed by referring to the transformation matrices TL and TR stored in the memory unit 106.

[0066] The pointing position detection unit 103 is a functional unit that detects the pointing position pointed to by the pointing object 30, using the right captured image and the left captured image received by the image receiving unit 101. Specifically, the pointing position detection unit 103 first extracts the object on the display 220 and the object on the pointing object 30 from the right captured image received by the image receiving unit 101 using the method described above, and identifies the coordinates of points Ei and Fi. Then, the pointing position detection unit 103 calculates the display position coordinates of point Ed on the display 220 from the coordinates of point Ei using the transformation matrix TR stored in the storage unit 106, using equations (5) and (6). Furthermore, the pointing position detection unit 103 calculates the display position coordinates of point Fd on the display 220 from the coordinates of point Fi using the transformation matrix TR, using equations (7) and (8). The pointing position detection unit 103 also extracts the object on the display 220 and the object on the pointing object 30 from the left captured image received by the image receiving unit 101 using the method described above, and identifies the coordinates of points Gi and Hi. The pointing position detection unit 103 then calculates the display position coordinates of point Gd on the display 220 from the coordinates of point Gi using the transformation matrix TL stored in the storage unit 106, using equations (9) and (10). The pointing position detection unit 103 then calculates the display position coordinates of point Fd on the display 220 from the coordinates of point Hi using the transformation matrix TL, using equations (11) and (12). The display position coordinates here are not limited to coordinates within the display area of ​​the display 220, but also include coordinates of areas outside the display area but on the same plane as the display area. The pointing position detection unit 103 then identifies the display position coordinates of point T, which is the intersection of line segments EdFd and GdHd, and detects this as the pointing position indicated by the pointing object 30.

[0067] The determination unit 104 is a functional unit that determines whether or not the pointing position pointed to by the pointing object 30 detected by the pointing position detection unit 103 is within a predetermined area.

[0068] For example, the area definition table shown in FIG. 7 associates the X coordinate (Left) of the left edge of each area, the Y coordinate (Top) of the top edge, the width (Width) of the area in the X direction, and the height (Height) of the area in the Y direction, thereby defining the range of each area. The area definition table is stored in the storage unit 106. In the example shown in FIG. 7, the area definition table defines the display area of ​​the display 220 shown in FIG. 8 as the "display display area," and defines the area of ​​the hard keys 221 at the bottom of the display 220 as the "lower front hard key area." As shown in FIG. 7, the display display area is an area with a width of 1920 pixels and a height of 1080 pixels, with the left and top ends as the origin. Furthermore, as shown in FIG. 7, the lower front hard key area has a left edge coordinate of 1440 pixels and a top edge coordinate of 1080 pixels, and is therefore an area outside the display area of ​​the display 220 (the above-mentioned display display area), and is an area with a width of 480 pixels and a height of 100 pixels.

[0069] The determination unit 104 refers to the area definition table stored in the storage unit 106 and determines whether the pointing position indicated by the pointing object 30 detected by the pointing position detection unit 103 is within each area defined in the area definition table. For example, when the horizontal coordinate of the pointing position detected by the pointing position detection unit 103 is within the range of 1440 to 1920 pixels and the vertical coordinate is within the range of 1080 to 1180 pixels, the determination unit 104 determines that the pointing position is within the area of ​​the hard key 221.

[0070] The display control unit 105 is a functional unit that controls the display operation of the display 220. For example, when the determination unit 104 determines that the pointed position is within the display area of ​​the display 220, the display control unit 105 displays a pointer P at the pointed position on the display 220 as shown in FIG. 8 to make the pointed position visible. By displaying the pointed position as the pointer P on the display 220 in this way, the user can visually recognize the pointed position and adjust the pointed position. Furthermore, when the determination unit 104 determines that the pointed position is within the area of ​​the hard keys 221 at the bottom of the display 220 (the "lower front hard key area" in the area definition table), the display control unit 105 displays a software menu 401 in a display area near the hard keys 221 on the display 220 as shown in FIG. 9. The software menu 401 is, for example, a menu including software keys corresponding to the hard keys of the hard keys 221 as shown in FIG. 9. In this way, for example, the user can easily point to the area of ​​the hard keys 221 outside the display area by first pointing to the area within the display area of ​​the display 220, and then, from the state in which the pointer P indicating the pointing position is displayed as shown in Figure 8, moving the pointing position to within the area of ​​the hard keys 221 outside the display area.

[0071] It should be noted that the software menu 401 displayed on the display 220 is not limited to including buttons corresponding to all of the buttons included in the hard keys 221, but may include buttons corresponding to at least some of the buttons included in the hard keys 221.

[0072] The storage unit 106 is a functional unit that stores the transformation matrices TR and TL calculated by the transformation matrix calculation unit 102, the region definition table shown in Fig. 7, etc. The storage unit 106 is realized by the RAM 203 or the SSD 204 shown in Fig. 2.

[0073] The image receiving unit 101, transformation matrix calculation unit 102, pointed position detection unit 103, determination unit 104, and display control unit 105 are realized by executing a program by the CPU 201 shown in Fig. 2. Note that at least some of the image receiving unit 101, transformation matrix calculation unit 102, pointed position detection unit 103, determination unit 104, and display control unit 105 may be realized by a hardware circuit such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0074] Note that the hard keys 221 are not limited to being installed at the bottom of the display 220, and may be installed on the side or top as shown in FIG. 11. FIG. 10 shows an example of an area definition table for when the hard keys 221 are installed on the side of the display 220 as shown in FIG. 11. In this case, the area of ​​the hard keys 221 (right side hard key area) has a left edge coordinate of 1920 pixels and a top edge coordinate of 720 pixels as shown in FIG. 10, and is therefore outside the display area (display display area) of the display 220, and is an area 100 pixels wide and 360 pixels high. Furthermore, for example, when the horizontal coordinate of the pointed position detected by the pointed position detection unit 103 is within the range of 1920 to 2020 pixels and the vertical coordinate is within the range of 720 to 1080 pixels, the determination unit 104 determines that the pointed position is within the area of ​​the hard keys 221. Then, when the determination unit 104 determines that the designated position is within the area of ​​the hard keys 221 on the side of the display 220 (the "right side hard key area" in the area definition table), the display control unit 105 displays the software menu 402 in the display area near the hard keys 221 on the display 220, as shown in FIG. 11.

[0075] 7 and 10 correspond to a specific model of IWB 10, but if the installation location of the hard keys differs for each of multiple models, the area definition table shown in Fig. 12, in which the model and each area are associated, may be used. This makes it possible to apply the same software to different models.

[0076] In addition, the image receiving unit 101, the transformation matrix calculation unit 102, the indication position detection unit 103, the determination unit 104, the display control unit 105, and the storage unit 106 are all implemented within the IWB 10, but this is not limitative and at least one of them may be implemented in another information processing device. In other words, the other information processing device is responsible for implementing processing that enables the functions of the hardware keys (hard key 221, etc.) of an electronic device such as the IWB 10 to be operated from a remote location.

[0077] Furthermore, the functions of each functional unit of the IWB 10 shown in Fig. 6 are conceptually illustrated, and the configuration is not limited to this. That is, each functional unit of the IWB 10 does not need to be configured as a clear software module as the block shown in Fig. 6, but the functions of each functional unit as a whole may be realized by executing a program on the IWB 10. For example, multiple functional units illustrated as independent functional units in the IWB 10 shown in Fig. 6 may be configured as a single functional unit. On the other hand, the function of one functional unit in the IWB 10 shown in Fig. 6 may be divided into multiple functional units and configured as multiple functional units.

[0078] (Flow of IWB software menu display operations) 13 is a flowchart showing an example of the flow of the software menu display operation of the IWB according to the first embodiment. The flow of the software menu display operation of the IWB 10 according to this embodiment will be described with reference to FIG.

[0079] <Step S11> The image receiving unit 101 of the IWB 10 receives the captured images (left captured image and right captured image) captured by the cameras 20a and 20g via the external device connection I / F 206. Then, the pointed position detecting unit 103 of the IWB 10 detects the pointed position pointed by the pointing object 30 such as a user's finger, using the right captured image and the left captured image received by the image receiving unit 101. Then, the process proceeds to step S12.

[0080] <Step S12> If the pointed position is detected by the pointed position detection unit 103 (step S12: Yes), the process proceeds to step S13, and if the pointed position is not detected (step S12: No), the process returns to step S11.

[0081] <Step S13> The determination unit 104 of the IWB 10 refers to the area definition table stored in the storage unit 106 and determines whether the pointing position indicated by the pointing object 30 detected by the pointing position detection unit 103 is within the display area of ​​the display 220 defined in the area definition table. If the pointing position is within the display area of ​​the display 220 (step S13: Yes), the process proceeds to step S14, and if it is outside the display area (step S13: No), the process proceeds to step S15.

[0082] <Step S14> To make the pointed position visible, the display control unit 105 of the IWB 10 displays a pointer P at the pointed position on the display 220. Then, the display operation of the software menu ends.

[0083] <Step S15> Furthermore, the determination unit 104 refers to the area definition table stored in the storage unit 106 and determines whether the pointing position indicated by the pointing object 30 detected by the pointing position detection unit 103 is within the area of ​​the hard key 221 defined in the area definition table. If the pointing position is within the area of ​​the hard key 221 (step S15: Yes), the process proceeds to step S16, and if it is outside the area (step S15: No), the process returns to step S11.

[0084] <Step S16> The display control unit 105 displays the software menu 401 in the display area of ​​the display 220 near the hard keys 221. Then, the software menu display operation ends.

[0085] The operations of steps S11 to S16 are repeatedly executed as described above. By this process, when the user points to the area of ​​the hard key 221, the software menu 401 corresponding to the hard key 221 is displayed on the display 220.

[0086] (IWB software menu execution flow) Fig. 14 is a flowchart showing an example of the flow of the execution operation of the software menu of the IWB according to the first embodiment. The flow of the execution operation of the software menu of the IWB 10 according to this embodiment will be described with reference to Fig. 14. Note that the description here is based on the premise that the software menu 401 corresponding to the hard key 221 is already displayed on the display 220.

[0087] <Step S21> The image receiving unit 101 receives the captured images (left captured image and right captured image) captured by the cameras 20a and 20g via the external device connection I / F 206. Then, the pointed position detecting unit 103 detects the pointed position pointed to by the pointing object 30, such as the user's finger, using the right captured image and the left captured image received by the image receiving unit 101. Then, the process proceeds to step S22.

[0088] <Step S22> If the pointed position is detected by the pointed position detection unit 103 (step S22: Yes), the process proceeds to step S23, and if the pointed position is not detected (step S22: No), the process returns to step S21.

[0089] <Step S23> The determination unit 104 refers to the area definition table stored in the storage unit 106 and determines whether the pointing position indicated by the pointing object 30 detected by the pointing position detection unit 103 is within the display area of ​​the display 220 defined in the area definition table. If the pointing position is within the display area of ​​the display 220 (step S23: Yes), the process proceeds to step S24, and if the pointing position is outside the display area (step S23: No), the process returns to step S21.

[0090] <Step S24> To make the indicated position visible, the display control unit 105 displays a pointer P at the indicated position on the display 220. Then, the process proceeds to step S25.

[0091] <Step S25> The determination unit 104 determines whether the pointing position indicated by the pointing object 30 detected by the pointing position detection unit 103 is on a button of the software menu 401 displayed on the display 220. If the pointing position is on the button (step S25: Yes), the process proceeds to step S26, and if the pointing position is not on the button (step S25: No), the process returns to step S21.

[0092] <Step S26> When the IWB 10 determines that the designated position is on a button of the software menu 401, it considers that the button of the software menu 401 has been pressed (selected) and executes the function of the button of the hard key 221 corresponding to that button. Note that the content of the function executed when the button at the designated position has been pressed (selected) is not particularly limited. Then, the execution operation of the software menu is terminated.

[0093] Through this process, the functions corresponding to the buttons on the displayed software menu 401 can be executed.

[0094] (Flow of operations for hiding the IWB software menu) Fig. 15 is a flowchart showing an example of the flow of the operation of hiding the software menu of the IWB according to the first embodiment. The flow of the operation of hiding the software menu of the IWB 10 according to this embodiment will be described with reference to Fig. 15. Note that the description here is based on the premise that the software menu 401 corresponding to the hard key 221 is already displayed on the display 220. The display control unit 105 starts counting a timer that indicates the elapsed time since the software menu 401 was displayed.

[0095] <Step S31> The image receiving unit 101 receives the captured images (left captured image and right captured image) captured by the cameras 20a and 20g via the external device connection I / F 206. Then, the pointed position detecting unit 103 detects the pointed position pointed to by the pointing object 30, such as the user's finger, using the right captured image and the left captured image received by the image receiving unit 101. Then, the process proceeds to step S32.

[0096] <Step S32> If the pointed position is detected by the pointed position detection unit 103 (step S32: Yes), the process proceeds to step S33, and if the pointed position is not detected (step S32: No), the process proceeds to step S37.

[0097] <Step S33> The determination unit 104 refers to the area definition table stored in the storage unit 106 and determines whether the pointing position indicated by the pointing object 30 detected by the pointing position detection unit 103 is within the display area of ​​the display 220 defined in the area definition table. If the pointing position is within the display area of ​​the display 220 (step S33: Yes), the process proceeds to step S34, and if the pointing position is outside the display area (step S33: No), the process proceeds to step S37.

[0098] <Step S34> To make the indicated position visible, the display control unit 105 displays a pointer P at the indicated position on the display 220. Then, the process proceeds to step S35.

[0099] <Step S35> The determination unit 104 determines whether the pointing position indicated by the pointing object 30 detected by the pointing position detection unit 103 is on a button of the software menu 401 displayed on the display 220. If the pointing position is on the button (step S35: Yes), the process proceeds to step S36, and if the pointing position is not on the button (step S35: No), the process proceeds to step S37.

[0100] <Step S36> The display control unit 105 resets the timer that started measuring when the software menu 401 was displayed, and then the process returns to step S31.

[0101] <Step S37> The display control unit 105 determines whether a predetermined time (e.g., 10 seconds) has elapsed on the timer that started measuring when the software menu 401 was displayed (i.e., the time that has elapsed without any button on the software menu 401 being pressed). If the predetermined time has elapsed (step S37: Yes), the process proceeds to step S38, and if the predetermined time has not elapsed (step S37: No), the process returns to step S31.

[0102] <Step S38> The display control unit 105 hides the software menu 401. Then, the operation of hiding the software menu is completed.

[0103] This allows the software menu 401 to be automatically hidden if no operation is performed on the displayed software menu 401 for a predetermined period of time.

[0104] As described above, in the IWB10 according to this embodiment, the image receiving unit 101 receives an image of the pointing object 30 from the cameras 20a and 20b that capture the image of the pointing object 30, the pointing position detection unit 103 detects the pointing position at which the pointing object 30 is pointing from the image received by the image receiving unit 101, the determination unit 104 determines whether the pointing position detected by the pointing position detection unit 103 is within the area of ​​the hard key 221 installed on the IWB10, and the IWB10 executes the function corresponding to the button on the hard key 221, provided that at least the determination unit 104 determines that the pointing position is within the area of ​​the hard key 221. Specifically, when the determination unit 104 determines that the pointing position is within the area of ​​the hard key 221, the display control unit 105 causes the display 220 of the IWB 10 to display a software menu 401 for executing the function of the button of the hard key 221, the determination unit 104 determines whether or not the pointing position is on a button of the software menu 401, and when the determination unit 104 determines that the pointing position is on a button of the software menu 401, the IWB 10 executes the function of the button of the hard key 221 corresponding to that button. This makes it possible to operate the function of the hard key 221 of the IWB 10 from a remote location.

[0105] [Second embodiment] The IWB according to the second embodiment will be described, focusing on the differences from the IWB 10 according to the first embodiment. In this embodiment, the operation of responding when the pointed position remains stationary for a predetermined period of time or more will be described. Note that the overall configuration of the information processing system according to this embodiment and the hardware configuration of the IWB are the same as those described in the first embodiment.

[0106] (Configuration and operation of IWB functional blocks) 16 is a diagram showing an example of the configuration of functional blocks of an IWB according to the second embodiment. The configuration and operation of the functional blocks of an IWB 10a according to this embodiment will be described with reference to FIG.

[0107] 16, the IWB 10a (an example of an electronic device, an example of an information processing device) includes an image receiving unit 101 (a receiving unit), a transformation matrix calculation unit 102, a pointed position detection unit 103 (a detection unit), a determination unit 104 (a first determination unit), a display control unit 105, a storage unit 106, and a display condition determination unit 107 (a second determination unit). Note that the operations of the image receiving unit 101, the transformation matrix calculation unit 102, the pointed position detection unit 103, the determination unit 104, the display control unit 105, and the storage unit 106 are the same as those in the first embodiment described above.

[0108] The display condition determination unit 107 is a functional unit that determines whether or not a condition for displaying the software menu 401 is met when the pointing position pointed to by the pointing object 30 detected by the pointing position detection unit 103 is within the area of ​​the hard keys 221. If the condition is met, the display control unit 105 displays the software menu 401 in a display area near the hard keys 221 on the display 220. On the other hand, if the condition is not met, the display control unit 105 does not display the software menu 401 on the display 220.

[0109] The image receiving unit 101, transformation matrix calculation unit 102, pointed position detection unit 103, determination unit 104, display control unit 105, and display condition determination unit 107 are realized by executing a program by the CPU 201 shown in Fig. 2. Note that at least some of the image receiving unit 101, transformation matrix calculation unit 102, pointed position detection unit 103, determination unit 104, display control unit 105, and display condition determination unit 107 may be realized by a hardware circuit such as an FPGA or an ASIC.

[0110] Furthermore, although the image receiving unit 101, transformation matrix calculation unit 102, indication position detection unit 103, judgment unit 104, display control unit 105, memory unit 106, and display condition judgment unit 107 are all assumed to be realized within IWB 10a, this is not limited to this, and at least any one of these may be realized in another information processing device.

[0111] Furthermore, the functions of each functional unit of the IWB 10a shown in Fig. 16 are conceptually illustrated, and the configuration is not limited to this. That is, each functional unit of the IWB 10a does not need to be configured as a clear software module as the block shown in Fig. 16, but the functions of each functional unit as a whole may be realized by executing a program on the IWB 10a. For example, multiple functional units illustrated as independent functional units in the IWB 10a shown in Fig. 16 may be configured as a single functional unit. On the other hand, the function of one functional unit in the IWB 10a shown in Fig. 16 may be divided into multiple functional units and configured as multiple functional units.

[0112] (Flow of IWB software menu display operations) 17 is a flowchart showing an example of the flow of the software menu display operation of the IWB according to the second embodiment. The flow of the software menu display operation of the IWB 10a according to the present embodiment will be described with reference to FIG.

[0113] <Steps S41 to S45> The processes of steps S41 to S45 are the same as the processes of steps S11 to S15 shown in Fig. 13. If the designated position is within the area of ​​the hard key 221 (step S45: Yes), the process proceeds to step S46, and if it is outside the area (step S45: No), the process returns to step S41.

[0114] <Step S46> The display condition determination unit 107 of the IWB 10a determines whether the pointing position indicated by the pointing object 30 detected by the pointing position detection unit 103 has stayed within the area of ​​the hard key 221 for a predetermined time or more (for example, 2 seconds, etc.). If the pointing position has stayed within the area of ​​the hard key 221 for the predetermined time or more (step S46: Yes), the process proceeds to step S47, and if the pointing position has not stayed within the area of ​​the hard key 221 (step S46: No), the process returns to step S41, and the software menu 401 is not displayed.

[0115] <Step S47> The display control unit 105 displays the software menu 401 in the display area of ​​the display 220 near the hard keys 221. Then, the software menu display operation ends.

[0116] (IWB software menu execution flow) Fig. 18 is a flowchart showing an example of the flow of the execution operation of the software menu of the IWB according to the second embodiment. The flow of the execution operation of the software menu of the IWB 10a according to this embodiment will be described with reference to Fig. 18. Note that the description here is based on the premise that the software menu 401 corresponding to the hard key 221 is already displayed on the display 220.

[0117] <Steps S51 to S55> The processes of steps S51 to S55 are the same as the processes of steps S21 to S25 shown in Fig. 14. If the designated position is on a button of the software menu 401 (step S55: Yes), the process proceeds to step S56, and if not on the button (step S55: No), the process returns to step S51.

[0118] <Step S56> The display condition determination unit 107 of the IWB 10a determines whether the pointing position indicated by the pointing object 30 detected by the pointing position detection unit 103 has remained on a button of the software menu 401 for a predetermined time or more (for example, 2 seconds, etc.). If the pointing position has remained on a button of the software menu 401 for the predetermined time or more (step S56: Yes), the process proceeds to step S57, and if the pointing position has not remained on the button of the software menu 401 (step S56: No), the process returns to step S51, and the function corresponding to the button is not executed.

[0119] <Step S57> When the display condition determination unit 107 determines that the designated position has remained on a button of the software menu 401 for a predetermined period of time or longer, the IWB 10a determines that the button of the software menu 401 has been pressed (selected), and executes the function of the button of the hard key 221 corresponding to the button. Note that the content of the function executed when the button at the designated position has been pressed (selected) is not particularly limited. Then, the execution operation of the software menu is terminated.

[0120] As described above, in the IWB 10a according to this embodiment, the display condition determination unit 107 determines whether the pointing position has remained within the area of ​​the hard keys 221 for a predetermined period of time or more, and the display control unit 105 displays the software menu 401 on the display 220 when the display condition determination unit 107 determines that the pointing position has remained within the area of ​​the hard keys 221 for a predetermined period of time or more, and the display condition determination unit 107 determines whether the pointing position has remained on a button of the software menu 401 for a predetermined period of time or more, and the IWB 10a executes the function of the button of the hard keys 221 corresponding to that button when the display condition determination unit 107 determines that the pointing position has remained on a button of the software menu 401 for a predetermined period of time or more. This achieves the same effects as the first embodiment described above, and can prevent the display 220 from being powered off unintentionally or the software menu 401 from being displayed unintentionally.

[0121] [Third embodiment] The IWB according to the third embodiment will be described, focusing on the differences from the IWB 10 according to the first embodiment. In this embodiment, the operation of responding to a specific gesture will be described.

[0122] (IWB hardware configuration) 19 is a diagram showing an example of the hardware configuration of an IWB according to the third embodiment. The hardware configuration of an IWB 10b according to this embodiment will be described with reference to FIG.

[0123] 19, an IWB 10b (an example of an electronic device, an example of an information processing device) according to this embodiment includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an SSD (Solid State Drive) 204, a network I / F 205, an external device connection I / F 206, a capture device 211, a GPU (Graphics Processing Unit) 212, a display controller 213, a hard key controller 214, a sensor controller 215, a contact sensor 216, an electronic pen controller 217, a display 220, hard keys 221, a sensor I / F 218, and a distance sensor 226. The operations of the above-described devices other than the sensor I / F 218 and the distance sensor 226 are the same as those of the first embodiment.

[0124] The sensor I / F 218 is an interface circuit for receiving a distance image measured by the distance sensor 226 .

[0125] The distance sensor 226 is a sensor that measures the distance to an object such as the user's pointing object 30. Specifically, the distance sensor 226 has a structure in which 500 x 500 pairs of infrared laser diodes and light receiving elements are arranged side by side, and measures the distance to the object from the time between when the infrared laser diode emits light and when it receives the reflected light, and outputs the distance as a distance image. The distance sensor 226 is installed, for example, on the top or bottom of the IWB 10b.

[0126] (Configuration and operation of IWB functional blocks) 20 is a diagram showing an example of the configuration of functional blocks of an IWB according to the third embodiment. The configuration and operation of the functional blocks of an IWB 10b according to this embodiment will be described with reference to FIG.

[0127] 20, the IWB 10b includes an image receiving unit 101 (receiving unit), a transformation matrix calculation unit 102, a pointed position detection unit 103 (detection unit), a determination unit 104 (first determination unit), a display control unit 105, a storage unit 106, a display condition determination unit 107 (second determination unit), and a gesture recognition unit 108 (recognition unit). Note that the operations of the image receiving unit 101, the transformation matrix calculation unit 102, the pointed position detection unit 103, the determination unit 104, the display control unit 105, the storage unit 106, and the display condition determination unit 107 are the same as those in the second embodiment described above.

[0128] The gesture recognition unit 108 is a functional unit that acquires distance image data measured by the distance sensor 226 via the sensor I / F 218 and recognizes the gesture of the pointing object 30 from the distance image. Specifically, the gesture recognition unit 108 detects a moving object from the acquired distance image and compares the shape and color of the object with pre-stored hand shape and color data to recognize that the object is a gesture such as a hand with one finger outstretched or a hand with two fingers outstretched. This recognition process may be performed, for example, by machine learning using shape data and color data of many hands.

[0129] The image receiving unit 101, transformation matrix calculation unit 102, pointed position detection unit 103, determination unit 104, display control unit 105, display condition determination unit 107, and gesture recognition unit 108 are realized by executing a program by CPU 201 shown in Fig. 2. Note that at least some of the image receiving unit 101, transformation matrix calculation unit 102, pointed position detection unit 103, determination unit 104, display control unit 105, display condition determination unit 107, and gesture recognition unit 108 may be realized by a hardware circuit such as an FPGA or an ASIC.

[0130] Furthermore, although the image receiving unit 101, transformation matrix calculation unit 102, indication position detection unit 103, judgment unit 104, display control unit 105, memory unit 106, display condition judgment unit 107 and gesture recognition unit 108 are all assumed to be realized within IWB 10b, this is not limited to this, and at least any one of these may be realized in another information processing device.

[0131] Furthermore, the functions of each functional unit of the IWB 10b shown in Fig. 20 are conceptually shown, and the configuration is not limited to this. That is, each functional unit of the IWB 10b does not need to be configured as a clear software module as the block shown in Fig. 20, but the functions of each functional unit as a whole may be realized by executing a program on the IWB 10b. For example, multiple functional units illustrated as independent functional units in the IWB 10b shown in Fig. 20 may be configured as a single functional unit. On the other hand, the function of one functional unit in the IWB 10b shown in Fig. 20 may be divided into multiple functional units and configured as multiple functional units.

[0132] (Flow of IWB software menu display operations) 21 is a flowchart showing an example of the flow of the software menu display operation of the IWB according to the third embodiment. The flow of the software menu display operation of the IWB 10b according to this embodiment will be described with reference to FIG.

[0133] <Steps S61 to S65> The processes of steps S61 to S65 are the same as the processes of steps S11 to S15 shown in Fig. 13. If the designated position is within the area of ​​the hard key 221 (step S65: Yes), the process proceeds to step S66, and if it is outside the area (step S65: No), the process returns to step S61.

[0134] <Step S66> The gesture recognition unit 108 of the IWB 10b acquires distance image data measured by the distance sensor 226 via the sensor I / F 218 and recognizes the gesture of the pointing object 30 from the distance image. The display condition determination unit 107 then determines whether the gesture recognized by the gesture recognition unit 108 is a predetermined gesture. If the recognized gesture is a predetermined gesture (step S66: Yes), the process proceeds to step S67. If the recognized gesture is not a predetermined gesture (step S66: No), the process returns to step S61, and the software menu 401 is not displayed. Note that the predetermined gesture is a gesture for displaying a predetermined software menu 401, such as a gesture in which, after pointing within the area of ​​the hard keys 221, the pointing is stopped and then the pointing is performed again immediately (for example, within one second) (for example, a gesture in which a fingertip is pointed up and then pointed again as if tapping a button), or a gesture in which one finger is changed to two fingers.

[0135] <Step S67> The display control unit 105 displays the software menu 401 in the display area of ​​the display 220 near the hard keys 221. Then, the software menu display operation ends.

[0136] (IWB software menu execution flow) Fig. 22 is a flowchart showing an example of the flow of the execution operation of the software menu of the IWB according to the third embodiment. The flow of the execution operation of the software menu of the IWB 10b according to this embodiment will be described with reference to Fig. 22. Note that the description here is based on the premise that the software menu 401 corresponding to the hard key 221 is already displayed on the display 220.

[0137] <Steps S71 to S75> The processes of steps S71 to S75 are the same as the processes of steps S21 to S25 shown in Fig. 14. If the designated position is on a button of the software menu 401 (step S75: Yes), the process proceeds to step S76, and if not on the button (step S75: No), the process returns to step S71.

[0138] <Step S76> The gesture recognition unit 108 of the IWB 10b acquires distance image data measured by the distance sensor 226 via the sensor I / F 218 and recognizes the gesture of the pointing object 30 from the distance image. Then, the display condition determination unit 107 determines whether the gesture recognized by the gesture recognition unit 108 is a predetermined gesture. If the recognized gesture is a predetermined gesture (step S76: Yes), the process proceeds to step S77. If the recognized gesture is not a predetermined gesture (step S76: No), the process returns to step S71, and the function corresponding to the button on the software menu 401 is not executed. Note that the predetermined gesture is a gesture for executing a predetermined function of a button on the software menu 401.

[0139] <Step S77> When the display condition determination unit 107 determines that the gesture recognized by the gesture recognition unit 108 is a predetermined gesture, the IWB 10b determines that the button on the software menu 401 at the designated position has been pressed (selected), and executes the function of the button on the hard key 221 corresponding to that button. Note that the content of the function executed when the button at the designated position has been pressed (selected) is not particularly limited. Then, the execution operation of the software menu is terminated.

[0140] As described above, in the IWB 10b according to this embodiment, the gesture recognition unit 108 recognizes the gesture of the pointing object 30, the display condition determination unit 107 determines whether the gesture recognized by the gesture recognition unit 108 is a predetermined gesture, and the display control unit 105, when the display condition determination unit 107 determines that the gesture recognized by the gesture recognition unit 108 is a predetermined gesture, causes the software menu 401 to be displayed on the display 220, and the display condition determination unit 107 determines whether the gesture recognized by the gesture recognition unit 108 is a predetermined gesture, and when the display condition determination unit 107 determines that the gesture recognized by the gesture recognition unit 108 is a predetermined gesture, the display control unit 105 executes the function of the button of the hard key 221 corresponding to the button of the software menu 401 that is determined by the determination unit 104 to be on the pointing position. This not only achieves the same effects as the first embodiment described above, but also prevents the display 220 from being unintentionally turned off or the software menu 401 from being displayed, and also reduces the annoyance of waiting times.

[0141] [Fourth embodiment] The IWB according to the fourth embodiment will be described, focusing on the differences from the IWB 10 according to the first embodiment. In this embodiment, an operation of directly executing a function corresponding to a hard key at a specified position, rather than displaying a software menu, will be described. Note that the overall configuration of the information processing system according to this embodiment, as well as the hardware configuration and functional block configuration of the IWB, are the same as those described in the first embodiment.

[0142] (Flow of IWB hard key function execution operations) 23 is a flowchart showing an example of the flow of operations for executing the functions of the hard keys of the IWB according to the fourth embodiment. The flow of operations for executing the functions of the hard keys of the IWB 10 according to the present embodiment will be described with reference to FIG.

[0143] <Steps S81 to S85> The processes of steps S81 to S85 are the same as the processes of steps S11 to S15 shown in Fig. 13. If the designated position is within the area of ​​the hard key 221 (step S85: Yes), the process proceeds to step S86, and if it is outside the area (step S85: No), the process returns to step S81.

[0144] <Step S86> The IWB 10 executes the function corresponding to the button, assuming that the button of the hard key 221 at the specified position has been pressed (selected). Note that the function here refers to a function executed when any of the hard keys 221 is pressed, such as a function to turn on the power of the display 220 and return from standby mode when the IWB 10 is in standby mode and the power of the display 220 is off. Then, the execution operation of the function of the hard key is terminated.

[0145] As described above, the IWB 10 according to this embodiment executes a function corresponding to the button of the hard key 221 at the designated position when the determination unit 104 determines that the designated position is within the area of ​​the hard key 221. This allows the function of the hard key 221 of the IWB 10 to be instantly operated from a remote location.

[0146] In each of the above-described embodiments, when at least one of the functional units of the IWBs 10, 10a, and 10b is realized by executing a program, the program is provided by being pre-installed in a ROM or the like. In each of the above-described embodiments, the program executed by the IWBs 10, 10a, and 10b may be provided by being recorded in an installable or executable file format on a computer-readable recording medium such as a CD-ROM (Compact Disc Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk-Recordable), or a DVD (Digital Versatile Disc). In each of the above-described embodiments, the program executed by the IWBs 10, 10a, and 10b may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. In each of the above-described embodiments, the program executed by the IWBs 10, 10a, and 10b may be provided or distributed via a network such as the Internet. Furthermore, in each of the above-mentioned embodiments, the program executed by IWB10, 10a, 10b has a modular structure including at least one of the above-mentioned functional units, and in terms of actual hardware, CPU201 reads and executes the program from the above-mentioned storage device (e.g., ROM202 or SSD204, etc.), thereby loading and generating each of the above-mentioned functional units onto the main storage device (RAM203).

[0147] The aspects of the present invention are as follows. <1> An information processing device, a receiving unit that receives a captured image of the pointing object from an imaging device that captures an image of the pointing object; a detection unit that detects a pointing position pointed to by the pointing object from the captured image received by the receiving unit; a first determination unit that determines whether the pointing position detected by the detection unit is within an area of ​​a hardware key installed in the electronic device; Equipped with The information processing device executes a function corresponding to a button of the hardware key, at least on condition that the first determination unit determines that the indication position is within an area of ​​the hardware key. <2> a display control unit that, when the first determination unit determines that the pointing position is within the area of ​​the hardware key, displays on a display device of the electronic device a software menu for executing a function of a button of the hardware key. <1> The information processing device is described in <3> a second determination unit that determines whether the pointing position has remained within the area of ​​the hardware key for a predetermined period of time or more; the display control unit causes the display device to display the software menu when the second determination unit determines that the pointing position has remained within an area of ​​the hardware key for the predetermined time or longer. <2> The information processing device is described in <4> a recognition unit that recognizes a gesture of the pointing object; a second determination unit that determines whether the gesture recognized by the recognition unit is a predetermined gesture; Furthermore, the display control unit causes the display device to display the software menu when the second determination unit determines that the gesture recognized by the recognition unit is the predetermined gesture. <2> The information processing device is described in <5> the first determination unit determines whether the pointing position is on a button of the software menu; When the first determination unit determines that the pointing position is on a button of the software menu, the information processing device executes a function of the button of the hardware key corresponding to the button. <2> The information processing device is described in <6> a second determination unit that determines whether the pointing position has remained on a button of the software menu for a predetermined period of time or more; When the second determination unit determines that the pointing position has remained on a button of the software menu for the predetermined time or longer, the information processing device executes a function of the button of the hardware key corresponding to the button. <5> The information processing device is described in <7> a recognition unit that recognizes a gesture of the pointing object; a second determination unit that determines whether the gesture recognized by the recognition unit is a predetermined gesture; Furthermore, when the second determination unit determines that the gesture recognized by the recognition unit is the predetermined gesture, the display control unit executes a function of a button of the hardware key corresponding to the button that corresponds to the pointing position determined by the first determination unit to be on a button of the software menu. <5> The information processing device is described in <8> the display control unit hides the software menu when a predetermined time has elapsed without any button being designated on the software menu. <2> ~ <7> 1 is an information processing device according to any one of the preceding claims. <9> When the first determination unit determines that the pointing position is within the area of ​​the hardware key, the information processing device executes a function corresponding to a button of the hardware key at the pointing position. <1> The information processing device is described in <10> a receiving step of receiving a captured image of the pointing object from an imaging device that captures an image of the pointing object; a detection step of detecting a pointing position pointed to by the pointing object from the received captured image; a determining step of determining whether the detected pointing position is within an area of ​​a hardware key installed on the electronic device; an execution step of executing a function corresponding to a button of the hardware key, at least on condition that it is determined in the determination step that the pointing position is within an area of ​​the hardware key; The information processing device has the following. <11> On the computer, a receiving step of receiving a captured image of the pointing object from an imaging device that captures an image of the pointing object; a detection step of detecting a pointing position pointed to by the pointing object from the received captured image; a determining step of determining whether the detected pointing position is within an area of ​​a hardware key installed on the electronic device; an execution step of executing a function corresponding to a button of the hardware key, at least on condition that it is determined in the determination step that the pointing position is within an area of ​​the hardware key; This is a program for executing the above. [Explanation of symbols]

[0148] 1. Information Processing Systems 10, 10a, 10b IWB 20a, 20b camera 30 Pointing object 101 Image receiving unit 102 Transformation matrix calculation unit 103 Pointing position detection unit 104 Judgment section 105 Display control unit 106 Storage section 107 Display condition judgment section 108 Gesture Recognition Unit 201 CPU 202 ROM 203 RAM 204 SSD 205 Network I / F 206 External device connection I / F 210 Bus 211 Capture Device 212 GPU 213 Display Controller 214 Hard Key Controller 215 Sensor Controller 216 Contact Sensor 217 Electronic Pen Controller 218 Sensor I / F 219 Near field communication circuit 219a Antenna 220 Display 221 Hard Key 222 Electronic Pen 223 Mike 224 Speaker 225 USB memory 226 Distance Sensor 301 PC 302 fingers 401, 402 Software Menu IML and IMR images IP indication position P pointer [Prior art documents] [Patent documents]

[0149] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-071510

Claims

1. An information processing device, a receiving unit that receives a captured image of the pointing object from an imaging device that captures an image of the pointing object; a detection unit that detects a pointing position pointed to by the pointing object from the captured image received by the receiving unit; a first determination unit that determines whether the pointing position detected by the detection unit is within an area of ​​a hardware key installed in the electronic device; Equipped with The information processing device executes a function corresponding to a button of the hardware key, at least on condition that the first determination unit determines that the pointing position is within an area of ​​the hardware key.

2. 2. The information processing device according to claim 1, further comprising a display control unit that, when the first determination unit determines that the indication position is within the area of ​​the hardware key, causes a software menu to be displayed on a display device of the electronic device for executing the function of a button of the hardware key.

3. a second determination unit that determines whether the pointing position has remained within the area of ​​the hardware key for a predetermined period of time or more; The information processing device according to claim 2 , wherein the display control unit causes the display device to display the software menu when the second determination unit determines that the pointing position has remained within the area of ​​the hardware key for the predetermined time or longer.

4. a recognition unit that recognizes a gesture of the pointing object; a second determination unit that determines whether the gesture recognized by the recognition unit is a predetermined gesture; Furthermore, The information processing apparatus according to claim 2 , wherein the display control unit causes the display device to display the software menu when the second determination unit determines that the gesture recognized by the recognition unit is the predetermined gesture.

5. the first determination unit determines whether the pointing position is on a button of the software menu; The information processing apparatus according to claim 2 , wherein when the first determination unit determines that the pointing position is on a button of the software menu, the information processing apparatus executes a function of a button of the hardware key corresponding to the button.

6. a second determination unit that determines whether the pointing position has remained on a button of the software menu for a predetermined period of time or more; The information processing device according to claim 5, wherein when the second determination unit determines that the pointing position has remained on a button of the software menu for more than the predetermined time, the information processing device executes the function of the button of the hardware key corresponding to the button.

7. a recognition unit that recognizes a gesture of the pointing object; a second determination unit that determines whether the gesture recognized by the recognition unit is a predetermined gesture; Furthermore, 6. The information processing device according to claim 5, wherein when the second determination unit determines that the gesture recognized by the recognition unit is the predetermined gesture, the display control unit executes a function of a button of the hardware key corresponding to the button corresponding to the indication position determined by the first determination unit to be on a button of the software menu.

8. 8. The information processing apparatus according to claim 5, wherein the display control unit makes the software menu invisible when a predetermined time has elapsed without any button on the software menu being designated.

9. The information processing device according to claim 1 , wherein when the first determination unit determines that the pointing position is within the area of ​​the hardware key, the information processing device executes a function corresponding to a button of the hardware key at the pointing position.

10. a receiving step of receiving a captured image of the pointing object from an imaging device that captures an image of the pointing object; a detection step of detecting a pointing position pointed to by the pointing object from the received captured image; a determining step of determining whether the detected pointing position is within an area of ​​a hardware key installed on the electronic device; an execution step of executing a function corresponding to a button of the hardware key, at least on condition that it is determined in the determination step that the pointing position is within an area of ​​the hardware key; An information processing method comprising:

11. On the computer, a receiving step of receiving a captured image of the pointing object from an imaging device that captures an image of the pointing object; a detection step of detecting a pointing position pointed to by the pointing object from the received captured image; a determining step of determining whether the detected pointing position is within an area of ​​a hardware key installed on the electronic device; an execution step of executing a function corresponding to a button of the hardware key, at least on condition that it is determined in the determination step that the pointing position is within an area of ​​the hardware key; A program to execute.

Images