Interactive system, position detection method, and input area setting method

Abstract

This system provides an interactive system that detects the position of a target object without being affected by ultrasonic noise in the environment. [Solution] The interactive system comprises a display device that displays an image, an indicator that emits electromagnetic waves, a first sensor that receives electromagnetic waves, a second sensor that receives electromagnetic waves, and one or more processors. The one or more processors perform the following: calculate a first distance, which is the distance between the indicator and the first sensor, based on the output from the first sensor; calculate a second distance, which is the distance between the indicator and the second sensor, based on the output from the second sensor; calculate the position of the indicator based on the first distance, the second distance, and the positional relationship between the first and second sensors; and display an image on the display device based on the position of the indicator.

Description

Technical Field

[0001] The present invention relates to an interactive system, a position detection method, and an input area setting method.

Background Art

[0002] Patent Document 1 discloses a projector including a light receiving unit that detects light output from an object, two sound wave receiving units that receive sound waves output from the object, a distance acquisition unit that acquires the distance to a projection surface, and a distance calculation unit that calculates the position of the object based on the outputs from these units.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technique described in Patent Document 1, since ultrasonic waves are used to detect the position of an object such as an electronic pen, it is affected by ultrasonic noise included in the environment.

Means for Solving the Problems

[0005] An interactive system according to one aspect of the present invention comprises a display device for displaying an image, an indicator for transmitting electromagnetic waves, a first sensor for receiving the electromagnetic waves, a second sensor for receiving the electromagnetic waves, and one or more processors, wherein the one or more processors perform the following: calculate a first distance, which is the distance between the indicator and the first sensor, based on the output from the first sensor; calculate a second distance, which is the distance between the indicator and the second sensor, based on the output from the second sensor; calculate the position of the indicator based on the first distance, the second distance, and the positional relationship between the first and second sensors; and display an image on the display device based on the position of the indicator.

[0006] An interactive system according to one aspect of the present invention comprises an indicator that transmits electromagnetic waves, a first sensor that receives the electromagnetic waves, a second sensor that receives the electromagnetic waves, and one or more processors, wherein the one or more processors perform the following: calculate a first distance, which is the distance between the indicator and the first sensor, based on the output from the first sensor; calculate a second distance, which is the distance between the indicator and the second sensor, based on the output from the second sensor; calculate the position of the indicator based on the first distance, the second distance, and the positional relationship between the first and second sensors; accept a first operation to specify three or more positions using the indicator; and set an input area, which is an area in which processing corresponding to the position of the indicator is performed by the one or more processors, based on the specified three or more positions.

[0007] A position detection method according to one embodiment of the present invention is a position detection method performed by one or more processors, comprising: calculating a first distance, which is the distance between an indicator and the first sensor, based on the output from a first sensor that receives electromagnetic waves from an indicator; calculating a second distance, which is the distance between the indicator and the second sensor, based on the output from a second sensor that receives electromagnetic waves from the indicator; and calculating the position of the indicator based on the first distance, the second distance, and the positional relationship between the first sensor and the second sensor.

[0008] An input region setting method according to one embodiment of the present invention is an input region setting method performed by one or more processors, comprising: calculating a first distance, which is the distance between an indicator and the first sensor, based on the output from a first sensor that receives electromagnetic waves from an indicator; calculating a second distance, which is the distance between the indicator and the second sensor, based on the output from a second sensor that receives electromagnetic waves from the indicator; calculating the position of the indicator based on the first distance, the second distance, and the positional relationship between the first sensor and the second sensor; receiving a first operation to specify three or more positions by the indicator; and setting an input region, which is a region in which processing corresponding to the position of the indicator is performed by the one or more processors, based on the specified three or more positions. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram shows the usage patterns of the interactive system of this embodiment. [Figure 2] This is a block diagram showing the configuration of a projector, a communication device, and an electronic pen. [Figure 3] This diagram shows the configuration of the optical device in a projector. [Figure 4] This flowchart shows the pen position calculation process performed by the first processor. [Figure 5]This flowchart shows the input area setting process performed by the first processor. [Figure 6] This figure shows an example where the first image, which includes four marks positioned at different locations, is displayed on the projection surface. [Figure 7] This is a diagram showing the object rendering process performed by the first processor. [Figure 8] This is a diagram showing an example of the second image. [Figure 9] This figure shows an example of a second image in which an object is displayed in the display area. [Modes for carrying out the invention]

[0010] The embodiments of this disclosure will be described below with reference to the drawings. In the following drawings, the scale of each component may differ from that of the actual component in order to make each component recognizable.

[0011] Figure 1 shows the usage configuration of the interactive system 1 of this embodiment. As shown in Figure 1, the interactive system 1 comprises a projector 2, a communication device 3, and an electronic pen 4.

[0012] Projector 2 displays an image 110 on the projection surface 100 by projecting an image light LC onto the projection surface 100 based on a video signal. The projection surface 100 may be a dedicated projection screen or a wall surface, etc. Projector 2 is equipped with a first wireless module 26 for wireless communication with an external device. For example, the first wireless module 26 communicates wirelessly with the electronic pen 4 according to a short-range wireless communication standard such as Bluetooth®. Projector 2 is an example of a display device.

[0013] Communication device 3 controls communication with the network. For example, communication device 3 is a dongle device equipped with a wireless module. For example, communication device 3 is connected to the HDMI (High-Definition Multimedia Interface: registered trademark) port of projector 2. Communication device 3 includes a second wireless module 31 for communicating with the network. For example, the second wireless module 31 accesses the internet via a wireless LAN (Local Area Network) that supports wireless communication standards such as Wi-Fi (registered trademark) and receives video signals from a video distribution server on the internet. Communication device 3 transmits the video signals received from the video distribution server via the second wireless module 31 to projector 2. Also, for example, the second wireless module 31 communicates wirelessly with the electronic pen 4 according to a short-range wireless communication standard such as Bluetooth (registered trademark).

[0014] The electronic pen 4 is a pointing device used by the user to operate the GUI (Graphical User Interface) displayed on the projection surface 100, or to draw objects on the electronic whiteboard displayed on the projection surface 100. Although not shown in Figure 1, the electronic pen 4 includes a third wireless module 41. For example, the third wireless module 41 communicates wirelessly with the projector 2 and the communication device 3 according to a short-range wireless communication standard such as Bluetooth®. In other words, the electronic pen 4 transmits radio waves when communicating wirelessly with the projector 2 and the communication device 3. The electronic pen 4 is an example of an indicator, and radio waves are an example of electromagnetic waves.

[0015] The first wireless module 26 and the second wireless module 31 receive radio waves transmitted from the electronic pen 4 and output intensity data indicating the strength of the received radio waves. As an example, the intensity data is RSSI (Received Signal Strength Indicator). The first wireless module 26 is an example of a first sensor, and the second wireless module 31 is an example of a second sensor. In the following description, the intensity data output from the first wireless module 26 may be referred to as "first intensity data," and the intensity data output from the second wireless module 31 may be referred to as "second intensity data."

[0016] As will be described in detail later, Projector 2 calculates a first distance, which is the distance between the electronic pen 4 and the first wireless module 26, based on first intensity data output from the first wireless module 26. Projector 2 calculates a second distance, which is the distance between the electronic pen 4 and the second wireless module 31, based on second intensity data output from the second wireless module 31. Projector 2 calculates the relative position of the electronic pen 4 with respect to the first wireless module 26 and the second wireless module 31, based on the first distance, the second distance, and the positional relationship between the first wireless module 26 and the second wireless module 31. The positional relationship here includes the distance between the first wireless module 26 and the second wireless module 31, and the relative positional relationship indicating whether the first wireless module 26 is located to the right or left of the second wireless module 31, as viewed from the user's position holding the electronic pen 4. Preferably, the positional relationship is set by the user as to whether the first wireless module 26 is located to the right or left of the second wireless module 31, as viewed from the user's position holding the electronic pen 4. When considered in a plane, if relative positional relationships are not taken into account, the position of the electronic pen 4 will be calculated as two points. However, if the relative positional relationship between the first wireless module 26 and the second wireless module 31 is determined to be a single point from the user's perspective, the calculated position of the electronic pen 4 will be determined to be a single point.

[0017] The projector 2 calculates the position of the electronic pen 4 in the input area 210 set on the operation surface 200, and performs image processing based on the position of the electronic pen 4. For example, the projector 2 performs a process of displaying an object such as a graphic, a character, or a symbol at the position of the whiteboard displayed on the projection surface 100 corresponding to the position of the electronic pen 4 in the input area 210. Also, for example, the projector 2 performs a process of displaying an object such as a line along the trajectory of the electronic pen 4 in the input area 210 on the whiteboard.

[0018] For example, the operation surface 200 is the surface of the top plate of a desk. The operation surface 200 is not limited to the surface of the top plate of a desk, and may be the surface of other objects. The operation surface 200 may be a horizontal surface or an inclined surface. The operation surface 200 may be a flat surface or a curved surface. However, in order to perform the above image processing based on the position of the electronic pen 4, it is necessary to associate the coordinate system of the input area 210 set on the operation surface 200 with the coordinate system of the image 110 displayed on the projection surface 100. The method of setting the input area 210 will be described later.

[0019] FIG. 2 is a block diagram showing the configurations of the projector 2, the communication device 3, and the electronic pen 4. As shown in FIG. 2, the projector 2 includes an optical device 10, an input device 25, a first wireless module 26, a first storage device 27, and a first processor 28.

[0020] The optical device 10 generates image light LC and projects it onto the projection surface 100 under the control of the first processor 28. FIG. 3 is a diagram showing an example of the configuration of the optical device 10. As shown in FIG. 3, the optical device 10 includes a light source 11, two dichroic mirrors 12 and 13, three reflection mirrors 14, 15 and 16, five relay lenses 17, 18, 19, 20 and 21, three liquid crystal panels 22R, 22G and 22B, a dichroic prism 23, and a projection optical system 24.

[0021] The light source 11 emits white light L0 to the dichroic mirror 12. The light source 11 is, for example, a halogen lamp, a mercury lamp, a light-emitting diode, or a laser light source.

[0022] The dichroic mirror 12 separates white light L0 into primary light L1 and secondary light L2. For example, primary light L1 is red light, and secondary light L2 is a mixture of green and blue light. The dichroic mirror 12 emits primary light L1 to the reflective mirror 14 and secondary light L2 to the dichroic mirror 13.

[0023] The dichroic mirror 13 separates the second color light L2 into the third color light L3 and the fourth color light L4. For example, the third color light L3 is green light, and the fourth color light L4 is blue light. The dichroic mirror 13 emits the third color light L3 to the relay lens 18 and the fourth color light L4 to the relay lens 19.

[0024] The first color light L1 emitted from the dichroic mirror 12 is incident on the liquid crystal panel 22R via the reflective mirror 14 and relay lens 17. The third color light L3 emitted from the dichroic mirror 13 is incident on the liquid crystal panel 22G via the relay lens 18. The fourth color light L4 emitted from the dichroic mirror 13 is incident on the liquid crystal panel 22B via the relay lens 19, reflective mirror 15, relay lens 20, reflective mirror 16, and relay lens 21.

[0025] The liquid crystal panels 22R, 22G, and 22B function as light modulation devices in the projector 2. For example, the liquid crystal panels 22R, 22G, and 22B are active-drive liquid crystal panels having multiple pixels arranged in a matrix. Liquid crystal panel 23R modulates the first color light L1, which is red. Liquid crystal panel 23G modulates the third color light L3, which is green. Liquid crystal panel 23B modulates the fourth color light L4, which is blue. The first processor 28 controls the transmittance of the pixels in each of the liquid crystal panels 22R, 22G, and 22B based on the video signal.

[0026] The dichroic prism 23 generates image light LC representing a color image by combining the first color light L1 modulated by the liquid crystal panel 23R, the third color light L3 modulated by the liquid crystal panel 23G, and the fourth color light L4 modulated by the liquid crystal panel 23B. The dichroic prism 23 emits the image light LC to the projection optical system 24.

[0027] The projection optical system 24 is composed of multiple optical elements such as lenses, and projects the image light LC emitted from the dichroic prism 23 toward the projection surface 100. Although not shown in the diagram, the projection optical system 24 is provided with a mechanism that can adjust optical parameters such as lens shift amount, lens focus amount, and lens zoom amount. These mechanisms are controlled by the first processor 28, thereby adjusting the optical parameters of the projection optical system 24.

[0028] Let's return to Figure 2 and continue the explanation. The input device 25 is a device that accepts user input. The input device 25 includes an operation unit 25a and a light receiving unit 25b. The operation unit 25a consists of a plurality of operation keys provided on the projector 2. For example, the operation keys include a power key, a menu call key, directional keys, a select key, and a volume adjustment key. The operation keys may be hardware keys or software keys displayed on a touch panel. The operation unit 25a outputs the electrical signals generated when each operation key is operated by the user as operation signals to the first processor 28.

[0029] The light-receiving unit 25b includes a photoelectric conversion circuit that receives infrared light transmitted from the remote controller (not shown) of the projector 2 and converts it into an electrical signal. The light-receiving unit 25b outputs the electrical signal obtained by the photoelectric conversion of infrared light to the first processor 28 as a remote operation signal. The remote controller is provided with multiple operation keys, similar to the operation unit 25a. The remote controller converts the electrical signals generated when each operation key on the remote controller is operated by the user into infrared light and transmits it to the projector 2. In other words, the remote operation signal output from the light-receiving unit 25b is substantially the same as the electrical signals generated when each operation key on the remote controller is operated by the user.

[0030] The first wireless module 26, under the control of the first processor 28, communicates wirelessly with the electronic pen 4 in accordance with a short-range wireless communication standard such as Bluetooth®. The first wireless module 26 receives radio waves transmitted from the electronic pen 4 and outputs first intensity data indicating the strength of the received radio waves to the first processor 28.

[0031] The first storage device 27 includes a non-volatile memory that stores programs and various setting data necessary for the first processor 28 to execute various processes, and a volatile memory that is used as a temporary storage location for data when the first processor 28 executes various processes. For example, the non-volatile memory is flash memory, and the volatile memory is RAM (Random Access Memory).

[0032] The first processor 28 is an arithmetic processing unit that controls the overall operation of the projector 2 according to a program stored in the first storage device 27. For example, the first processor 28 is composed of one or more processors such as a CPU (Central Processing Unit). Some or all of the functions of the first processor 28 may be composed of circuits such as a DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), and FPGA (Field Programmable Gate Array). The first processor 28 executes various processes in parallel or sequentially.

[0033] The communication device 3 comprises a second wireless module 31, a second storage device 32, and a second processor 33. As already described, the communication device 3 is connected to the HDMI port of the projector 2. In other words, the communication device 3 is connected to the projector 2 in a communicative manner.

[0034] The second wireless module 31, under the control of the second processor 33, accesses the internet via wireless LAN and receives video signals from a video distribution server on the internet. The second wireless module 31 outputs the video signals received from the video distribution server to the second processor 33. The second wireless module 31 also communicates wirelessly with the electronic pen 4 under the control of the second processor 33, in accordance with a short-range wireless communication standard such as Bluetooth (registered trademark). The second wireless module 31 receives radio waves transmitted from the electronic pen 4 and outputs second strength data indicating the strength of the received radio waves to the second processor 33.

[0035] The second storage device 32 includes a non-volatile memory that stores programs and various setting data necessary for the second processor 33 to execute various processes, and a volatile memory that is used as a temporary storage location for data when the second processor 33 executes various processes.

[0036] The second processor 33 is an arithmetic processing unit that controls the overall operation of the communication device 3 according to a program stored in the second storage device 32. For example, the second processor 33 is composed of one or more processors such as a CPU. Some or all of the functions of the second processor 33 may be composed of circuits such as a DSP, ASIC, PLD, and FPGA. The second processor 33 executes various processes in parallel or sequentially. The second processor 33 can communicate with the first processor 28 of the projector 2.

[0037] The electronic pen 4 comprises a third wireless module 41, a button 42, a third storage device 43, and a third processor 44.

[0038] The third wireless module 41, under the control of the third processor 44, communicates wirelessly with the first wireless module 26 of the projector 2 and the second wireless module 31 of the communication device 3 in accordance with short-range wireless communication standards such as Bluetooth®. In other words, the electronic pen 4 transmits radio waves when communicating wirelessly with the first wireless module 26 of the projector 2 and the second wireless module 31 of the communication device 3.

[0039] Button 42 outputs a signal to the third processor 44 indicating whether or not the user is performing an input operation using the electronic pen 4. For example, if button 42 is pressed, that is, if the user is performing an input operation using the electronic pen 4, button 42 outputs a first signal having a first level to the third processor 44. On the other hand, if button 42 is not pressed, that is, if the user is not performing an input operation using the electronic pen 4, button 42 outputs a second signal having a second level to the third processor 44. Of the first and second levels, one is a high level and the other is a low level.

[0040] The third storage device 43 includes a non-volatile memory that stores programs and various setting data necessary for the third processor 44 to execute various processes, and a volatile memory that is used as a temporary storage location for data when the third processor 44 executes various processes.

[0041] The third processor 44 is an arithmetic processing unit that controls the overall operation of the electronic pen 4 according to a program stored in the third storage device 43. For example, the third processor 44 is composed of one or more processors such as a CPU. Some or all of the functions of the third processor 44 may be composed of circuits such as a DSP, ASIC, PLD, and FPGA. The third processor 44 executes various processes in parallel or sequentially.

[0042] Figure 4 is a flowchart showing the pen position calculation process performed by the first processor 28 of the projector 2 when the first processor 28 calculates the position of the electronic pen 4. The first processor 28 performs the pen position calculation process shown in Figure 4 by executing a program stored in the first storage device 27.

[0043] Furthermore, when the third processor 44 of the electronic pen 4 receives a first signal from the button 42, it controls the third wireless module 41 to transmit radio waves containing first information indicating that the user is performing an input operation using the electronic pen 4 to the first wireless module 26 of the projector 2 and the second wireless module 31 of the communication device 3. The first wireless module 26 of the projector 2 outputs first intensity data indicating the strength of the radio waves received from the electronic pen 4, and the first information contained in the radio waves received from the electronic pen 4, to the first processor 28. When the first processor 28 receives the first information from the first wireless module 26, it recognizes that the user is performing an input operation using the electronic pen 4 and executes a pen position calculation process.

[0044] As shown in Figure 4, when the first processor 28 starts the pen position calculation process, it first calculates a first distance, which is the distance between the electronic pen 4 and the first wireless module 26, based on the first intensity data output from the first wireless module 26 (step S1). Since the method for calculating the distance between two points from the received radio wave intensity is generally known, it will not be described herein.

[0045] Next, the first processor 28 calculates a second distance, which is the distance between the electronic pen 4 and the second wireless module 31, based on the second intensity data output from the second wireless module 31 (step S2). When the second wireless module 31 of the communication device 3 receives radio waves containing the first information from the electronic pen 4, it outputs to the second processor 33 the second intensity data indicating the strength of the received radio waves and the first information contained in the radio waves received from the electronic pen 4. If the second processor 33 has obtained the first information from the second wireless module 31, it transmits the second intensity data to the first processor 28 of the projector 2. As a result, when the first processor 28 executes step S2, it has obtained the second intensity data output from the second wireless module 31.

[0046] Next, the first processor 28 calculates the relative position of the electronic pen 4 to the first wireless module 26 and the second wireless module 31 based on the first distance, the second distance, and the positional relationship between the first wireless module 26 and the second wireless module 31 (step S3). The positional relationship between the first wireless module 26 and the second wireless module 31 is known and is stored in the first storage device 27 beforehand.

[0047] Figure 5 is a flowchart showing the input area setting process performed by the first processor 28 when the first processor 28 of the projector 2 sets the input area 210. The first processor 28 executes the input area setting process shown in Figure 5 by executing a program stored in the first storage device 27.

[0048] For example, the first processor 28 executes an input area setting process when it detects that it has received an operation requesting the setting of the input area 210 based on the output from the input device 25.

[0049] As shown in Figure 5, when the first processor 28 starts the input area setting process, it first causes the projector 2 to display a first image 120 containing three or more marks placed at different positions as image 110 (step S11). In other words, in step S11, the first processor 28 controls the liquid crystal panels 22R, 22G, and 22B of the optical device 10 so that the first image 120 containing three or more marks placed at different positions is displayed on the projection surface 100 as image 110.

[0050] Next, the first processor 28 receives a first operation in which the electronic pen 4 specifies three or more positions that correspond one-to-one with three or more marks (step S12). Subsequently, the first processor 28 sets an input area 210, which is an area in which the first processor 28 performs a process to calculate the position of the electronic pen 4 based on the specified three or more positions (step S13).

[0051] The first processor 28 then generates a correspondence between the coordinate system of the input region 210 and the coordinate system of the image 110 displayed on the projection surface 100 (step S14). For example, the correspondence between the coordinate system of the input region 210 and the coordinate system of the image 110 displayed on the projection surface 100 is coordinate transformation data for converting the coordinates of any point included in the input region 210 to the coordinates of any point included in the image 110.

[0052] The following section will explain the input area setting process in detail, referring to Figure 6. Figure 6 shows an example of a first image 120 containing four marks positioned at different locations being displayed on the projection surface 100. As shown in Figure 6, the first mark M1 is positioned in the upper right corner of the first image 120. The second mark M2 is positioned in the upper left corner of the first image 120. The third mark M3 is positioned in the lower right corner of the first image 120. The fourth mark M4 is positioned in the lower left corner of the first image 120.

[0053] For example, the first mark M1 is a mark that contains the number "1". The second mark M2 is a mark that contains the number "2". The third mark M3 is a mark that contains the number "3". The fourth mark M4 is a mark that contains the number "4".

[0054] For example, the first processor 28 controls the liquid crystal panels 22R, 22G, and 22B of the optical device 10 so that, with the first image 120 containing the four marks described above displayed, the message "Starting input area setup. First, please specify the position corresponding to mark number 1 with the electronic pen." is displayed within the first image 120.

[0055] Upon seeing the above message, the user moves the electronic pen 4 to the first position P1, which corresponds to the first mark M1 on the operating surface 200. The user can determine any position on the operating surface 200 as the first position P1. While holding the electronic pen 4 at the first position P1, the user presses the button 42 on the electronic pen 4. As a result, the first signal is output from the button 42.

[0056] As already explained, when the third processor 44 of the electronic pen 4 receives a first signal from the button 42, it controls the third wireless module 41 to transmit radio waves containing first information indicating that the user is performing an input operation using the electronic pen 4 to the first wireless module 26 of the projector 2 and the second wireless module 31 of the communication device 3. In this case, the first processor 28 of the projector 2 calculates the position of the electronic pen 4 by performing the pen position calculation process described above, and acquires the calculated position of the electronic pen 4 as the first position P1 corresponding to the first mark M1.

[0057] After acquiring the first position P1 corresponding to the first mark M1, the first processor 28 controls the liquid crystal panels 22R, 22G, and 22B of the optical device 10 so that the message "Next, please specify the position corresponding to the second mark with the electronic pen." is displayed in the first image 120.

[0058] Upon seeing the above message, the user moves the electronic pen 4 to the second position P2, which corresponds to the second mark M2 on the operating surface 200. The user can determine any position on the operating surface 200 as the second position P2. While holding the electronic pen 4 at the second position P2, the user presses the button 42 on the electronic pen 4. As a result, the first signal is output from the button 42.

[0059] Similarly, when the third processor 44 of the electronic pen 4 receives a first signal from the button 42, it controls the third wireless module 41 to transmit radio waves containing first information indicating that the user is performing an input operation using the electronic pen 4 to the first wireless module 26 of the projector 2 and the second wireless module 31 of the communication device 3. In this case, the first processor 28 of the projector 2 calculates the position of the electronic pen 4 by executing the pen position calculation process described above, and acquires the calculated position of the electronic pen 4 as the second position P2 corresponding to the second mark M2.

[0060] After acquiring the second position P2 corresponding to the second mark M2, the first processor 28 controls the liquid crystal panels 22R, 22G, and 22B of the optical device 10 so that the message "Next, please specify the position corresponding to mark number 3 with the electronic pen." is displayed in the first image 120.

[0061] Upon seeing the above message, the user moves the electronic pen 4 to the third position P3, which corresponds to the third mark M3 on the operating surface 200. The user can determine any position on the operating surface 200 as the third position P3. While holding the electronic pen 4 at the third position P3, the user presses the button 42 on the electronic pen 4. As a result, the first signal is output from the button 42.

[0062] Similarly, when the third processor 44 of the electronic pen 4 receives a first signal from the button 42, it controls the third wireless module 41 to transmit radio waves containing first information indicating that the user is performing an input operation using the electronic pen 4 to the first wireless module 26 of the projector 2 and the second wireless module 31 of the communication device 3. In this case, the first processor 28 of the projector 2 calculates the position of the electronic pen 4 by executing the pen position calculation process described above, and acquires the calculated position of the electronic pen 4 as the third position P3 corresponding to the third mark M3.

[0063] After acquiring the third position P3 corresponding to the third mark M3, the first processor 28 controls the liquid crystal panels 22R, 22G, and 22B of the optical device 10 so that the message "Finally, please specify the position corresponding to mark number 4 with the electronic pen." is displayed in the first image 120.

[0064] Upon seeing the above message, the user moves the electronic pen 4 to the fourth position P4, which corresponds to the fourth mark M4 on the operating surface 200. The user can determine any position on the operating surface 200 as the fourth position P4. While holding the electronic pen 4 at the fourth position P4, the user presses the button 42 on the electronic pen 4. As a result, the first signal is output from the button 42.

[0065] Similarly, when the third processor 44 of the electronic pen 4 receives a first signal from the button 42, it controls the third wireless module 41 to transmit radio waves containing first information indicating that the user is performing an input operation using the electronic pen 4 to the first wireless module 26 of the projector 2 and the second wireless module 31 of the communication device 3. In this case, the first processor 28 of the projector 2 calculates the position of the electronic pen 4 by performing the pen position calculation process described above, and acquires the calculated position of the electronic pen 4 as the fourth position P4 corresponding to the fourth mark M4.

[0066] As described above, the first processor 28 receives a first operation in which the electronic pen 4 specifies four positions that correspond one-to-one with the four marks contained in the first image 120, and then sets the region formed by connecting the four obtained positions with straight lines as the input region 210. The first processor 28 then generates coordinate transformation data as a correspondence between the coordinate system of the input region 210 and the coordinate system of the image 110 displayed on the projection surface 100. Here, when the user determines an arbitrary position on the operating surface 200 corresponding to a mark, they press the button 42 on the electronic pen 4 while holding the electronic pen 4 in the arbitrary position. However, the user is not limited to this, and for example, the user may determine the position corresponding to the mark by holding the electronic pen 4 in the arbitrary position corresponding to the mark for a predetermined time. In this example, the region formed by connecting four positions with straight lines was set as the input region 210. However, since a plane can be defined by determining three or more positions, the region formed by connecting three or five positions with straight lines could also be set as the input region 210.

[0067] After generating coordinate transformation data, the first processor 28 controls the liquid crystal panels 22R, 22G, and 22B of the optical device 10 so that the message "Input area setting complete." is displayed in the first image 120. After performing the above processing, the first processor 28 terminates the input area setting process.

[0068] Figure 7 is a flowchart showing the object display process performed by the first processor 28 of the projector 2. The first processor 28 executes the object display process shown in Figure 7 by executing a program stored in the first storage device 27.

[0069] For example, the first processor 28 executes object display processing when it detects that it has received an operation requesting to operate in whiteboard mode, based on the output from the input device 25.

[0070] As shown in Figure 7, when the first processor 28 starts object display processing, it first displays a second image 130, which includes a display area W1 for displaying an object, as image 110 on the projector 2 (step S21). In other words, in step S21, the first processor 28 controls the liquid crystal panels 22R, 22G, and 22B of the optical device 10 so that the second image 130, which includes a display area W1 for displaying an object, is displayed on the projection surface 100 as image 110. The second image 130 is an electronic whiteboard.

[0071] Figure 8 shows an example of the second image 130. As shown in Figure 8, the second image 130 includes a display area W1, which is a white area, and a toolbar area W2, which is the area where the toolbar TB is displayed. The toolbar TB includes a menu display icon C1, which is an operation icon for instructing the user to display a menu. Although not shown in Figure 8, the toolbar TB includes several operation icons other than the menu display icon C1.

[0072] Returning to Figure 7, the first processor 28 receives a second operation to draw an object in the input area 210 using the electronic pen 4 (step S22). Then, based on the second operation, the first processor 28 performs the process of displaying the object in the display area W1 of the second image 130 (step S23).

[0073] Figure 9 shows an example of a second image 130 in which an object is displayed in the display area W1. For example, when the first processor 28 receives a second operation, which is pressing the button 42 of the electronic pen 4 while the electronic pen 4 is held at a specific position in the input area 210, it performs the process of displaying an object such as a shape, character, or symbol at a position in the display area W1 that corresponds to the position of the electronic pen 4 in the input area 210. Also, for example, when the first processor 28 receives a second operation, which is moving the electronic pen 4 on the input area 210 while the button 42 of the electronic pen 4 is pressed, it performs the process of displaying an object such as a line in the display area W1 along the trajectory of the electronic pen 4 in the input area 210. Here, the first processor 28 performs processing when the button 42 of the electronic pen 4 is pressed, but it does not need to press the button 42 if it can determine that the electronic pen 4 is moving on the configured input area 210.

[0074] As described above, the interactive system 1 of this embodiment comprises a projector 2 for displaying an image 110, an electronic pen 4 for transmitting radio waves, a first wireless module 26 for receiving radio waves, a second wireless module 31 for receiving radio waves, and a first processor 28. The first processor 28 performs the following actions: calculate a first distance, which is the distance between the electronic pen 4 and the first wireless module 26, based on the output from the first wireless module 26; calculate a second distance, which is the distance between the electronic pen 4 and the second wireless module 31, based on the output from the second wireless module 31; calculate the position of the electronic pen 4 based on the first distance, the second distance, and the positional relationship between the first wireless module 26 and the second wireless module 31; and display an image based on the position of the electronic pen 4 on the projector 2. According to this embodiment as described above, since radio waves are used to calculate the position of the electronic pen 4, an interactive system 1 can be provided that is not affected by ultrasonic noise contained in the environment when calculating the position of the electronic pen 4.

[0075] In the interactive system 1 of this embodiment, the first processor 28 performs the following actions: displays a first image 120 containing three or more marks placed at different positions as an image 110 on the projector 2; accepts a first operation in which the electronic pen 4 specifies three or more positions that correspond one-to-one with the three or more marks; sets an input area 210 in which the first processor 28 performs a process to calculate the position of the electronic pen 4 based on the specified three or more positions; and generates a correspondence between the coordinate system of the input area 210 and the coordinate system of the image 110 displayed by the projector 2. According to the embodiment described above, the user can set an area located at any location as an input area 210, which is an area in which processing corresponding to the position of the electronic pen 4 is performed by the first processor 28.

[0076] In the interactive system 1 of this embodiment, the first processor 28 performs the following actions: displaying a second image 130, which includes a display area W1 for displaying an object, as image 110 on the projector 2; receiving a second operation to draw an object on the input area 210 using an electronic pen 4; and displaying the object in the display area W1 based on the second operation. According to this embodiment described above, the user can display a desired object in the display area W1 of the second image 130 by operating the electronic pen 4 within the input area 210 set at any location.

[0077] In the interactive system 1 of this embodiment, the input area 210 is different from the area where the image 110 is displayed by the projector 2. According to this embodiment described above, the user can display a desired object in the display area W1 by operating the electronic pen 4 within the input area 210, which is set to a location different from the area where the image 110 is displayed by the projector 2.

[0078] The interactive system 1 of this embodiment is a communication device 3 that controls communication with a network, and further comprises a communication device 3 connected to a projector 2, the projector 2 includes a first wireless module 26 as a first sensor that outputs first intensity data indicating the intensity of radio waves received from an electronic pen 4, and the communication device 3 includes a second wireless module 31 as a second sensor that outputs second intensity data indicating the intensity of radio waves received from an electronic pen 4. According to this embodiment described above, the position of the electronic pen 4 can be detected using an existing wireless module without the need to prepare a special device for detecting the position of the electronic pen 4.

[0079] The interactive system 1 of this embodiment includes an electronic pen 4 that transmits radio waves, a first wireless module 26 that receives radio waves, a second wireless module 31 that receives radio waves, and a first processor 28. The first processor 28 performs the following actions: calculates a first distance, which is the distance between the electronic pen 4 and the first wireless module 26, based on the output from the first wireless module 26; calculates a second distance, which is the distance between the electronic pen 4 and the second wireless module 31, based on the output from the second wireless module 31; calculates the position of the electronic pen 4 based on the first distance, the second distance, and the positional relationship between the first wireless module 26 and the second wireless module 31; accepts a first operation to specify three or more positions using the electronic pen 4; and sets an input area 210, which is an area in which processing corresponding to the position of the electronic pen 4 is executed by the first processor 28, based on the three or more specified positions. According to this embodiment described above, since radio waves are used to calculate the position of the electronic pen 4, an interactive system 1 can be provided that is not affected by ultrasonic noise contained in the environment when calculating the position of the electronic pen 4. In addition, the user can set an area located at any location as an input area 210, which is an area in which processing corresponding to the position of the electronic pen 4 is executed by the first processor 28.

[0080] The position detection method of this embodiment is a position detection method performed by a first processor 28, and includes: calculating a first distance, which is the distance between the electronic pen 4 and the first wireless module 26, based on the output from a first wireless module 26 that receives radio waves from the electronic pen 4; calculating a second distance, which is the distance between the electronic pen 4 and the second wireless module 31, based on the output from a second wireless module 31 that receives radio waves from the electronic pen 4; and calculating the position of the electronic pen 4 based on the first distance, the second distance, and the positional relationship between the first wireless module 26 and the second wireless module 31. According to this embodiment as described above, since radio waves are used to calculate the position of the electronic pen 4, a position detection method is provided that is not affected by ultrasonic noise contained in the environment when calculating the position of the electronic pen 4.

[0081] The input area setting method of this embodiment is an input area setting method executed by a first processor 28, and includes: calculating a first distance, which is the distance between the electronic pen 4 and the first wireless module 26, based on the output from a first wireless module 26 that receives radio waves from the electronic pen 4; calculating a second distance, which is the distance between the electronic pen 4 and the second wireless module 31, based on the output from a second wireless module 31 that receives radio waves from the electronic pen 4; calculating the position of the electronic pen 4 based on the first distance, the second distance, and the positional relationship between the first wireless module 26 and the second wireless module 31; receiving a first operation to specify three or more positions using the electronic pen 4; and setting an input area 210, which is an area in which processing corresponding to the position of the electronic pen 4 is executed by the first processor 28, based on the three or more specified positions. According to this embodiment described above, since radio waves are used to calculate the position of the electronic pen 4, it is possible to provide an input area setting method that is not affected by ultrasonic noise contained in the environment when calculating the position of the electronic pen 4. In addition, the user can set an area located at any location as the input area 210, which is the area in which the processing corresponding to the position of the electronic pen 4 is executed by the first processor 28.

[0082] While embodiments of this disclosure have been described above, the technical scope of this disclosure is not limited to the embodiments described above, and various modifications can be made without departing from the spirit of this disclosure.

[0083] In the above embodiment, an example was given in which the positional relationship between the first wireless module 26 and the second wireless module 31 is pre-stored in the first storage device 27. The disclosure is not limited thereto, and for example, the first processor 28 of the projector 2 may perform the following: receiving a third operation to set the positional relationship between the first wireless module 26 and the second wireless module 31, and calculating the position of the electronic pen 4 based on a first distance, a second distance, and the positional relationship set by the third operation. With such a modification, for example, the positional relationship between the first wireless module 26 and the second wireless module 31 can be set on the projection surface 100, so that even if the positional relationship between the first wireless module 26 and the second wireless module 31 changes, the user can manually set the optimal positional relationship. One example of a third operation for setting the positional relationship between the first wireless module 26 and the second wireless module 31 is to project a user interface from the projector 2 that allows the user to select the relative positional relationship, indicating whether the first wireless module 26 is located to the right or left of the second wireless module 31 from the user's perspective, and the user then selects one of the two options.

[0084] In the above embodiment, the first processor 28 of the projector 2 is exemplified as one or more processors, but the disclosure is not limited thereto, and the second processor 31 of the communication device 3 or the third processor 44 of the electronic pen 4 may be one or more processors.

[0085] In the above embodiment, a projector 2 was exemplified as an example of a display device, but the display device of this disclosure is not limited to a projector 2. For example, the display device of this disclosure may be a liquid crystal television or liquid crystal display equipped with a liquid crystal panel as the optical system for displaying images. Alternatively, the display device of this disclosure may be a display device equipped with an OLED (Organic Light Emitting Diode) panel as the optical system for displaying images.

[0086] In the above embodiment, an example of processing corresponding to the position of an indicator is shown as displaying an image based on the position of an indicator on a display device, but this disclosure is not limited thereto. For example, it could be turning a lighting device on or off, or opening and closing a door.

[0087] [Summary of this disclosure] A summary of this disclosure is provided below.

[0088] (Note 1) An interactive system comprising: a display device for displaying an image; an indicator for transmitting electromagnetic waves; a first sensor for receiving the electromagnetic waves; a second sensor for receiving the electromagnetic waves; and one or more processors, wherein the one or more processors perform the following actions: calculate a first distance, which is the distance between the indicator and the first sensor, based on the output from the first sensor; calculate a second distance, which is the distance between the indicator and the second sensor, based on the output from the second sensor; calculate the position of the indicator based on the first distance, the second distance, and the positional relationship between the first and second sensors; and display an image on the display device based on the position of the indicator.

[0089] According to Appendix 1, since electromagnetic waves are used to calculate the position of the indicator, an interactive system can be provided that is not affected by ultrasonic noise contained in the environment when calculating the position of the indicator.

[0090] (Note 2) The interactive system according to Note 1, wherein one or more processors perform the following actions: display a first image containing three or more marks arranged at different positions as the image on the display device; accept a first operation in which three or more positions corresponding one-to-one with the three or more marks are specified by the indicator; set an input area which is an area in which the one or more processors perform a process to calculate the position of the indicator based on the specified three or more positions; and generate a correspondence relationship which associates the coordinate system of the input area with the coordinate system of the image displayed on the display device.

[0091] According to Appendix 2, the user can set an area located at any location as an input area, which is an area where processing corresponding to the position of the indicator is performed by one or more processors.

[0092] (Note 3) The interactive system according to Note 2, wherein one or more processors perform the following: cause a second image including a display area for displaying an object to be displayed on the display device as the image; receive a second operation by the indicator to draw the object on the input area; and display the object on the display area based on the second operation.

[0093] According to Appendix 3, the user can display a desired object in the display area of ​​the second image by manipulating an indicator within an input area set at any location.

[0094] (Note 4) The interactive system according to Note 2 or 3, wherein the input area is different from the area on which an image is displayed by the display device.

[0095] According to Appendix 4, the user can display a desired object in the display area by operating an indicator within an input area set in a location different from the area where the image is displayed by the display device.

[0096] (Note 5) An interactive system according to any one of Notes 2 to 4, comprising a communication device for controlling communication with a network, further comprising a communication device connected to the display device, wherein the display device includes a first wireless module as the first sensor that outputs first intensity data indicating the intensity of the electromagnetic waves received from the indicator, and the communication device includes a second wireless module as the second sensor that outputs second intensity data indicating the intensity of the electromagnetic waves received from the indicator.

[0097] According to Appendix 5, the position of the indicator can be detected using an existing wireless module without the need to prepare a special device for detecting the position of the indicator.

[0098] (Appendix 6) The interactive system according to any one of Appendix 1 to 5, wherein the one or more processors perform the following: receiving a third operation to set the positional relationship between the first sensor and the second sensor; and calculating the position of the indicator based on the first distance, the second distance, and the positional relationship set by the third operation.

[0099] According to Appendix 6, even if the relative positions between the first and second sensors change, the user can manually set the optimal relative position.

[0100] (Note 7) An interactive system comprising: an indicator that transmits electromagnetic waves; a first sensor that receives the electromagnetic waves; a second sensor that receives the electromagnetic waves; and one or more processors, wherein the one or more processors perform the following: calculate a first distance, which is the distance between the indicator and the first sensor, based on the output from the first sensor; calculate a second distance, which is the distance between the indicator and the second sensor, based on the output from the second sensor; calculate the position of the indicator based on the first distance, the second distance, and the positional relationship between the first sensor and the second sensor; accept a first operation to specify three or more positions using the indicator; and set an input area, which is an area in which processing corresponding to the position of the indicator is performed by the one or more processors, based on the specified three or more positions.

[0101] According to Appendix 7, since electromagnetic waves are used to calculate the position of the indicator, an interactive system can be provided that is not affected by ultrasonic noise in the environment when calculating the position of the indicator. In addition, the user can set any area at any location as an input area, which is an area where processing corresponding to the position of the indicator is performed by one or more processors.

[0102] (Note 8) A position detection method performed by one or more processors, comprising: calculating a first distance, which is the distance between an indicator and the first sensor, based on the output from a first sensor that receives electromagnetic waves from an indicator; calculating a second distance, which is the distance between the indicator and the second sensor, based on the output from a second sensor that receives electromagnetic waves from the indicator; and calculating the position of the indicator based on the first distance, the second distance, and the positional relationship between the first sensor and the second sensor.

[0103] According to Appendix 8, since electromagnetic waves are used to calculate the position of the indicator, a position detection method can be provided that is not affected by ultrasonic noise contained in the environment when calculating the position of the indicator.

[0104] (Note 9) An input area setting method performed by one or more processors, comprising: calculating a first distance which is the distance between an indicator and the first sensor based on the output from a first sensor which receives electromagnetic waves from an indicator; calculating a second distance which is the distance between the indicator and the second sensor based on the output from a second sensor which receives electromagnetic waves from the indicator; calculating the position of the indicator based on the first distance, the second distance, and the positional relationship between the first sensor and the second sensor; receiving a first operation which specifies three or more positions using the indicator; and setting an input area which is an area in which processing corresponding to the position of the indicator is performed by the one or more processors based on the specified three or more positions.

[0105] According to Appendix 9, since electromagnetic waves are used to calculate the position of the indicator, a method for setting the input area that is not affected by ultrasonic noise contained in the environment when calculating the position of the indicator can be provided. In addition, the user can set an area located at any location as an input area in which the processing corresponding to the position of the indicator is executed by one or more processors. [Explanation of Symbols]

[0106] 1…Interactive system, 2…Projector, 3…Communication device, 4…Electronic pen, 10…Optical device, 25…Input device, 26…First wireless module, 27…First memory device, 28…First processor, 31…Second wireless module, 31…Second memory device, 33…Second processor, 41…Third wireless module, 42…Button, 43…Third memory device, 44…Third processor, 100…Projection surface, 110…Image, 200…Operation surface, 210…Input area, LC…Image light

Claims

1. A display device for displaying images, An indicator that transmits electromagnetic waves, The first sensor that receives the electromagnetic waves, The second sensor that receives the aforementioned electromagnetic waves, One or more processors, Equipped with, The one or more processors described above are: Based on the output from the first sensor, a first distance is calculated, which is the distance between the indicator and the first sensor. Based on the output from the second sensor, a second distance is calculated, which is the distance between the indicator and the second sensor. The position of the indicator is calculated based on the first distance, the second distance, and the positional relationship between the first sensor and the second sensor. To display an image on the display device based on the position of the indicator, An interactive system that executes [this].

2. The one or more processors described above are: A first image containing three or more marks arranged in different positions is displayed on the display device as the image, The system accepts a first operation in which the indicator designates three or more positions that correspond one-to-one with the three or more marks, Based on the three or more specified positions, The process for calculating the position of the indicator is performed in the region where one or more processors are used. Setting the input area, To generate a correspondence between the coordinate system of the input area and the coordinate system of the image displayed by the display device, The interactive system according to claim 1, which performs the following:

3. The one or more processors described above are: To display a second image, which includes a display area for displaying an object, as the image on the display device, The indicator accepts a second operation to draw the object in the input area, Based on the second operation, the object is displayed in the display area, The interactive system according to claim 2, which performs the following:

4. The interactive system according to claim 2 or 3, wherein the input area is different from the area on which an image is displayed by the display device.

5. A communication device for controlling communication with a network, further comprising a communication device connected to the display device, The display device includes a first wireless module as the first sensor, which outputs first intensity data indicating the intensity of the electromagnetic wave received from the indicator. The interactive system according to claim 2 or 3, wherein the communication device includes a second wireless module as the second sensor, which outputs second intensity data indicating the intensity of the electromagnetic wave received from the indicator.

6. The one or more processors described above are: The system accepts a third operation to set the positional relationship between the first sensor and the second sensor, The position of the indicator is calculated based on the first distance, the second distance, and the positional relationship set by the third operation. The interactive system according to claim 1, which performs the following:

7. An indicator that transmits electromagnetic waves, The first sensor that receives the electromagnetic waves, The second sensor that receives the aforementioned electromagnetic waves, One or more processors, Equipped with, The one or more processors described above are: Based on the output from the first sensor, a first distance is calculated, which is the distance between the indicator and the first sensor. Based on the output from the second sensor, a second distance is calculated, which is the distance between the indicator and the second sensor. The position of the indicator is calculated based on the first distance, the second distance, and the positional relationship between the first sensor and the second sensor. The system accepts a first operation in which three or more positions are specified by the indicator, Setting an input region which is an area in which processing corresponding to the position of the indicator is performed by the one or more processors based on the three or more specified positions, An interactive system that executes [this].

8. A position detection method performed by one or more processors, Based on the output from a first sensor that receives electromagnetic waves from an indicator, a first distance is calculated, which is the distance between the indicator and the first sensor. Based on the output from the second sensor that receives the electromagnetic waves from the indicator, a second distance is calculated, which is the distance between the indicator and the second sensor. The position of the indicator is calculated based on the first distance, the second distance, and the positional relationship between the first sensor and the second sensor. A method for detecting location, including the method described above.

9. An input area setting method performed by one or more processors, Based on the output from a first sensor that receives electromagnetic waves from an indicator, a first distance is calculated, which is the distance between the indicator and the first sensor. Based on the output from the second sensor that receives the electromagnetic waves from the indicator, a second distance is calculated, which is the distance between the indicator and the second sensor. The position of the indicator is calculated based on the first distance, the second distance, and the positional relationship between the first sensor and the second sensor. The system accepts a first operation in which three or more positions are specified by the indicator, Setting an input region which is an area in which processing corresponding to the position of the indicator is performed by the one or more processors based on the three or more specified positions, A method for setting the input area, including [specific details].

Images