Display method and display device

Abstract

To improve operability of instruction operation to an operation image.SOLUTION: A display method includes detecting an indicator, and displaying an operation image including a plurality of item images. When the indicator includes a right hand, in the operation image, the plurality of item images are arranged in a first mode. When the indicator includes a left hand, in the operation image, the plurality of item images are arranged in a second mode.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The present invention relates to a display method and a display device.

[0002] Display devices equipped with so-called interactive functions have been developed, which allow users to perform operations and write characters and pictures on a display screen using a pointer such as an electronic pen or a finger. For example, Patent Document 1 discloses an interactive projector that receives instructions and operations from a user by detecting the tip position of a pointer. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-186678 Summary of the Invention [Problem to be solved by the invention]

[0004] The GUI (Graphical User Interface) displayed on the display screen was the same whether the pointer included the right hand or the left hand, which left room for improvement in GUI operability. [Means for solving the problem]

[0005] One aspect of the display method of the present invention includes detecting an indicator and displaying an operation image including a plurality of item images, and is characterized in that if the detected indicator includes a right hand, the plurality of item images in the operation image are arranged in a first manner, and if the detected indicator includes a left hand, the plurality of item images in the operation image are arranged in a second manner.

[0006] One aspect of the display device of the present invention comprises a detection unit that detects an indicator and a display control unit that displays an operation image including a plurality of item images, and is characterized in that when the detected indicator includes a right hand, the plurality of item images in the operation image are arranged in a first manner, and when the detected indicator includes a left hand, the plurality of item images in the operation image are arranged in a second manner. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram for explaining an overview of a projector 1 according to a first embodiment. [Figure 2] 10 is a schematic diagram for explaining a plurality of regions that an operation image GU has. FIG. [Figure 3] FIG. 10 is an explanatory diagram illustrating a state in which a user uses his / her right hand H1 to display an operation image GU1. [Figure 4] FIG. 10 is a schematic diagram for explaining an operation image GU1. [Figure 5] FIG. 10 is an explanatory diagram illustrating a state in which a user uses his / her left hand H2 to display an operation image GU2. [Figure 6] FIG. 10 is a schematic diagram for explaining an operation image GU2. [Figure 7] FIG. 10 is an explanatory diagram illustrating a state in which a user uses a pointer H9 to display an operation image GU3. [Figure 8] FIG. 10 is a schematic diagram for explaining an operation image GU3. [Figure 9] 10 is an explanatory diagram illustrating a state in which a user performs a pointing operation on an operation image GU1 using a right hand H1. FIG. [Figure 10] 1 is a block diagram showing the configuration of a projector 1 according to a first embodiment. [Figure 11] 2 is a block diagram showing the configuration of a storage unit 11 according to the first embodiment. FIG. [Figure 12] 10 is a flowchart for explaining the operation of the projector 1 according to the first embodiment. [Figure 13]FIG. 10 is a schematic diagram for explaining an operation image GU1A. [Figure 14] FIG. 10 is a schematic diagram for explaining an operation image GU2A. [Figure 15] FIG. 10 is a block diagram showing the configuration of a projector 1A according to a second embodiment. [Figure 16] FIG. 10 is a block diagram showing the configuration of a storage unit 11A according to a second embodiment. [Figure 17] 10 is a flowchart for explaining the operation of a projector 1A according to the second embodiment. [Figure 18] FIG. 10 is a schematic diagram for explaining an operation image GU2m. [Figure 19] FIG. 10 is a schematic diagram for explaining an operation image GR1. [Figure 20] FIG. 10 is a schematic diagram for explaining an operation image GR2. DETAILED DESCRIPTION OF THE INVENTION

[0008] Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. Note that the dimensions and scale of each part in the drawings may differ from the actual dimensions, and some parts are shown schematically to facilitate understanding. Furthermore, the scope of the present invention is not limited to these embodiments unless otherwise specified in the following description to the effect that the present invention is limited thereto. 。

[0009] 1. First embodiment In the first embodiment, a display method and a display device according to the present invention will be described using as an example a projector that changes the GUI to be displayed depending on whether the user's hand is the right hand or the left hand when performing an instruction operation on the GUI.

[0010] 1.1.Projector Overview FIG. 1 is a schematic diagram for explaining an overview of a projector 1 according to a first embodiment. In this embodiment, it is assumed that the projector 1 projects projection light onto a wall surface W1 to display a projection image GP. The projection image GP is a general term for an image displayed by the projector 1 projecting projection light onto a display surface. In this embodiment, the projection image GP may include an operation image GU that allows a user to perform an input operation on the projector 1. The operation image GU is a GUI that includes multiple item images GC. In this embodiment, the operation image GU is an image with a rectangular outline. The item image GC is an image related to a process executed by the projector 1. For example, a user of the projector 1 performs a pointing operation on one of the multiple item images GC with the fingertip H10 of the right hand H1, which serves as a pointing object, to cause the projector 1 to execute a process corresponding to the one item image GC.

[0011] In this embodiment, it is assumed that a user of the projector 1 faces the wall surface W1, more specifically, the projection image GP displayed on the wall surface W1. In this embodiment, it is assumed that the wall surface W1 is arranged parallel to the X-axis and Y-axis. Here, it is assumed that the Y-axis is an axis parallel to the vertical direction. Of the directions parallel to the Y-axis, the vertical direction is defined as the -Y direction, and the direction opposite to the -Y direction is defined as the +Y direction. In addition, it is assumed that the X-axis is an axis perpendicular to the Y-axis and parallel to the wall surface W1. Of the directions parallel to the X-axis, the direction toward the right as seen by the user is defined as the +X direction, and the direction opposite to the +X direction is defined as the -X direction. In addition, the axis perpendicular to the X-axis and Y-axis is referred to as the Z-axis. The Z-axis is an axis perpendicular to the wall surface W1. Among the directions parallel to the Z axis, the direction from the wall W1 toward the user is defined as the +Z direction, and the direction opposite to the +Z direction is defined as the -Z direction. That is, by facing the -Z direction, the user faces the projected image GP displayed on the wall W1. It is assumed that the projected image GP is displayed as a rectangle with two sides parallel to the X axis and two sides parallel to the Y axis. In this embodiment, the +Y direction may be referred to as "upward from the user's perspective," the -Y direction as "downward from the user's perspective," the +X direction as "rightward from the user's perspective," and the -X direction as "leftward from the user's perspective." When describing the positional relationship between two different locations, if one location is located in the +Y direction relative to the other, it may be said that "one location is located above the other location relative to the user." Similarly, if one location is located in the -X direction relative to the other location, it may be said that "one location is located to the left of the other location relative to the user."

[0012] 1.2. Features and display methods of operation images 2 is a schematic diagram illustrating multiple regions of an operation image GU. In this embodiment, the operation image GU has regions R11, R12, R13, R21, R22, R23, R31, R32, and R33. For convenience, it is assumed that regions R11, R12, R13, R21, R22, R23, R31, R32, and R33 all have similar dimensions and areas.

[0013] Regions R21, R22, and R23 are set in the +Y direction relative to regions R11, R12, and R13. In other words, regions R21, R22, and R23 are positioned above regions R11, R12, and R13 as viewed from the user. Furthermore, regions R31, R32, and R33 are set in the +Y direction relative to regions R21, R22, and R23. In other words, regions R31, R32, and R33 are positioned higher than regions R21, R22, and R23 as seen from the user. Furthermore, regions R31, R32, and R33 are positioned higher than regions R11, R12, and R13 as seen from the user.

[0014] Regions R12, R22, and R32 are set to be further in the +X direction than regions R11, R21, and R31. In other words, regions R11, R21, and R31 are located to the left of regions R12, R22, and R32 as viewed from the user. Furthermore, regions R13, R23, and R33 are set in the +X direction relative to regions R12, R22, and R32. In other words, regions R12, R22, and R32 are located to the left of regions R13, R23, and R33 as viewed from the user. Furthermore, regions R11, R21, and R31 are located to the left of regions R13, R23, and R33 as viewed from the user.

[0015] Hereinafter, with reference to FIGS. 3 and 4, the operation image GU1 that is displayed when the user performs an input operation on the projector 1 using the right hand H1 will be described.

[0016] 3 is an explanatory diagram illustrating a state in which a user uses his / her right hand H1 to display an operation image GU1. In FIG. 3, the projector 1 displays a projection image GP0 on a wall surface W1. The projection image GP0 is an image that does not include the operation image GU. The user causes the projector 1 to detect the right hand H1 by bringing the right hand H1, which is an indicator, close to the area in which the projection image GP0 is displayed.

[0017] 4 is a schematic diagram for explaining the operation image GU1. When the projector 1 detects a right hand H1, it displays a projection image GP1 on the wall surface W1. The projection image GP1 is an image that includes the operation image GU1. The operation image GU1 is a GUI that includes item images GC1 to GC9.

[0018] The item image GC1 is, for example, an item image related to a drawing tool calling process. In the operation image GU1, the item image GC1 is arranged in an area R21. The item image GC2 is, for example, an item image relating to an eraser tool calling process. In the operation image GU1, the item image GC2 is arranged in the area R31. The item image GC3 is, for example, an item image related to a movement tool calling process. In the operation image GU1, the item image GC3 is arranged in the area R32. The item image GC4 is, for example, an item image related to redo processing. In the operation image GU1, the item image GC4 is arranged in the area R33. The item image GC5 is, for example, an item image relating to a screen magnification tool calling process. In the operation image GU1, the item image GC5 is arranged in the area R22. The item image GC6 is, for example, an item image related to a save process. In the operation image GU1, the item image GC6 is arranged in the region R11. The item image GC7 is, for example, an item image related to a screen capture process. In the operation image GU1, the item image GC7 is arranged in the region R23. The item image GC8 is, for example, an item image related to a setting screen call process. In the operation image GU1, the item image GC8 is arranged in the region R12. The item image GC9 is, for example, an item image related to a cutout tool calling process. In the operation image GU1, the item image GC9 is arranged in the region R13.

[0019] That is, in the operation image GU1, the item image GC2 arranged in the region R31 is arranged to the left of the item image GC4 arranged in the region R33 as viewed from the user. Furthermore, the item image GC1 arranged in the region R21 is arranged to the left of the item image GC7 arranged in the region R23 as viewed from the user.

[0020] In this embodiment, it is assumed that the eraser tool calling process is executed more frequently than the redo process. It is also assumed that the drawing tool calling process is executed more frequently than the eraser tool calling process. It is also assumed that the drawing tool calling process is executed more frequently than the screen capture process.

[0021] Hereinafter, with reference to FIGS. 5 and 6, the operation image GU2 that is displayed when the user performs an input operation on the projector 1 using the left hand H2 will be described.

[0022] Fig. 5 is an explanatory diagram illustrating a state in which a user uses his / her left hand H2 to display an operation image GU2. In Fig. 5, the projector 1 displays a projection image GP0 on a wall surface W1. As described above, the projection image GP0 is an image that does not include the operation image GU. The user causes the projector 1 to detect the left hand H2 by bringing the left hand H2, which is an indicator, close to the area in which the projection image GP0 is displayed.

[0023] 6 is a schematic diagram for explaining the operation image GU2. When the projector 1 detects the left hand H2, it displays the projection image GP2 on the wall surface W1. The projection image GP2 is an image that includes the operation image GU2. Like the operation image GU1, the operation image GU2 is a GUI that includes item images GC1 to GC9.

[0024] In the operation image GU2, the item image GC1 is arranged in the area R23. In addition, in the operation image GU2, the item image GC2 is arranged in the area R33. In addition, in the operation image GU2, the item image GC3 is arranged in the area R32. In addition, in the operation image GU2, the item image GC4 is arranged in the area R31. In addition, in the operation image GU2, the item image GC5 is arranged in the area R22. In addition, in the operation image GU2, the item image GC6 is arranged in the area R13. In addition, in the operation image GU2, the item image GC7 is arranged in the area R21. In addition, in the operation image GU2, the item image GC8 is arranged in the area R12. In addition, in the operation image GU2, the item image GC9 is arranged in the area R11.

[0025] That is, in the operation image GU2, the item image GC2 arranged in the region R33 is arranged to the right of the item image GC4 arranged in the region R31 as viewed from the user. Furthermore, the item image GC1 arranged in the region R23 is arranged to the right of the item image GC7 arranged in the region R21 as viewed from the user.

[0026] Hereinafter, with reference to FIGS. 7 and 8, the operation image GU3 that is displayed when the user performs an input operation on the projector 1 using the pointer H9 will be described.

[0027] 7 is an explanatory diagram illustrating a state in which a user uses a pointer H9 to display an operation image GU3. In FIG. 7, the projector 1 displays a projection image GP0 on a wall surface W1. As described above, the projection image GP0 is an image that does not include the operation image GU. The user brings the pointer H9 close to the area in which the projection image GP0 is displayed, causing the projector 1 to detect the pointer H9.

[0028] 8 is a schematic diagram for explaining the operation image GU3. When the projector 1 detects the pointer H9, it displays the projection image GP3 on the wall surface W1. The projection image GP3 is an image that includes the operation image GU3. The operation image GU3 is a GUI that includes item images GC1 to GC9, similar to the operation images GU1 and GU2.

[0029] The indicator H9 is, for example, a wand used when a user performs an instruction operation on the operation image GU from a position where the user's hand cannot reach the wall surface W1. When the user brings the indicator H9 close to the area where the projection image GP0 is displayed, the projector 1 detects the indicator H9 as an indicator that does not include either the right hand H1 or the left hand H2. In other words, even if the user uses the right hand H1 or the left hand H2 as the indicator, if the projector 1 detects the indicator as one that does not include either the right hand H1 or the left hand H2, the operation image GU3 is displayed on the wall surface W1.

[0030] In the operation image GU3, the item image GC1 is arranged in the area R31. In addition, in the operation image GU3, the item image GC2 is arranged in the area R21. In addition, in the operation image GU3, the item image GC3 is arranged in the area R32. In addition, in the operation image GU3, the item image GC4 is arranged in the area R23. In addition, in the operation image GU3, the item image GC5 is arranged in the area R22. In addition, in the operation image GU3, the item image GC6 is arranged in the area R33. In addition, in the operation image GU3, the item image GC7 is arranged in the area R11. In addition, in the operation image GU3, the item image GC8 is arranged in the area R12. In addition, in the operation image GU3, the item image GC9 is arranged in the area R13.

[0031] That is, in the operation image GU3, the item image GC1 arranged in the region R31 is arranged above the item image GC2 arranged in the region R21 as viewed from the user. Furthermore, the item image GC1 arranged in the region R31 is arranged above the item image GC7 arranged in the region R11 as viewed from the user.

[0032] FIG. 9 is an explanatory diagram illustrating a situation in which a user uses his or her right hand H1 to perform a pointing operation on an operation image GU1. The user selects one of the multiple item images GC included in the operation image GU1 and performs a pointing operation on the selected item image GC using the fingertip H10 of the right hand H1, thereby causing the projector 1 to execute a process corresponding to the selected item image GC. At this time, the projector 1 detects the pointing position indicated by the fingertip H10 of the right hand H1 as the pointing position of the pointer. Since the detected pointing position is within the area in which the selected item image GC is displayed, the projector 1 executes a process corresponding to the selected item image GC. For example, when the user points to an item image GC3 using the fingertip H10 of the right hand H1, the projector 1 detects that the XY coordinates of the pointing position indicated by the fingertip H10 of the right hand H1 are within area R32. The projector 1 then executes a movement tool call process corresponding to the item image GC3 displayed in area R32.

[0033] When a user performs a pointing operation on an operation image GU, some of the multiple item images GC included in the operation image GU may be hidden behind the pointer as viewed from the user. For example, in the operation image GU1 shown in Fig. 9, the item image GC7 arranged in region R23 is shown hidden behind the right hand H1. For this reason, the user cannot easily visually recognize the item image GC7 hidden behind the right hand H1, and it may take some time to determine the location of the item image GC7.

[0034] On the other hand, as can be seen from FIG. 9, the item image GC1 arranged in region R21 is not hidden behind the right hand H1. Generally, the projection image GP is displayed at about the same height as the user's face or above the face. When the projection image GP1 is displayed at about the same height as the user's face or above the face, when the user uses his or her right hand H1 to perform a command operation on the operation image GU1, the right hand H1 extends from the lower right to the upper left as viewed from the user. Because region R21 is located to the left of region R23, the item image GC1 arranged in region R21 is less likely to be hidden behind the right hand H1 than the item image GC7 arranged in region R23. In other words, in the operation image GU1, the item image GC1 related to the drawing tool call process, which is executed more frequently than the screen capture process, is displayed to the left of the item image GC7 related to the screen capture process, so the user can easily determine the location of the item image GC1 that is frequently designated. Similarly, the item image GC2 relating to the eraser tool call process, which is executed more frequently than the redo process, is displayed to the left of the item image GC4 relating to the redo process, so the user can easily grasp the position of the item image GC2 that is frequently indicated.

[0035] Furthermore, if the display position of the projection image GP2 is at the same height as or above the user's face, when the user uses his or her left hand H2 to perform a command on the operation image GU2, the left hand H2 extends from the lower left to the upper right as viewed from the user. Because region R23 is located to the right of region R21, the item image GC1 located in region R23 is less likely to be hidden by the left hand H2 than the item image GC7 located in region R21. That is, in the operation image GU2, the item image GC1 related to the drawing tool call process, which is executed more frequently than the screen capture process, is displayed to the right of the item image GC7 related to the screen capture process, allowing the user to easily grasp the location of the item image GC1 that is frequently specified. Similarly, the item image GC2 related to the eraser tool call process, which is executed more frequently than the redo process, is displayed to the right of the item image GC4 related to the redo process, allowing the user to easily grasp the location of the item image GC2 that is frequently specified.

[0036] Furthermore, if the display position of the projected image GP3 is at the same height as or above the user's face, when the user uses the indicator H9 to perform a pointing operation on the operation image GU3, the indicator H9 is generally considered to extend from below to above as viewed from the user. Because region R31 is located above region R11, the item image GC1 located in region R31 is less likely to be hidden by the indicator H9 than the item image GC7 located in region R11. That is, in the operation image GU3, the item image GC1 related to the drawing tool call process, which is executed more frequently than the screen capture process, is displayed above the item image GC7 related to the screen capture process, allowing the user to easily grasp the location of the item image GC1 that is frequently pointed to. Similarly, the item image GC1 related to the drawing tool call process, which is executed more frequently than the eraser tool call process, is displayed above the item image GC2 related to the eraser tool call process, allowing the user to easily grasp the location of the item image GC1 that is frequently pointed to.

[0037] 1.3.Projector configuration and functions The configuration and functions of the projector 1 according to the first embodiment will be described below with reference to FIGS.

[0038] 10 is a block diagram showing the configuration of a projector 1 according to a first embodiment. The projector 1 includes a storage unit 11 that stores various information, a control unit 12 that controls the operation of the projector 1, an imaging unit 13 that captures images to detect a pointer, an operation unit 14 that accepts input operations from a user of the projector 1, and a projection unit 15 that displays an image on a display surface by projecting projection light. The control unit 12 functions as a detection unit 120, a projection control unit 124, and a processing unit 125. The detection unit 120 functions as a pointer detection unit 121, an indication position detection unit 122, and an imaging control unit 123.

[0039] The storage unit 11 includes, for example, a volatile memory such as a RAM and a non-volatile memory such as a ROM. Here, RAM is an abbreviation for Random Access Memory, and ROM is an abbreviation for Read Only Memory.

[0040] FIG. 11 is a block diagram showing the configuration of the storage unit 11 according to the first embodiment. The nonvolatile memory of the storage unit 11 stores a program 110 that defines the operation of the projector 1, operation image information 112 that represents an operation image GU, and imaging information 116 that is used to detect a pointer. In this embodiment, the operation image information 112 includes first operation image information 113, second operation image information 114, and third operation image information 115. The first operation image information 113 is information that represents an operation image GU1. The second operation image information 114 is information that represents an operation image GU2. The third operation image information 115 is information that represents an operation image GU3. The imaging information 116 includes first imaging information 117 and second imaging information 118.

[0041] The operation image information 112 can be arbitrarily changed, for example, by receiving an editing operation from the user through the operation unit 14. That is, by editing the operation image information 112, the user can arbitrarily change the arrangement, color, size, etc. of the item images GC included in the operation image GU.

[0042] The volatile memory of the storage unit 11 is used by the control unit 12 as a work area when the program 110 is executed.

[0043] Note that part or all of the storage unit 11 may be provided in an external storage device, an external server, etc. Also, part or all of the various information stored in the storage unit 11 may be stored in advance in the storage unit 11, or may be acquired from an external storage device, an external server, etc.

[0044] The control unit 12 is configured to include one or more CPUs. However, the control unit 12 may include a programmable logic device such as an FPGA instead of or in addition to a CPU. Here, CPU is an abbreviation for Central Processing Unit, and FPGA is an abbreviation for Field-Programmable Gate Array.

[0045] 10 by a CPU or the like included in the control unit 12 executing a program 110 and operating in accordance with the program 110. The detection unit 120 specifically functions as a pointer detection unit 121, a pointed position detection unit 122, and an imaging control unit 123.

[0046] The detection unit 120 detects the pointer. The detection unit 120 also detects the type of pointer. In addition, the detection unit 120 detects the pointing position of the pointer. Note that "detecting the type of pointer" means, for example, determining whether the pointer includes the right hand H1, the left hand H2, or neither the right hand H1 nor the left hand H2. Also, "detecting the pointing position of the pointer" means, for example, determining the XY coordinates of the point where the pointer is in contact with the wall surface W1.

[0047] The imaging control unit 123 controls the imaging unit 13 to capture an image of an area including the area on the wall surface W1 where the projection image GP is displayed and the space near the area on the wall surface W1. The imaging control unit 123 then acquires, from the imaging unit 13, imaging information 116 representing an imaging result of capturing an area including the area on the wall surface W1 where the projection image GP is displayed and the space near the area on the wall surface W1. The imaging control unit 123 also stores the acquired imaging information 116 in the storage unit 11. In this embodiment, the imaging control unit 123 acquires first imaging information 117 and second imaging information 118 as the imaging information 116. The imaging control unit 123 then stores the acquired first imaging information 117 and second imaging information 118 in the storage unit 11. The first imaging information 117 and the second imaging information 118 are acquired at different times. The space near the area on the wall surface W1 where the projection image GP is displayed is, for example, a space including a range that is a predetermined distance away in the +Z direction from the area on the wall surface W1.

[0048] The pointer detection unit 121 determines, based on the imaging information 116, whether or not a pointer is included in the captured image represented by the imaging information 116. Furthermore, when a pointer is included in the captured image represented by the imaging information 116, that is, when a pointer is detected, the pointer detection unit 121 determines, based on the imaging information 116, whether the pointer includes the right hand H1, the left hand H2, or neither the right hand H1 nor the left hand H2. In this embodiment, the pointer detection unit 121 determines, based on the first imaging information 117, whether or not a pointer is included in the captured image represented by the first imaging information 117. Furthermore, when a pointer is included in the captured image represented by the first imaging information 117, the pointer detection unit 121 determines, based on the first imaging information 117, whether the pointer includes the right hand H1, the left hand H2, or neither the right hand H1 nor the left hand H2.

[0049] Additionally, when a pointer is included in the captured image indicated by the imaging information 116, the pointer detection unit 121 determines whether or not the pointer is in contact with the wall surface W1, based on the imaging information 116. In this embodiment, when a pointer is included in the captured image indicated by the second imaging information 118, the pointer detection unit 121 determines whether or not the pointer is in contact with the wall surface W1, based on the second imaging information 118.

[0050] When a pointer included in the captured image represented by the imaging information 116 is in contact with the wall surface W1, the pointing position detection unit 122 identifies the location where the pointer is in contact with the wall surface W1 based on the imaging information 116. In this embodiment, when a pointer included in the captured image represented by the second imaging information 118 is in contact with the wall surface W1, the pointing position detection unit 122 identifies the XY coordinates of the location where the pointer is in contact with the wall surface W1 based on the second imaging information 118. Note that the "location where the pointer is in contact with the wall surface W1" is an example of the "pointing position of the pointer," and specifically, it may be "the location where the fingertip H10 of the right hand H1 is in contact with the wall surface W1," etc.

[0051] The function of the detection unit 120, i.e., the function related to the detection of a pointer, may use known image processing techniques such as template matching using a plurality of template images, machine learning, etc. In this specification, a detailed technical description related to the detection of a pointer will be omitted.

[0052] The projection control unit 124 controls the projection unit 15 to project projection light onto the display surface to display a projection image GP. If the indicator detection unit 121 determines that the captured image indicated by the imaging information 116 includes a right hand H1, the projection control unit 124 projects projection light onto the display surface to display a projection image GP1. If the indicator detection unit 121 determines that the captured image indicated by the imaging information 116 includes a left hand H2, the projection control unit 124 projects projection light onto the display surface to display a projection image GP2. If the indicator detection unit 121 determines that the captured image indicated by the imaging information 116 does not include either the right hand H1 or the left hand H2, the projection control unit 124 projects projection light onto the display surface to display a projection image GP3. In the present embodiment, when the indicator included in the captured image represented by the first imaging information 117 includes a right hand H1, the projection control unit 124 projects projection light onto the wall surface W1 to display a projection image GP1. That is, the projection control unit 124 controls the projection unit 15 to display the operation image GU1 represented by the first operation image information 113 on the wall surface W1. Also, when the indicator included in the captured image represented by the first imaging information 117 includes a left hand H2, the projection control unit 124 projects projection light onto the wall surface W1 to display a projection image GP2. That is, the projection control unit 124 controls the projection unit 15 to display the operation image GU2 represented by the second operation image information 114 on the wall surface W1. Also, when the indicator included in the captured image represented by the first imaging information 117 does not include either the right hand H1 or the left hand H2, the projection control unit 124 projects projection light onto the wall surface W1 to display a projection image GP3. That is, the projection control unit 124 controls the projection unit 15 to display the operation image GU3 indicated by the third operation image information 115 on the wall surface W1.

[0053] The processing unit 125 executes various processes based on the XY coordinates of the point where the pointer is in contact with the wall surface W1 and the item image GC displayed at that point. In other words, when the pointing position of the pointer is within an area where one of the multiple item images GC included in the operation image GU is displayed, the processing unit 125 executes a process corresponding to that one item image GC. Note that when no item image GC is displayed at the point where the pointer is in contact with the wall surface W1, the processing unit 125 may execute a process to end the display of the operation image GU, for example.

[0054] The imaging unit 13 is, for example, a camera equipped with an imaging element that converts condensed light into an electrical signal and an imaging lens. The imaging element is, for example, an image sensor such as a CCD or CMOS. Here, CCD is an abbreviation for Charge Coupled Device, and CMOS is an abbreviation for Complementary Metal Oxide Semiconductor. The imaging lens is provided to capture an image of a range including the area on the wall surface W1 where the projection image GP is displayed and the space near the area on the wall surface W1. The imaging unit 13 acquires the captured image under the control of the imaging control unit 123. The captured image is output to the control unit 12 as imaging information 116 representing the captured image.

[0055] In this embodiment, the projector 1 preferably includes a plurality of cameras as the imaging unit 13. The plurality of cameras included in the projector 1 are preferably provided at different positions. The imaging information 116 is preferably information representing the imaging results of imaging a range including the area on the wall surface W1 where the projection image GP is displayed and the space in the vicinity of the area on the wall surface W1, from a plurality of different positions. This makes it possible to improve the accuracy of calculation of the three-dimensional position by triangulation, and to more accurately detect the position pointed to by the pointer.

[0056] The operation unit 14 accepts input operations on the projector 1 from a user of the projector 1. The operation unit 14 is, for example, a touch panel or operation buttons provided on the housing of the projector 1. If the operation unit 14 is configured to include a touch panel, the operation unit 14 outputs data indicating the detected touch position to the control unit 12. If the operation unit 14 is configured to include operation buttons, the operation unit 14 outputs data identifying the pressed button to the control unit 12. In this way, the content of the input operation on the projector 1 is transmitted to the control unit 12.

[0057] The projection unit 15 projects projection light for displaying the projection image GP on the display surface under the control of the projection control unit 124. In this embodiment, the projection unit 15 projects projection light for displaying the projection image GP on the wall surface W1. The projection unit 15 includes, for example, a light source, a light modulation device, and a projection lens (not shown). The light modulation device may be, for example, a liquid crystal panel or a digital mirror device. The light source may be, for example, a halogen lamp, a xenon lamp, an ultra-high pressure mercury lamp, an LED, or a laser light source. Here, LED is an abbreviation for Light Emitting Diode.

[0058] 1.4.Projector Operation 12 is a flowchart for explaining the operation of the projector 1 according to the first embodiment. The series of operations shown in the flowchart is started, for example, when the projector 1 is turned on and the operation unit 14 receives an input operation related to the start of operation from the user of the projector 1. Note that in this embodiment, it is assumed that the projector 1 projects projection light for displaying the projection image GP0 onto the wall surface W1 in conjunction with the power being turned on.

[0059] In step S101, the imaging control unit 123 controls the imaging unit 13 to capture an image of a range including the area on the wall surface W1 where the projection image GP0 is displayed and the space near the area on the wall surface W1. Then, the imaging control unit 123 acquires, from the imaging unit 13, first imaging information 117 representing the imaging result of capturing an image of the area including the area on the wall surface W1 where the projection image GP0 is displayed and the space near the area on the wall surface W1. Furthermore, the imaging control unit 123 stores the acquired first imaging information 117 in the storage unit 11.

[0060] In step S102, the pointer detection unit 121 determines, based on the first imaging information 117, whether or not a pointer is included in the captured image indicated by the first imaging information 117. If the result of the determination in step S102 is positive, that is, if YES in step S102, the pointer detection unit 121 proceeds to the process of step S103. If the result of the determination in step S102 is negative, that is, if NO in step S102, the pointer detection unit 121 proceeds to the process of step S101.

[0061] 12 starts, the user moves the pointer within the imaging range of the imaging unit 13, causing the projector 1 to capture an image of the pointer. The projector 1 captures an image of the pointer and detects the pointer based on the captured image, allowing the process to proceed to step S103. Until the user causes the projector 1 to detect the pointer, the projector 1 repeatedly executes the processes in steps S101 and S102.

[0062] In step S103, the indicator detection unit 121 determines, based on the first imaging information 117, whether or not the indicator included in the captured image indicated by the first imaging information 117 includes the right hand H1. If the result of the determination in step S103 is positive, that is, if YES in step S103, the indicator detection unit 121 proceeds to step S105. If the result of the determination in step S103 is negative, that is, if NO in step S103, the indicator detection unit 121 proceeds to step S104.

[0063] In step S104, the indicator detection unit 121 determines, based on the first imaging information 117, whether or not the indicator included in the captured image indicated by the first imaging information 117 includes the left hand H2. If the result of the determination in step S104 is positive, that is, if YES in step S104, the indicator detection unit 121 proceeds to the process of step S106. If the result of the determination in step S104 is negative, that is, if NO in step S104, the indicator detection unit 121 proceeds to the process of step S107.

[0064] If the result of the determination in step S103 is positive, the right hand H1 is deemed to be included in the captured image represented by the first imaging information 117. If the result of the determination in step S104 is positive, the left hand H2 is deemed to be included in the captured image represented by the first imaging information 117. If the results of the determinations in both steps S103 and S104 are negative, the captured image represented by the first imaging information 117 is deemed to include neither the right hand H1 nor the left hand H2.

[0065] In step S105, the projection control unit 124 controls the projection unit 15 to project projection light for displaying the projection image GP1 onto the wall surface W1. That is, the projection control unit 124 controls the projection unit 15 to display the operation image GU1 indicated by the first operation image information 113 on the wall surface W1.

[0066] In step S106, the projection control unit 124 controls the projection unit 15 to project projection light for displaying the projection image GP2 onto the wall surface W1. That is, the projection control unit 124 controls the projection unit 15 to display the operation image GU2 indicated by the second operation image information 114 on the wall surface W1.

[0067] In step S107, the projection control unit 124 controls the projection unit 15 to project projection light for displaying the projection image GP3 onto the wall surface W1. That is, the projection control unit 124 controls the projection unit 15 to display the operation image GU3 indicated by the third operation image information 115 on the wall surface W1.

[0068] If the indicator detection unit 121 determines that the captured image represented by the first imaging information 117 includes a right hand H1, the projection control unit 124 displays an operation image GU1. In other words, if the detected indicator includes a right hand H1, the item images GC1 to GC9 are arranged in the form of the operation image GU1. If the indicator detection unit 121 determines that the captured image represented by the first imaging information 117 includes a left hand H2, the projection control unit 124 displays an operation image GU2. In other words, if the detected indicator includes a left hand H2, the item images GC1 to GC9 are arranged in the form of the operation image GU2. If the indicator detection unit 121 determines that the captured image represented by the first imaging information 117 does not include either the right hand H1 or the left hand H2, the projection control unit 124 displays an operation image GU3. In other words, if the detected pointer does not include the right hand H1 and the left hand H2, the item images GC1 to GC9 are arranged in the manner of the operation image GU3.

[0069] In step S108, the imaging control unit 123 controls the imaging unit 13 to capture an image of a range including the area on the wall surface W1 where the projection image GP is displayed and the space near the area on the wall surface W1. Then, the imaging control unit 123 acquires, from the imaging unit 13, second imaging information 118 representing the imaging result of capturing an image of the area including the area on the wall surface W1 where the projection image GP is displayed and the space near the area on the wall surface W1. Furthermore, the imaging control unit 123 stores the acquired second imaging information 118 in the storage unit 11.

[0070] In step S109, the pointer detection unit 121 determines, based on the second imaging information 118, whether or not a pointer is included in the captured image indicated by the second imaging information 118. If the result of the determination in step S109 is positive, that is, if YES in step S109, the pointer detection unit 121 proceeds to the process of step S110. If the result of the determination in step S109 is negative, that is, if NO in step S109, the pointer detection unit 121 proceeds to the process of step S108.

[0071] In step S110, the pointer detection unit 121 determines, based on the second imaging information 118, whether or not the pointer included in the captured image indicated by the second imaging information 118 is in contact with the wall surface W1. If the result of the determination in step S110 is positive, that is, if YES in step S110, the pointer detection unit 121 proceeds to step S111. If the result of the determination in step S110 is negative, that is, if NO in step S110, the pointer detection unit 121 proceeds to step S108.

[0072] After displaying the operation image GU in step S105, S106, or S107, the projector 1 detects the pointer again. Then, in step S110, the projector 1 determines, based on the captured image, whether or not the pointer is in contact with the wall surface W1, more specifically, whether or not the pointer is in contact with the projection image GP displayed on the wall surface W1. In other words, in step S110, the projector 1 determines whether or not the user has performed a pointing operation on the projection image GP. The projector 1 repeatedly executes the processes in steps S108 to S110 until the user has performed a pointing operation on the projection image GP.

[0073] In step S111, based on the second imaging information 118, the indication position detection unit 122 identifies the XY coordinates of the location where the indicator included in the captured image indicated by the second imaging information 118 is in contact with the wall surface W1.

[0074] In step S112, the processing unit 125 executes various processes based on the XY coordinates of the location where the indicator included in the captured image shown by the second imaging information 118 is in contact with the wall surface W1 and the item image GC displayed at that location. In other words, when the pointing position of the indicator included in the captured image shown by the second imaging information 118 is within an area where one of the multiple item images GC included in the operation image GU is displayed, the processing unit 125 executes the process corresponding to that one item image GC.

[0075] For example, when the operation image GU1 is displayed on the wall surface W1 and the pointing position of the right hand H1 is within area R31, the projector 1 executes eraser tool call processing corresponding to the item image GC2 displayed in area R31. Also, when the operation image GU1 is displayed on the wall surface W1 and the pointing position of the right hand H1 is within area R33, the projector 1 executes redo processing corresponding to the item image GC4 displayed in area R33. Also, when the operation image GU1 is displayed on the wall surface W1 and the pointing position of the right hand H1 is within area R21, the projector 1 executes drawing tool call processing corresponding to the item image GC1 displayed in area R21. Also, when the operation image GU1 is displayed on the wall surface W1 and the pointing position of the right hand H1 is within area R23, the projector 1 executes screen capture processing corresponding to the item image GC7 displayed in area R23.

[0076] For example, when the operation image GU2 is displayed on the wall surface W1 and the position of the left hand H2 is within area R33, the projector 1 executes eraser tool call processing corresponding to the item image GC2 displayed in area R33. When the operation image GU2 is displayed on the wall surface W1 and the position of the left hand H2 is within area R31, the projector 1 executes redo processing corresponding to the item image GC4 displayed in area R31. When the operation image GU2 is displayed on the wall surface W1 and the position of the left hand H2 is within area R23, the projector 1 executes drawing tool call processing corresponding to the item image GC1 displayed in area R23. When the operation image GU2 is displayed on the wall surface W1 and the position of the left hand H2 is within area R21, the projector 1 executes screen capture processing corresponding to the item image GC7 displayed in area R21.

[0077] For example, when the operation image GU3 is displayed on the wall surface W1 and the position pointed to by the indicator H9 is within the area R21, the projector 1 executes eraser tool call processing corresponding to the item image GC2 displayed in the area R21. When the operation image GU3 is displayed on the wall surface W1 and the position pointed to by the indicator H9 is within the area R23, the projector 1 executes redo processing corresponding to the item image GC4 displayed in the area R23. When the operation image GU3 is displayed on the wall surface W1 and the position pointed to by the indicator H9 is within the area R31, the projector 1 executes drawing tool call processing corresponding to the item image GC1 displayed in the area R31. When the operation image GU3 is displayed on the wall surface W1 and the position pointed to by the indicator H9 is within the area R11, the projector 1 executes screen capture processing corresponding to the item image GC7 displayed in the area R11.

[0078] In addition, if the item image GC is not displayed at the location where the indicator included in the captured image shown by the second captured information 118 is in contact with the wall surface W1, in step S112, the processing unit 125 executes, for example, a process of terminating the display of the operation image GU.

[0079] After the operation in step S112 is performed, the control unit 12 ends the series of operations shown in the flowchart of FIG.

[0080] As described above, according to the first embodiment, the projector 1 detects the indicator used when the user performs an instruction operation, and further identifies the type of indicator, thereby changing the operation image GU to be displayed. In other words, the projector 1 can display an optimum operation image GU in accordance with the type of indicator.

[0081] Furthermore, according to the first embodiment, the layout of the multiple item images GC included in the operation image GU is changed according to the type of the identified indicator. That is, the projector 1 can display the item images GC related to frequently executed processes in a layout that is easy for the user to operate.

[0082] Furthermore, according to the first embodiment, when the projector 1 determines that the indicator does not include either the right hand H1 or the left hand H2, it can display an operation image GU in a manner different from when the indicator includes the right hand H1 and when the indicator includes the left hand H2. In other words, even when the type of indicator cannot be accurately identified, for example, the projector 1 can display the operation image GU in a versatile manner.

[0083] As described above, the display method according to the first embodiment includes detecting an indicator and displaying an operation image GU including item images GC1 to GC9, and when the indicator includes a right hand H1, the item images GC1 to GC9 are arranged in the operation image GU in the manner of operation image GU1, and when the indicator includes a left hand H2, the item images GC1 to GC9 are arranged in the operation image GU in the manner of operation image GU2.

[0084] Furthermore, the projector 1 according to the first embodiment includes one or more CPUs, which detect an indicator and display an operation image GU including item images GC1 to GC9. When the indicator includes a right hand H1, the item images GC1 to GC9 are arranged in the operation image GU in the manner of operation image GU1, and when the indicator includes a left hand H2, the item images GC1 to GC9 are arranged in the operation image GU in the manner of operation image GU2.

[0085] That is, the projector 1 according to this embodiment detects the pointer when the user performs a pointing operation, and can change the arrangement of the multiple item images GC depending on whether the pointer includes the right hand H1 or the left hand H2. This allows the projector 1 to display an easy-to-operate operation image GU according to the type of pointer.

[0086] In the first embodiment, the projector 1 is an example of a "display device," the item images GC1 to GC9 are an example of a "plurality of item images," the operation image GU is an example of an "operation image," the right hand H1 is an example of a "right hand," the left hand H2 is an example of a "left hand," the CPU is an example of a "processor," the detection unit 120 is an example of a "detection unit," and the projection control unit 124 is an example of a "display control unit." Also, the right hand H1 and the left hand H2 are examples of an "indicator," the aspect of the operation image GU1 is an example of a "first aspect," and the aspect of the operation image GU2 is an example of a "second aspect."

[0087] Furthermore, in the display method according to the first embodiment, the operation image GU has an area R31 and an area R33, and the item images GC1 to GC9 include an item image GC2 related to the eraser tool call process, and in the operation image GU1, the item image GC2 is arranged in the area R31, and in the operation image GU2, the item image GC2 is arranged in the area R33.

[0088] As a result, the projector 1 according to this embodiment can display the item images GC related to the predetermined process in different arrangements according to the type of the indicator.

[0089] In the first embodiment, the region R31 is an example of a "first region," the region R33 is an example of a "second region," the eraser tool call process is an example of a "first process," the operation image GU1 is an example of a "first mode operation image," and the operation image GU2 is an example of a "second mode operation image." Also, the item image GC2 is an example of an "item image related to the first process."

[0090] Furthermore, in the display method according to the first embodiment, the item images GC1 to GC9 include an item image GC4 related to redo processing, and in the operation image GU1, the item image GC4 is placed in the area R33, and in the operation image GU2, the item image GC4 is placed in the area R31.

[0091] As a result, the projector 1 according to this embodiment can display two types of item images GC relating to different processes by switching the positions according to the type of the pointer.

[0092] In the first embodiment, the redo process is an example of a “second process.” Also, the item image GC4 is an example of an “item image related to the second process.”

[0093] In addition, the display method according to the first embodiment further includes detecting the pointing position of the indicator, executing an eraser tool calling process if the pointing position is within the area where item image GC2 is displayed, and executing a redo process if the pointing position is within the area where item image GC4 is displayed, and area R31 is to the left of area R33 as seen from the user.

[0094] That is, the projector 1 according to this embodiment can execute a process corresponding to the item image GC displayed at the indicated position in response to a user's instruction operation. Furthermore, the projector 1 can display two types of item images GC relating to different processes by swapping their positions so that one is positioned on the right side and the other on the left side, depending on the type of pointer. This makes it less likely that the item image GC2 relating to the eraser tool call process will be hidden by the pointer, whether the pointer includes the right hand H1 or the left hand H2. This improves operability when performing an instruction operation on the item image GC2.

[0095] In the first embodiment, an example of the "pointing position of the indicator" is "the location where the fingertip H10 of the right hand H1 is in contact with the wall surface W1."

[0096] Furthermore, in the display method according to the first embodiment, the eraser tool calling process is executed more frequently than the redo process.

[0097] That is, when the indicator includes the right hand H1, the projector 1 according to this embodiment can display the item image GC2 related to a process that is frequently executed to the left of the item image GC4 related to a process that is less frequently executed. Also, when the indicator includes the left hand H2, the projector 1 can display the item image GC2 related to a process that is frequently executed to the right of the item image GC4 related to a process that is less frequently executed. This allows the projector 1 to display the item image GC related to a process that is frequently executed in a highly visible arrangement according to the type of indicator.

[0098] Furthermore, in the display method according to the first embodiment, when the indicator does not include either the right hand H1 or the left hand H2, the item images GC1 to GC9 are arranged in the operation image GU in the manner of the operation image GU3.

[0099] That is, when the projector 1 according to this embodiment determines that the indicator does not include either the right hand H1 or the left hand H2, it can display an operation image GU in a manner different from when the indicator includes the right hand H1 and when the indicator includes the left hand H2. This allows the projector 1 to display the operation image GU in a versatile manner, even when, for example, it cannot accurately identify the type of indicator.

[0100] In the first embodiment, the mode of the operation image GU3 is an example of the "third mode."

[0101] Furthermore, in the display method according to the first embodiment, the operation image GU has an area R31 and an area R21, and the area R31 is above the area R21 as viewed from the user, and the item images GC1 to GC9 include an item image GC2 relating to an eraser tool calling process and an item image GC1 relating to a drawing tool calling process, and further includes detecting the pointing position of the pointer, and if the pointing position is within the area in which the item image GC2 is displayed, executing the eraser tool calling process, and if the pointing position is within the area in which the item image GC1 is displayed, executing the drawing tool calling process, and in the operation image GU1, the item image GC2 is placed in the area R31, and the item image GC1 is placed in the area R21, and in the operation image GU3, the item image GC2 is placed in the area R21, and the item image GC1 is placed in the area R31.

[0102] As a result, the projector 1 according to this embodiment can execute a process corresponding to the item image GC displayed at the indicated position in response to a user's instruction operation. Furthermore, the projector 1 can display two types of item images GC relating to different processes by swapping their positions so that one is positioned on the top and the other on the bottom, in accordance with the type of pointer.

[0103] In the first embodiment, the area R21 is an example of a "third area," the drawing tool call process is an example of a "third process," the operation image GU3 is an example of a "third mode operation image," and the item image GC1 is an example of an "item image related to the third process."

[0104] Furthermore, in the display method according to the first embodiment, the drawing tool calling process is executed more frequently than the eraser tool calling process.

[0105] That is, when the pointer does not include either the right hand H1 or the left hand H2, the projector 1 according to this embodiment can display the item image GC1 related to the frequently executed process above the item image GC2 related to the less frequently executed process. This allows the projector 1 to display the item image GC related to the frequently executed process in a highly visible arrangement, even when the type of the pointer cannot be accurately identified, for example.

[0106] 2. Second embodiment A second embodiment of the present invention will be described below. In each of the following exemplary embodiments, the elements having the same actions and functions as those of the first embodiment will be designated by the same reference numerals as those used in the description of the first embodiment, and detailed descriptions thereof will be omitted as appropriate.

[0107] A projector 1A according to the second embodiment identifies the frequency at which a process corresponding to each of item images GC1 to GC9 is executed, and changes and displays the arrangement of item images GC1 to GC9 based on the identified frequencies. For convenience, the following describes the projector 1A according to the present embodiment by illustrating a case in which the frequency at which an eraser tool calling process corresponding to item image GC2 is executed and the frequency at which a redo process corresponding to item image GC4 is executed are identified, and the arrangement of item image GC2 and item image GC4 is changed based on the two identified frequencies.

[0108] Hereinafter, with reference to FIGS. 13 and 14, an operation image GU that is displayed when the user performs an input operation on the projector 1A according to the second embodiment will be described.

[0109] 13 is a schematic diagram for explaining operation image GU1A. Operation image GU1A is displayed on wall surface W1 when the redo process is executed more frequently than the eraser tool call process and when the detected pointer includes the right hand H1. Operation image GU1A is included in projected image GP1A. Operation image GU1A is a GUI that includes item images GC1 to GC9, similar to operation images GU1, GU2, and GU3. Operation image GU1A is configured similarly to operation image GU1, except that item image GC4 is placed in area R31 instead of item image GC2, and item image GC2 is placed in area R33 instead of item image GC4.

[0110] FIG. 14 is a schematic diagram for explaining operation image GU2A. Operation image GU2A is displayed on wall surface W1 when the redo process is executed more frequently than the eraser tool call process and when the detected pointer includes left hand H2. Operation image GU2A is included in projected image GP2A. Operation image GU2A is a GUI that includes item images GC1 to GC9, similar to operation images GU1, GU1A, GU2, and GU3. Operation image GU2A is configured similarly to operation image GU2, except that item image GC2 is placed in area R31 instead of item image GC4, and item image GC4 is placed in area R33 instead of item image GC2.

[0111] In addition, when the eraser tool calling process is executed more frequently than the redo process and the detected pointer includes the right hand H1, the projector 1 A In addition, when the eraser tool calling process is executed more frequently than the redo process and the detected pointer includes the left hand H2, the projector 1 A displays an operation image GU2 on the wall surface W1.

[0112] The configuration and functions of a projector 1A according to the second embodiment will be described below with reference to FIGS.

[0113] FIG. 15 is a block diagram showing the configuration of a projector 1A according to the second embodiment. FIG. 16 is a block diagram showing the configuration of a storage unit 11A according to the second embodiment. The projector 1A has the same configuration as the projector 1 according to the first embodiment, except that it includes a storage unit 11A instead of the storage unit 11 and a control unit 12A instead of the control unit 12. The storage unit 11A differs from the storage unit 11 according to the first embodiment in that it stores a program 110A instead of the program 110, stores operation image information 112A instead of the operation image information 112, and stores frequency information 111 in addition to the imaging information 116, etc. The operation image information 112A differs from the operation image information 112 in that it includes first operation image information 113A and second operation image information 114A in addition to the first operation image information 113, second operation image information 114, and third operation image information 115, etc. The first operation image information 113A is information representing the operation image GU1A. Second operation image information 114A is information that represents operation image GU2A. Control unit 12A is configured similarly to control unit 12 according to the first embodiment, except that control unit 12A has the function of projection control unit 124A instead of projection control unit 124, and has the function of frequency management unit 126 in addition to detection unit 120, processing unit 125, etc. A CPU or the like included in control unit 12A executes program 110A and operates in accordance with program 110A, so that control unit 12A functions as detection unit 120, projection control unit 124A, processing unit 125, and frequency management unit 126 shown in FIG. 15 .

[0114] The frequency management unit 126 manages the frequency at which a process corresponding to each of the multiple item images GC included in the operation image GU is executed. Specifically, the frequency management unit 126 manages frequency information 111 that indicates the frequency at which a process corresponding to each of the multiple item images GC included in the operation image GU is executed. More specifically, the frequency management unit 126 updates the frequency information 111 every time one of the processes corresponding to each of the multiple item images GC included in the operation image GU is executed. For example, when a process corresponding to one item image GC out of the multiple item images GC included in the operation image GU is executed, the frequency management unit 126 reflects in the frequency information 111 that the process corresponding to that one item image GC has been executed.

[0115] Furthermore, the frequency management unit 126 refers to the frequency information 111 and specifies how often a process corresponding to each of the multiple item images GC included in the operation image GU is executed. In this embodiment, the frequency management unit 126 refers to the frequency information 111 and specifies how often an eraser tool call process is executed and how often a redo process is executed. This allows the projector 1A to specify which of the eraser tool call process and the redo process is executed more frequently.

[0116] The projection control unit 124A controls the projection unit 15 to project projection light for displaying the projection image GP onto the wall surface W1.

[0117] In this embodiment, when the eraser tool calling process is executed more frequently than the redo process and the indicator included in the captured image represented by the first imaging information 117 includes the right hand H1, the projection control unit 124A projects projection light for displaying the projection image GP1 onto the wall surface W1. That is, the projection control unit 124A controls the projection unit 15 to display the operation image GU1 represented by the first operation image information 113 on the wall surface W1. Furthermore, when the eraser tool calling process is executed more frequently than the redo process and the indicator included in the captured image represented by the first imaging information 117 includes the left hand H2, the projection control unit 124A projects projection light for displaying the projection image GP2 onto the wall surface W1. That is, the projection control unit 124A controls the projection unit 15 to display the operation image GU2 represented by the second operation image information 114 on the wall surface W1.

[0118] Furthermore, when the redo process is executed more frequently than the eraser tool calling process and the indicator included in the captured image represented by the first imaging information 117 includes the right hand H1, the projection control unit 124A projects projection light for displaying the projection image GP1A onto the wall surface W1. That is, the projection control unit 124A controls the projection unit 15 to display the operation image GU1A represented by the first operation image information 113A on the wall surface W1. Furthermore, when the redo process is executed more frequently than the eraser tool calling process and the indicator included in the captured image represented by the first imaging information 117 includes the left hand H2, the projection control unit 124A projects projection light for displaying the projection image GP2A onto the wall surface W1. That is, the projection control unit 124A controls the projection unit 15 to display the operation image GU2A represented by the second operation image information 114A on the wall surface W1.

[0119] Furthermore, when the indicator included in the captured image represented by the first imaging information 117 does not include either the right hand H1 or the left hand H2, the projection control unit 124A projects projection light for displaying the projection image GP3 onto the wall surface W1. That is, the projection control unit 124A controls the projection unit 15 to display the operation image GU3 represented by the third operation image information 115 on the wall surface W1.

[0120] Fig. 17 is a flowchart for explaining the operation of projector 1A according to embodiment 2. The flowchart shown in Fig. 17 is similar to the flowchart shown in Fig. 12 except that control unit 12A executes the process of step S122 instead of step S105, executes the process of step S123 instead of step S106, and executes the processes of steps S121 and S124 in addition to the processes of steps S101 to S103 and steps S107 to S112.

[0121] In step S121, the frequency management unit 126 identifies the frequency with which the eraser tool calling process is executed and the frequency with which the redo process is executed, with reference to the frequency information 111. That is, the frequency management unit 126 identifies which of the eraser tool calling process and the redo process is executed more frequently.

[0122] If the eraser tool calling process is executed more frequently than the redo process, in step S122, the projection control unit 124A controls the projection unit 15 to project projection light for displaying the projection image GP1 onto the wall surface W1. That is, the projection control unit 124A controls the projection unit 15 to display the operation image GU1 indicated by the first operation image information 113 on the wall surface W1.

[0123] If the redo process is executed more frequently than the eraser tool calling process, in step S122, the projection control unit 124A controls the projection unit 15 to project projection light for displaying the projection image GP1A onto the wall surface W1. That is, the projection control unit 124A controls the projection unit 15 to display the operation image GU1A indicated by the first operation image information 113A on the wall surface W1.

[0124] If the eraser tool calling process is executed more frequently than the redo process, in step S123, the projection control unit 124A controls the projection unit 15 to project projection light for displaying the projection image GP2 onto the wall surface W1. That is, the projection control unit 124A controls the projection unit 15 to display the operation image GU2 indicated by the second operation image information 114 on the wall surface W1.

[0125] If the redo process is executed more frequently than the eraser tool calling process, in step S123, the projection control unit 124A controls the projection unit 15 to project projection light for displaying the projection image GP2A onto the wall surface W1. That is, the projection control unit 124A controls the projection unit 15 to display the operation image GU2A indicated by the second operation image information 114A on the wall surface W1.

[0126] In step S124, the frequency management unit 126 updates the frequency information 111 based on the processing executed in step S112. In other words, the frequency management unit 126 causes the frequency information 111 to reflect that the processing executed in step S112 has been executed.

[0127] The above describes an example in which the arrangement of the item images GC2 and GC4 is changed based on the frequency with which the eraser tool call process is executed and the frequency with which the redo process is executed, but the present invention is not limited to this example. That is, the frequency with which processes corresponding to each of the multiple item images GC included in the operation image GU are executed, not limited to the eraser tool call process and the redo process, can be identified, and the arrangement of the item images GC can be changed based on the identified frequencies. For example, the frequency with which the drawing tool call process corresponding to the item image GC1 is executed and the frequency with which the screen capture process corresponding to the item image GC7 is executed can be measured, and the arrangement of the item images GC1 and GC7 can be changed based on the two identified frequencies.

[0128] As described above, according to the second embodiment, the projector 1A can identify the frequency with which a process corresponding to each of the multiple item images GC included in the operation image GU is executed, and change the arrangement of the item images GC based on the frequency. In other words, the projector 1A can arrange the item images GC related to the processes that are executed frequently in positions that are easy to operate.

[0129] As described above, in the display method according to the second embodiment, the operation image GU has the region R31 and the region R33, the region R31 is to the left of the region R33 as viewed from the user, the item images GC1 to GC9 include the item image GC2 relating to the eraser tool calling process and the item image GC4 relating to the redo process, and the method includes detecting the pointing position of the pointer, executing the eraser tool calling process if the pointing position is within the region where the item image GC2 is displayed, executing the redo process if the pointing position is within the region where the item image GC4 is displayed, and specifying the frequency at which the eraser tool calling process is executed and the frequency at which the redo process is executed. , and when the frequency with which the eraser tool calling process is executed is higher than the frequency with which the redo process is executed, in the operation image GU1, the item image GC2 is placed in the region R31 and the item image GC4 is placed in the region R33, and in the operation image GU2, the item image GC2 is placed in the region R33 and the item image GC4 is placed in the region R31; and when the frequency with which the redo process is executed is higher than the frequency with which the eraser tool calling process is executed, in the operation image GU1A, the item image GC2 is placed in the region R33 and the item image GC4 is placed in the region R31, and in the operation image GU2A, the item image GC2 is placed in the region R31 and the item image GC4 is placed in the region R33.

[0130] That is, the projector 1A according to this embodiment can specify the frequency with which a process corresponding to each of the multiple item images GC included in the operation image GU is executed, and change the arrangement of the item images GC based on the frequency. This allows the projector 1A to display the item images GC related to the processes that are executed frequently in an arrangement that is easier to operate than when the frequency with which the process is executed is not specified, depending on the frequency with which the process is executed and the type of pointer.

[0131] In the second embodiment, operation image GU is an example of an "operation image," region R31 is an example of a "first region," region R33 is an example of a "second region," item images GC1 to GC9 are examples of "plurality of item images," eraser tool calling processing is an example of a "first process," and redo processing is an example of a "second process." Also, item image GC2 is an example of an "item image related to the first process," item image GC4 is an example of an "item image related to the second process," operation image GU1 and operation image GU1A are examples of a "first aspect of operation image," and operation image GU2 and operation image GU2A are examples of a "second aspect of operation image."

[0132] 3. Variations Each of the above embodiments can be modified in various ways. Specific modified embodiments are exemplified below. Furthermore, two or more embodiments arbitrarily selected from the following examples can be appropriately combined within a range that does not contradict each other. For elements in the modified examples exemplified below that have the same actions and functions as the above-described embodiments, the symbols used in the above explanations will be used and detailed explanations of each will be omitted where appropriate.

[0133] 3.1. Variation 1 In the above-described embodiment, an example was given in which the operation image GU displayed when the detected indicator includes a left hand and the operation image GU displayed when the detected indicator includes a right hand are displayed in a manner in which only the arrangement of the item images GC differs, but the present invention is not limited to such an embodiment. For example, the design of the item images GC included in the operation image GU displayed when the detected indicator includes a left hand may be different from the design of the item images GC included in the operation image GU displayed when the detected indicator includes a right hand.

[0134] FIG. 18 is a schematic diagram for explaining operation image GU2m. Operation image GU2m is an image obtained by flipping operation image GU1 horizontally. Operation image GU2m is included in projection image GP2m. Operation image GU2m is displayed on wall surface W1 when the detected pointer includes left hand H2. Operation image GU2m is a GUI including item images GC1m to GC9m. Item images GC1m to GC9m are images obtained by flipping item images GC1 to GC9 horizontally, respectively.

[0135] Like the item image GC1, the item image GC1m is an item image related to the drawing tool call process. In the operation image GU2m, the item image GC1m is placed in the region R23. Like the item image GC2, the item image GC2m is an item image related to the eraser tool calling process. In the operation image GU2m, the item image GC2m is placed in the area R33. Like the item image GC3, the item image GC3m is an item image related to the movement tool calling process. In the operation image GU2m, the item image GC3m is placed in the area R32. Like the item image GC4, the item image GC4m is an item image related to the redo process. In the operation image GU2m, the item image GC4m is arranged in the area R31. Like the item image GC5, the item image GC5m is an item image related to the screen magnification tool calling process. In the operation image GU2m, the item image GC5m is arranged in the region R22. The item image GC6m is an item image related to the save process, similar to the item image GC6. In the operation image GU2m, the item image GC6m is arranged in the region R13. Like the item image GC7, the item image GC7m is an item image related to the screen capture process. In the operation image GU2m, the item image GC7m is arranged in the region R21. Like the item image GC8, the item image GC8m is an item image related to the setting screen call process. In the operation image GU2m, the item image GC8m is arranged in the region R12. Like the item image GC9, the item image GC9m is an item image related to the cutout tool calling process. In the operation image GU2m, the item image GC9m is placed in the region R11.

[0136] That is, in the operation image GU2m, the item image GC2m arranged in the region R33 is arranged to the right of the item image GC4m arranged in the region R31 as viewed from the user. Furthermore, the item image GC1m arranged in the region R23 is arranged to the right of the item image GC7m arranged in the region R21 as viewed from the user.

[0137] As described above, according to Modification 1, operation image GU2m is displayed in a manner that is a left-right inversion of operation image GU1. That is, operation image GU2m can be displayed by simply displaying operation image GU1 in a left-right inversion manner without creating operation image GU2 in advance.

[0138] The image of the item image GC included in the operation image GU may be flipped not only horizontally but also vertically. The image of the item image GC included in the operation image GU may be displayed rotated 90 degrees clockwise, for example. The operation image GU displayed when the detected indicator includes a left hand and the operation image GU displayed when the detected indicator includes a right hand may be displayed in different colors.

[0139] As described above, in the display method according to the first modification, the operation image GU2m is an image obtained by flipping the operation image GU1 horizontally.

[0140] That is, the operation image GU2m according to this modification is displayed in a manner in which the operation image GU1 is flipped left and right. As a result, even if the projector does not store the second operation image information 114, when the detected indicator includes the left hand H2, the operation image GU1 is flipped left and right, so that the operation image GU2m with good operability can be displayed on the wall surface W1.

[0141] In Modification 1, operation image GU2m is an example of a "second-mode operation image," and operation image GU1 is an example of a "first-mode operation image."

[0142] 3.2. Variation 2 In the above-described embodiment and modified examples, the operation image is a rectangular image, but the present invention is not limited to this. For example, the operation image may be an image having a circular outline.

[0143] Hereinafter, operation image GR according to Modification 2 will be described with reference to Figs. 19 and 20. Fig. 19 is a schematic diagram for explaining operation image GR1. Operation image GR1 is displayed when the detected indicator includes a right hand H1. Fig. 20 is a schematic diagram for explaining operation image GR2. Operation image GR2 is displayed when the detected indicator includes a left hand H2. In this modification, operation image GR is a collective term for operation image GR1 and operation image GR2.

[0144] The operation image GR has regions R41, R42, R51, R52, R61, R71, R72, R81, and R82. The operation image GR is a GUI that includes item images GC1 to GC9.

[0145] In this modification, regions R51 and R52 are considered to be located higher than regions R41 and R42 when viewed from the user's perspective. Region R61 is considered to be located higher than regions R51 and R52 when viewed from the user's perspective. Regions R71 and R72 are considered to be located higher than region R61 when viewed from the user's perspective. Regions R81 and R82 are considered to be located higher than regions R71 and R72 when viewed from the user's perspective.

[0146] In this modification, regions R42 and R82 are considered to be located to the left of regions R52 and R72 as seen from the user. Region R61 is considered to be located to the left of regions R42 and R82 as seen from the user. Regions R41 and R81 are considered to be located to the left of region R61 as seen from the user. Regions R51 and R71 are considered to be located to the left of regions R41 and R81 as seen from the user.

[0147] In the operation image GR1, the item image GC1 is arranged in an area R71. In addition, in the operation image GR1, the item image GC2 is arranged in an area R81. In addition, in the operation image GR1, the item image GC3 is arranged in an area R61. In addition, in the operation image GR1, the item image GC4 is arranged in the area R82. In addition, in the operation image GR1, the item image GC5 is arranged in an area R51. In addition, in the operation image GR1, the item image GC6 is arranged in the area R41. In addition, in the operation image GR1, the item image GC7 is arranged in the area R42. In addition, in the operation image GR1, the item image GC8 is placed in the area R72. In addition, in the operation image GR1, the item image GC9 is arranged in the area R52.

[0148] That is, in the operation image GR1, the item image GC2 arranged in the region R81 is arranged to the left of the item image GC4 arranged in the region R82 as viewed from the user. Furthermore, the item image GC1 arranged in the region R71 is arranged to the left of the item image GC7 arranged in the region R42 as viewed from the user.

[0149] In the operation image GR2, the item image GC1 is arranged in the area R72. In addition, in the operation image GR2, the item image GC2 is arranged in the area R82. In addition, in the operation image GR2, the item image GC3 is arranged in the area R61. In addition, in the operation image GR2, the item image GC4 is arranged in the area R81. In addition, in the operation image GR2, the item image GC5 is arranged in the area R52. In addition, in the operation image GR2, the item image GC6 is arranged in the area R42. In addition, in the operation image GR2, the item image GC7 is arranged in the area R41. In addition, in the operation image GR2, the item image GC8 is arranged in the area R71. In addition, in the operation image GR2, the item image GC9 is arranged in the area R51.

[0150] That is, in the operation image GR2, the item image GC2 arranged in the region R82 is arranged to the right of the item image GC4 arranged in the region R81 as viewed from the user. Furthermore, the item image GC1 arranged in the region R72 is arranged to the right of the item image GC7 arranged in the region R41 as viewed from the user.

[0151] The shape of the operation image is not limited to a rectangle or a circle, but may be, for example, an ellipse or a polygon such as a regular hexagon. The shape of the operation image may be changed depending on the situation in which the operation image is displayed, the user's preferences, etc. In other words, it is possible to display an operation image of an optimal shape depending on the situation.

[0152] As described above, according to the second modification, similarly to the above-described embodiment and modification, the arrangement of the plurality of item images GC can be changed depending on whether the pointer includes the right hand H1 or the left hand H2. In other words, it is possible to display an operation image GR with good operability according to the type of pointer and the usage situation.

[0153] 3.3. Variation 3 In the above-described embodiment and modified examples, examples have been given in which the indicator is the right hand H1, the left hand H2, or the indicator H9, but the present invention is not limited to such an embodiment. For example, a user may use the right hand holding an electronic pen or the left hand holding an electronic pen as the indicator. For example, the projector detects the right hand holding an electronic pen as an indicator including the right hand. Furthermore, the projector detects the left hand holding an electronic pen as an indicator including the left hand. A pointing operation on an item image GC may be performed, for example, by touching the area where the item image GC is displayed with the tip of the electronic pen. Alternatively, in a case in which a cursor that moves in conjunction with the electronic pen is displayed on the projected image GP, and the cursor is positioned on the item image GC, the user may perform a pointing operation on the item image GC by pressing a button on the electronic pen.

[0154] 3.4. Variation 4 In the above-described embodiment and modified examples, a projector that projects an image has been used to illustrate the display method and display device according to the present invention, but the present invention is not limited to such an embodiment. For example, the display device according to the present invention is not limited to a projector, and a direct-view display device may also be used. Specifically, instead of the projection unit 15 that projects the projection image GP, a display panel that displays the image may be provided. More specifically, the display device may be an interactive whiteboard that includes a camera that can capture an image of a pointer, or that can communicate with the camera. [Explanation of symbols]

[0155] 1...projector, 11...memory unit, 12...control unit, 13...imaging unit, 14...operation unit, 15...projection unit, 110...program, 111...frequency information, 112...operation image information, 113...first operation image information, 114...second operation image information, 115...third operation image information, 116...imaging information, 117...first imaging information, 118...second imaging information, 120...detection unit, 121...pointer detection unit, 122...pointing position detection unit, 123...imaging control unit, 124...projection control unit, 125...processing unit, 126...frequency management unit, GP...projected image, GU...operation image, GR...operation image, GC...item image, H1...right hand, H2...left hand, H9...pointer, H10...fingertip, R11...area, W1...wall surface.

Claims

1. Detecting an indicator; Displaying an operation image including a plurality of item images; Including, When the indicator includes a right hand, the plurality of item images in the operation image are are arranged in a manner When the indicator includes a left hand, the plurality of item images in the operation image are are arranged in a manner the operation image has a first area and a second area, the first area is to the left of the second area as viewed from the user, The plurality of item images include an item image related to a first process and an item image related to a second process. and Detecting a pointing position of the indicator; When the pointing position is within an area where an item image related to the first process is displayed, executing the first process; When the pointing position is within an area where an item image related to the second process is displayed, executing the second process; Specifying a frequency at which the first process is executed and a frequency at which the second process is executed. And, Further comprising: When the frequency of execution of the first process is higher than the frequency of execution of the second process 、 In the operation image of the first aspect, the item image relating to the first process is an item image relating to the second process is arranged in the first area, and an item image relating to the second process is arranged in the second area; In the operation image of the second aspect, the item image relating to the first process is and the item image relating to the second process is arranged in the first area, When the frequency of execution of the second process is higher than the frequency of execution of the first process 、 In the operation image of the first aspect, the item image relating to the first process is and the item image relating to the second process is arranged in the first area, In the operation image of the second aspect, the item image relating to the first process is and the item image relating to the second process is arranged in the first area, and the item image relating to the second process is arranged in the second area. A display method comprising:

2. The operation image of the second aspect is an image obtained by flipping the operation image of the first aspect horizontally. That is, 2. The display method according to claim 1, wherein:

3. When the indicator does not include either the right hand or the left hand, in the operation image The plurality of item images are arranged in a third manner.

2. The display method according to claim 1, wherein:

4. the operation image has a third area, the first region is located above the third region as viewed from the user; the plurality of item images includes an item image related to a third process, When the pointing position is within an area where an item image related to the third process is displayed, performing the third process, In the operation image of the first aspect, the item image relating to the first process is the item image relating to the third process is arranged in the first area, and the item image relating to the third process is arranged in the third area; The frequency with which the third process is executed in the operation image of the third aspect is When the frequency of execution of the process of the first type is higher than the frequency of execution of the process of the third type, the item image related to the first type is an item image relating to the third processing is arranged in the first area; 4. The display method according to claim 3, wherein:

5. one or more processors; The processor: Detecting an indicator; Displaying an operation image including a plurality of item images; Run When the indicator includes a right hand, the plurality of item images in the operation image include a first are arranged in a manner When the indicator includes a left hand, the plurality of item images in the operation image are are arranged in a manner the operation image has a first area and a second area, the first area is to the left of the second area as viewed from the user, The plurality of item images include an item image related to a first process and an item image related to a second process. and Detecting a pointing position of the indicator; When the pointing position is within an area where an item image related to the first process is displayed, executing the first process; When the pointing position is within an area where an item image related to the second process is displayed, executing the second process; Specifying a frequency at which the first process is executed and a frequency at which the second process is executed. And, Further comprising: When the frequency of execution of the first process is higher than the frequency of execution of the second process 、 In the operation image of the first aspect, the item image relating to the first process is an item image relating to the second process is arranged in the first area, and an item image relating to the second process is arranged in the second area; In the operation image of the second aspect, the item image relating to the first process is and the item image relating to the second process is arranged in the first area, When the frequency of execution of the second process is higher than the frequency of execution of the first process 、 In the operation image of the first aspect, the item image relating to the first process is and the item image relating to the second process is arranged in the first area, In the operation image of the second aspect, the item image relating to the first process is and the item image relating to the second process is arranged in the first area, and the item image relating to the second process is arranged in the second area. A display device characterized by:

Images