Display method, display system, and display program

Abstract

This invention provides a display method, display system, and display program that improve operability in user operation instructions. [Solution] In the projection system 1, the processor performs the following actions: determine whether the first indicator that indicates a position on the display surface 10 is an indicator pen; if the first indicator is determined to be an indicator pen, display a first image 200 on the display surface 10 that includes icon images 201 to 205 that accept operation instructions from the indicator pen; determine whether the second indicator that indicates a position on the display surface 10 is the user's finger; if the second indicator is determined to be the user's finger, display a second image 300 on the display surface 10 that includes icon images 301 to 304 that accept operation instructions from the user's finger; the smallest display size image among the icon images 301 to 304 is larger than the smallest display size image among the icon images 201 to 205.

Description

Technical Field

[0001] The present invention relates to a display method, a display system, and a display program.

Background Art

[0002] Conventionally, a display system having a GUI that accepts operations by a user's finger or an operation by a touch pen is known.

[0003] In Patent Document 1, in response to detecting a predetermined touch operation, by displaying a partial image that displays a setting state of a process executed by a control unit, a user can easily confirm what process is assigned to a touch operation by a finger. Also, in Patent Document 1, by displaying a partial image that displays a setting state of a process assigned to a touch operation by a touch pen, a user can easily confirm what process is currently assigned to a pen operation.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the above Patent Document 1, when performing an operation by a user's finger using a partial image, there is a case where the image is rather small for selection by a finger and the selection operation is difficult. For this reason, the operability by a user's finger may deteriorate.

Means for Solving the Problems

[0006] One aspect of the present disclosure is a display method in which a processor performs the following actions: determine whether a first indicator that indicates a position on a display surface is an indicator pen; if the first indicator is determined to be the indicator pen, display a first image on the display surface that includes a plurality of first icon images that accept operation instructions from the indicator pen; determine whether a second indicator that indicates a position on the display surface is a user's finger; and if the second indicator is determined to be the user's finger, display a second image on the display surface that includes a plurality of second icon images that accept operation instructions from the user's finger, wherein the second small image, which is one of the plurality of second icon images and has the smallest display size, has a larger display size than the first small image, which is one of the plurality of first icon images and has the smallest display size.

[0007] One aspect of the present disclosure is a display system comprising a processor that performs the following: determining whether a first indicator that indicates a position on a display surface is an indicator pen; if the first indicator is determined to be the indicator pen, displaying a first image on the display surface that includes a plurality of first icon images that accept operation instructions from the indicator pen; determining whether a second indicator that indicates a position on the display surface is a user's finger; and if the second indicator is determined to be the user's finger, displaying a second image on the display surface that includes a plurality of second icon images that accept operation instructions from the user's finger, wherein the second small image, which is one of the plurality of second icon images and has the smallest display size, has a larger display size than the first small image, which is one of the plurality of first icon images and has the smallest display size.

[0008] One aspect of the present disclosure is a display program that causes a processor to perform the following actions: determine whether a first indicator that indicates a position on a display surface is an indicator pen; if the first indicator is determined to be the indicator pen, display a first image on the display surface that includes a plurality of first icon images that accept operation instructions from the indicator pen; determine whether a second indicator that indicates a position on the display surface is a user's finger; if the second indicator is determined to be the user's finger, display a second image on the display surface that includes a plurality of second icon images that accept operation instructions from the user's finger, wherein the second small image, which is one of the plurality of second icon images and has the smallest display size, has a larger display size than the first small image, which is one of the plurality of first icon images and has the smallest display size. [Brief explanation of the drawing]

[0009] [Figure 1] A perspective view showing an example of a projection system configuration. [Figure 2] A side view showing an example of the side of a projection system. [Figure 3] A diagram showing an example configuration of a projector and a first indicator. [Figure 4] A flowchart illustrating an example of the operation of the first control unit. [Figure 5] An explanatory diagram showing an example of the first image. [Figure 6] An explanatory diagram showing an example of the second image. [Figure 7] An explanatory diagram illustrating the minimum size of icon images. [Figure 8] An explanatory diagram showing an example of the third image. [Figure 9] An explanatory diagram illustrating the movement of the second image on the display surface. [Figure 10] An explanatory diagram illustrating the display of a third image created by combining the first and second images. [Modes for carrying out the invention]

[0010] Embodiments relating to this disclosure will be described in detail below with reference to the drawings. However, some descriptions may be omitted to avoid unnecessary detail. For example, detailed explanations of already well-known matters or redundant explanations of substantially identical configurations may be omitted.

[0011] In addition, this disclosure may use common abbreviations. For example, GUI is an abbreviation for Graphical User Interface. LED is an abbreviation for Light Emitting Diode. CPU is an abbreviation for Central Processing Unit. GPU is an abbreviation for Graphics Processing Unit. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for field-programmable gate array. LD is an abbreviation for Laser Diode. CMOS is an abbreviation for Complementary Metal Oxide Semiconductor. CCD is an abbreviation for Charge Coupled Device. RAM is an abbreviation for Random Access Memory. ROM is an abbreviation for Read Only Memory. MPU is an abbreviation for Micro Processor Unit. LSI is an abbreviation for Large Scale Integration. PLD is an abbreviation for Programmable Logic Device. MCU is an abbreviation for Micro Control Unit. SoC is an abbreviation for System-on-a-Chip. DVD is an abbreviation for Digital Versatile Disc. USB is an abbreviation for Universal Serial Bus. SSD is an abbreviation for Solid State Drive.

[0012] [1. Projection System Configuration] Figure 1 is a perspective view showing an example of the system configuration of a projection system. As shown in Figure 1, the projection system 1 comprises a projector 100, a display surface 10 onto which the projector 100 projects and displays an image, and a first indicator 50 used to specify a position on the display surface 10. The projection system 1 is an example of a display system. Specifying a position on the display surface 10 is also possible using a second indicator 70, such as the user's finger. In the following explanation, the normal to the display surface 10 will be referred to as the Z-axis, the vertical axis as the Y-axis, and the axis perpendicular to the Z-axis and Y-axis as the X-axis.

[0013] The projector 100 generates image light corresponding to the image data and projects the generated image light onto the display surface 10 to display the image. The projector 100 also has an interactive function. The interactive function is a function that displays images corresponding to the positions and trajectories of the first indicator 50 and the second indicator 70 based on the positions of the detected first indicator 50 and the second indicator 70, or modifies the displayed image.

[0014] The first indicator 50 is an example of an indicator pen that a user holds in their hand and has a light-emitting tip 51, a shaft 52 for the user to grip, etc. The tip 51 is equipped with a light source such as an LED that emits near-infrared light. The first indicator 50 emits light at a predetermined interval when the tip 51 is in contact with the display surface 10. The first indicator 50 also emits light at a predetermined interval even when the tip 51 is not in contact with the display surface 10. Although Figure 1 shows one first indicator 50, the number of first indicators 50 that can be used simultaneously is not limited to one, and it is possible to use multiple first indicators 50 at the same time.

[0015] Also, when using the user's finger as the second indicator 70, the user touches the tip such as the fingertip to the display surface 10 and designates a position on the display surface 10. The projector 100 emits detection light along the display surface 10. When the tip of the second indicator 70 touches the display surface 10, the detection light is blocked. This detection light is hereinafter referred to as the second light 180. The second light 180 hits the second indicator 70 and is reflected, and a part of the reflected light of the second light 180 travels from the second indicator 70 toward the projector 100. The projector 100 detects the reflected light of the second light 180 reflected by the second indicator 70 and detects the position indicated by the second indicator 70.

[0016] For example, the projector 100 detects a change in the position of the first indicator 50 or the second indicator 70 when the first indicator 50 or the second indicator 70 is in contact with the display surface 10. Also, the projector 100 can detect a change in the position of the first indicator 50 or the second indicator 70 when the first indicator 50 or the second indicator 70 is not in contact with the display surface 10. The state where the first indicator 50 or the second indicator 70 is not in contact with the display surface 10 here means that the first indicator 50 or the second indicator 70 is in the space where the range of the display surface is extended in the normal direction of the display surface 10. The distance in the normal direction when the first indicator 50 or the second indicator 70 is not in contact with the display surface 10 shall be within a predetermined threshold value. That is, the change in its position when the first indicator 50 or the second indicator 70 is not in contact with the display surface 10 corresponds to the hovering operation of the first indicator 50 or the second indicator 70 near the display surface 10.

[0017] As a result, the projector 100 can detect, for example, characters or figures drawn on the display surface 110 by the first indicator 50 or the second indicator 70 when the first indicator 50 or the second indicator 70 is brought into contact with the display surface 10. Note that the drawing of characters or figures may include erasing characters or figures by the function of an eraser. Also, the projector 100 can detect, for example, characters or figures drawn on the display surface 110 by the first indicator 50 or the second indicator 70 when the first indicator 50 or the second indicator 70 is not in contact with the display surface 10. Further, the projector 100 can detect, for example, a hovering operation in which the position of the first indicator 50 or the second indicator 70 changes while the first indicator 50 or the second indicator 70 is separated from the display surface 10 without contacting the display surface 10. As a specific example, the projector 100 can detect that a specific item has been indicated by the first indicator 50 or the second indicator 70 from a menu including a plurality of items displayed on the display surface 10.

[0018] FIG. 2 is a side view showing an example of a side of the projection system 1. As shown in FIG. 2, the projector 100 according to the present embodiment is fixed to a wall surface and installed in front of and above the display surface 10, and projects image light toward the obliquely downward display surface 10. The area of the display surface 10 onto which the image light is projected by the projector 100 is referred to as a projection area 20. Further, the projector 100 irradiates the second light 180 used for detecting the first indicator 五十 and the second indicator 70 in a direction corresponding to the display surface 10. Specifically, it is a direction in which the reflected light reflected by the second indicator 70 approaching within a predetermined distance from the display surface 10 can be made to enter the imaging unit 135. The second light 180 is light used for detecting the second indicator 70, and infrared light is used in the present embodiment. By using infrared light, the second indicator 70 can be detected without being affected by the image light mainly composed of visible light, and the display by the image light is not affected either. The second light 180 is irradiated in a range including at least a part of the display surface 10. In the present embodiment, it is projected in a range covering the entire display surface 10.

[0019] [2. Projector Configuration] Figure 3 is a configuration diagram showing the configuration of the projector 100 and the first indicator 50. First, the configuration of the projector 100 will be explained.

[0020] As shown in Figure 3, the projector 100 includes an image generation unit 110, a projection unit 120, a transmission unit 131, an illumination unit 133, an imaging unit 135, a light receiving unit 140, and a first control unit 150.

[0021] The image generation unit 110 is a processing unit composed of a CPU and a GPU. The image generation unit 110 has an image storage unit 115 that stores projected images. The image storage unit 115 is a so-called frame memory that stores images projected by the projection unit 120 in frame units.

[0022] The image generation unit 110 draws the projected image to be projected onto the display surface 10 into the image storage unit 115 based on the image data. The image generation unit 110 outputs an image signal indicating the image drawn in the image storage unit 115 to the optical modulator 123, causing the projection unit 120 to project image light corresponding to the image data onto the display surface 10.

[0023] Furthermore, the image generation unit 110 performs image processing on the image to be drawn to the image storage unit 115. For example, the image generation unit 110 performs geometric correction processing to correct trapezoidal distortion of the projection area 20, digital zoom processing to enlarge or reduce the size of the image to be displayed on the display surface 10, and color correction processing to correct the color tone of the image to be displayed on the display surface 10.

[0024] The projection unit 120 includes a light source 121, a light modulator 123, and an optical unit 125.

[0025] The light source 121 may include a xenon lamp, a high-pressure mercury lamp, an LED, a laser light source, or other light source. The light source 121 may also include a reflector and an auxiliary reflector to guide the light emitted from the light source to the light modulator 123.

[0026] The optical modulator 123 includes, for example, a modulation element such as a liquid crystal panel. The optical modulator 123 modulates the light incident from the light source 121 according to the image signal input from the image storage unit 115 to form image light. The image light is typically color image light containing three visible colors: red (R), green (G), and blue (B).

[0027] The optical unit 125 projects the image light formed by the optical modulator 123 onto the display surface 10 to form an image on the display surface 10. The optical unit 125 includes at least one of a lens and a mirror. The optical unit 125 may also include a zoom mechanism for enlarging or reducing the image projected onto the display surface 10, and a focus adjustment mechanism for adjusting the focus.

[0028] The transmitter 131 outputs a signal light 170, which is a signal for synchronizing the light emission timing of the first indicator 50 with the imaging timing of the imaging unit 135. The signal light 170 is shown as a dashed line in Figure 3. The signal light 170 is a near-infrared light signal that the first indicator 50 can receive with the receiver 53. The transmitter 131 periodically transmits the signal light 170 while the projector 100 is running.

[0029] The signal light 170 is, for example, a control signal that specifies the timing for the first indicator 50 to transmit the first light 190. The first light 190 is near-infrared light having a predetermined emission pattern. In Figure 3, the first light 190 is shown by a dashed line. The first indicator 50 transmits the first light 190 in synchronization with the timing of receiving the signal light 170.

[0030] Therefore, the projector 100 can cause the imaging unit 135 to perform imaging in accordance with the timing when the first indicator 50 emits the first light 190. The transmitting unit 131 includes, for example, a light source such as an LED and a device for controlling the turning on and off of the light source. The control device can be configured, for example, with an ASIC or FPGA.

[0031] The illumination unit 133 emits a second beam of light 180 in a direction including the display surface 10 to detect the first indicator 50 and the second indicator 70. The second beam of light 180 is shown by a dashed line in Figure 3. The illumination unit 133 has an LD or LED as a light source that emits infrared light. The illumination unit 133 may also include an optical component that diffuses the infrared light emitted by the light source toward the display surface 10.

[0032] The imaging unit 135 is a camera equipped with an image sensor such as a CMOS or CCD that receives near-infrared light emitted from the light-emitting part 55 of the first indicator 50. The imaging unit 135 also includes an optical system for forming an image on the image sensor, and an aperture for limiting the light incident on the image sensor.

[0033] The imaging unit 135 captures an area including the display surface 10 and generates an image. The imaging unit 135 receives the first light 190 emitted by the first indicator 50 and captures the image. The image generated by the imaging unit 135 is output to the first control unit 150.

[0034] When the imaging unit 135 performs imaging, it outputs an interrupt signal to the first control unit 150. When the imaging unit 135 receives a response corresponding to the interrupt signal from the first control unit 150, it outputs the captured image to the first control unit 150.

[0035] The light receiving unit 140 receives infrared signals transmitted from the remote control 5. The light receiving unit 140 generates an operation signal corresponding to the received infrared signal and outputs the generated operation signal to the first control unit 150. The operation signal is a signal corresponding to a switch on the remote control 5 operated by the user.

[0036] The first control unit 150 includes a processor 165 such as a CPU, a storage unit 160 such as memory, and various peripheral circuits. In other words, the first control unit 150 has the functionality of a computer. The storage unit 160 includes, for example, volatile memory such as RAM, which is used as a work area for the processor 165, and non-volatile memory such as ROM, which stores programs and setting information in advance. The storage unit 160 stores information related to the control program executed by the processor 165. This information related to the control program includes firmware, calibration data, as well as application programs 161, display image information 163, and history information 164.

[0037] In the projector 100, the correspondence between the image captured by the imaging unit 135 and the projection area 20 on the display surface 10 is performed using calibration data stored in the memory unit 160. More specifically, the calibration data is data that associates the coordinates of each pixel in the coordinate system of the image captured image with the coordinates of each pixel in the coordinate system of the image memory unit 115. These coordinates of the image captured image are called camera coordinates, and the coordinates of the image memory unit 115 are called panel coordinates. In the projector 100, by referring to the calibration data stored in the memory unit 160, the corresponding position in the projection area 20 is uniquely identified for each position on the image captured image.

[0038] Application program 161 is a program executed by processor 165. Application program 161 is a program that implements interactive functions. Specifically, application program 161 is read by processor 165 and executed sequentially. As a result, processor 165 performs various processes to display images, figures, characters, symbols, etc., corresponding to the positions on the display surface 10 indicated by the first indicator 50 and the second indicator 70.

[0039] The display image information 163 is information about the image to be displayed on the display surface 10. For example, the display image information 163 includes information such as a plurality of icon images that accept operation instructions from the first indicator 50 or the second indicator 70, and their display sizes. The display image information 163 also includes information such as a GUI image that displays the plurality of icon images, its display size, and the placement position of the icon images within the GUI image.

[0040] History information 164 is information that shows the history of operations related to the projector 100. For example, history information 164 includes the history of operation instructions given via icon images, etc., by the first indicator 50 or the second indicator 70. More specifically, history information 164 includes operation instructions such as select, confirm, enlarge, reduce, erase, cancel, redo, etc., and the order in which these operation instructions were received.

[0041] The processor 165 is composed of, for example, a CPU or an MPU. The processor 165 controls the various parts of the projector 100 by executing a control program. The first control unit 150 may be configured to include multiple processors.

[0042] The first control unit 150 detects the reflected light from the first light 190 and the second light 180 reflected by the second indicator 70 in the captured image. The first control unit 150 analyzes the captured image generated by the imaging unit 135 to determine the indicated positions of the first indicator 50 and the second indicator 70. The first control unit 150 generates coordinate information indicating the positions of the light points of the detected first light 190 and the light points of the reflected light of the second light 180. This coordinate information is expressed in camera coordinates.

[0043] The first control unit 150 converts coordinate information expressed in camera coordinates into coordinate information in panel coordinates. This conversion is performed, for example, based on calibration data generated by a prior calibration.

[0044] The first control unit 150 controls various parts of the projector 100 by having the processor 165 read and sequentially execute the application program 161, thereby displaying an image on the display surface 10. The first control unit 150 also generates drawing data representing images, figures, characters, symbols, etc., corresponding to the trajectories of positions on the display surface 10 indicated by the first indicator 50 and the second indicator 70. The first control unit 150 outputs the generated drawing data to the image generation unit 110. The first control unit 150 controls the image generation unit 110 to unfold the image data onto the coordinates on the image storage unit 115 based on the converted coordinate information. As a result, the drawing data is superimposed on the image data and displayed on the display surface 10.

[0045] [3. Configuration of the first indicator] Next, the configuration of the first indicator 50 will be described. As shown in Figure 3, the first indicator 50 comprises a tip portion 51, a shaft portion 52, a receiving portion 53, a tip switch 54, a light-emitting portion 55, a power supply portion 57, and a second control portion 58.

[0046] The receiving unit 53 includes a light-receiving element that receives infrared light and receives the signal light 170 transmitted by the projector 100. The receiving unit 53 outputs a control signal, etc., indicating the timing of the reception of the signal light 170 to the second control unit 58.

[0047] The tip switch 54 is a switch that turns on when the tip 51 contacts the display surface 10 and the tip 51 is pressed, and turns off when the contact between the tip 51 and the display surface 10 is released.

[0048] The light-emitting unit 55 includes an LED that emits near-infrared light, and its emission is controlled by the second control unit 58, which outputs a first light 190 that is near-infrared light.

[0049] The power supply unit 57 is equipped with a battery such as a primary battery, a secondary battery, or a photocell, and supplies power to each part of the first indicator unit 50. The first indicator unit 50 may be equipped with a power switch to turn the power supply from the power supply unit 57 on or off.

[0050] The second control unit 58 includes a processor such as a CPU, a storage device such as memory, and various peripheral circuits. In other words, the second control unit 58 functions as a computer. The second control unit 58 controls each part of the first instruction unit 50 by having the processor execute a program stored in the storage device. The second control unit 58 may also be configured to include multiple processors.

[0051] The second control unit 58 determines the timing for emitting light from the light-emitting unit 55 based on the control signal input from the receiving unit 53. The second control unit 58 emits light from the light-emitting unit 55 at the determined timing and outputs the first light 190.

[0052] [4. Projector Operation] Next, the operation of the projector 100 due to the processing performed by the first control unit 150 will be described. Figure 4 is a flowchart showing an example of the operation of the first control unit 150.

[0053] As shown in Figure 4, when processing begins, the first control unit 150 determines the presence or absence of an indicator in the projection area 20 corresponding to the display surface 10 based on the analysis results of the captured image generated by the imaging unit 135 (S1). Specifically, the first control unit 150 determines that an indicator is present if there is a light point of the first light 190 or a light point of the reflected light of the second light 180 detected by the analysis results of the captured image. If there is no indicator (S1: No), the first control unit 150 waits for processing to begin.

[0054] If an indicator is present (S1: Yes), the first control unit 150 determines whether the indicator is the first indicator 50, which is an indicator pen (S2). Specifically, if the first control unit 150 detects a light point of the first light 190 based on the analysis results of the captured image, it determines that the indicator is the first indicator 50. If it is not the first indicator 50 (S2: No), the first control unit 150 skips S3 and proceeds to S4.

[0055] If it is the first indicator 50 (S2: Yes), the first control unit 150 displays a first image on the display surface 10, which is a GUI image that includes multiple icon images that accept operation instructions from the first indicator 50 (S3).

[0056] Specifically, the first control unit 150 reads from the display image information 163 a plurality of icon images that accept operation instructions from the first indicator 50, along with information such as their display size. The first control unit 150 also reads from the display image information 163 a first image to be displayed, including the plurality of icon images, along with information such as its display size and the placement position of the icon images within the first image. Then, based on the information read from the display image information 163, the first control unit 150 generates drawing data for displaying the first image on the display surface 10. Next, the first control unit 150 outputs the generated drawing data to the image generation unit 110. Here, the first control unit 150 may specify the display position on the display surface 10 using the drawing data, such as near the position of the detected first indicator 50. For example, if the detected position of the first indicator 50 is not in contact with the display surface 10, the first control unit 150 generates drawing data to display the first image at a position on the display surface 10 corresponding to the position of the first indicator 50 in space away from the display surface 10 in the normal direction. Specifically, if the distance between the first indicator 50 and the display surface 10 is large in the Z-axis direction, the first control unit 150 generates drawing data to display the first image at a position on the display surface 10 corresponding to the position of the first indicator 50 considering only the X and Y axes. As a result, the projector 100 can facilitate user operation using the icon image of the first image, even when the first indicator 50 is not in contact with the display surface 10.

[0057] Figure 5 is an explanatory diagram showing an example of the first image 200. As shown in Figure 5, the first image 200 includes icon images 201 to 205 that accept operation instructions from the first indicator 50. Icon images 201 to 205 are examples of the first icon images.

[0058] For example, icon image 201 is an icon image used to instruct the deletion of the first image 200. Note that icon image 201 is the icon image with the smallest area among the icon images 201-205 included in the first image 200. In other words, icon image 201 is an example of the first sub-image.

[0059] Icon image 202 is used to specify tabs for different purposes, such as pens, highlighters, shapes, and erasers. In the example shown, icon image 202 is used to specify the pen.

[0060] Icon image 203 is an icon image used to specify various elements for the purpose specified in the tab. In the example shown, icon image 203 is used to specify the color and thickness of the pen tip.

[0061] Icon image 204 is an icon image used to instruct operations such as select, undo, redo, and delete. Icon image 205 is a representative image of the first image 200, indicating that the first image 200 is a GUI image that accepts operation instructions from the first indicator 50. This icon image 205 represents the drag position when the first image 200 is moved using operations such as drag and drop. For example, a user can move the first image 200 by dragging icon image 205 with the first indicator 50, moving it to a desired position on the display surface 10, and dropping it.

[0062] Following S3, the first control unit 150 determines whether the indicator is the user's finger, the second indicator 70 (S4). Specifically, if the first control unit 150 detects a point of reflected light from the second light 180 based on the analysis results of the captured image, it determines that it is the second indicator 70. If it is not the second indicator 70 (S4: No), the first control unit 150 skips S5 and proceeds to S6.

[0063] If it is the second indicator 70 (4:Yes), the first control unit 150 displays the second image, which is a GUI image including multiple icon images that accept operation instructions from the second indicator 70, on the display surface 10 (S5).

[0064] Specifically, the first control unit 150 reads from the display image information 163 a plurality of icon images that accept operation instructions from the second indicator 70, along with information such as their display size. The first control unit 150 also reads from the display image information 163 a second image that includes the plurality of icon images, along with information such as its display size and the placement position of the icon images within the second image. Then, based on the information read from the display image information 163, the first control unit 150 generates drawing data for displaying the second image on the display surface 10. Next, the first control unit 150 outputs the generated drawing data to the image generation unit 110. Here, the first control unit 150 may specify the display position on the display surface 10 using the drawing data, such as near the position of the detected second indicator 70. For example, if the detected position of the second indicator 70 is not in contact with the display surface 10, the first control unit 150 generates drawing data to display the second image at a position on the display surface 10 corresponding to the position of the second indicator 70 in space away from the display surface 10 in the normal direction. Specifically, if the distance between the second indicator 70 and the display surface 10 is large in the Z-axis direction, the first control unit 150 generates drawing data to display the second image at a position on the display surface 10 corresponding to the position of the second indicator 70 considering only the X and Y axes. This makes it easier for the user to operate the projector 100 using the icon image of the second image, even when the second indicator 70 is not in contact with the display surface 10.

[0065] Figure 6 is an explanatory diagram showing an example of the second image. As shown in Figure 6, the second image 300 includes icon images 301 to 305 that accept operation instructions from the second indicator 70. Icon images 301 to 305 are examples of the second icon images.

[0066] For example, icon image 301 is an icon image used to instruct the deletion of the second image 300. Note that icon image 301 is the icon image with the smallest area among the icon images 301-304 included in the second image 300. In other words, icon image 301 is an example of the second sub-image.

[0067] Icon image 302 is an icon image for instructing operations such as reducing, increasing, scaling to 1:1, and partial scaling. For example, the images in icon image 302 corresponding to reducing, increasing, and scaling to 1:1 support zoom operations on the image displayed on the display surface 10, such as reducing or increasing to an arbitrary magnification, or returning to the original 1:1 scale. Also, the image in icon image 302 corresponding to partial scaling supports zoom operations on the image displayed on the display surface 10, such as enlarging an arbitrary part of the image. Icon image 303 is an icon image for instructing operations such as selecting, undoing, redoing, and deleting. For example, the image in icon image 303 corresponding to selection supports the operation of rectangularly selecting an arbitrary range on the display surface 10 and selecting the objects contained within the selected rectangle. The projector 100 improves user usability because icon images 302 and 303, which are often used by the second indicator 70 acting as the user's finger for operations such as zooming and selecting objects, are included in the second image 300.

[0068] Icon image 304 is a representative image of the second image 300, indicating that the second image 300 is a GUI image that accepts operation instructions from the second indicator 70. This icon image 304 represents the drag position when the second image 300 is moved using operations such as drag and drop. For example, a user can move the second image 300 by dragging icon image 304 with the second indicator 70, moving it to a desired position on the display surface 10, and dropping it.

[0069] Here, we will compare and explain the first image 200 and the second image 300 displayed on the display surface 10, as well as the smallest icon image 201 in the first image 200 and the smallest icon image 301 in the second image 300. Figure 7 is an explanatory diagram for explaining the sizes of the smallest icon images included in the first image 200 and the second image 300.

[0070] As shown in Figure 7, the first image 200 is rectangular with width X1 and height Y1. The second image 300 is rectangular with width X2 and height Y2. When comparing the display sizes of the first image 200 and the second image 300, it is preferable that X2 > X1, Y2 > Y1, and X1·Y1 > X2·Y2. In this way, by displaying the second image 300, which receives operation instructions from the second indicator body 70 (the user's finger), larger than the first image 200, the operability of operation instructions from the user's finger can be improved.

[0071] In the illustrated example, the first image 200 and the second image 300 are shown as rectangular in shape, but the shapes of the first image 200 and the second image 300 are not limited to rectangular. For example, the shapes of the first image 200 and the second image 300 may be circular or triangular. Furthermore, the shapes of the first image 200 and the second image 300 may be different from each other, as long as the display size of the second image 300 is larger than that of the first image 200.

[0072] The smallest icon image 201 in the first image 200 is a rectangle with width x1 and height y1. The smallest icon image 301 in the second image 300 is a rectangle with width x2 and height y2. When comparing the display sizes of icon image 201 and icon image 301, x2 > x1, y2 > y1, and x1·y1 > x2·y2. Thus, the smallest icon image 201, which accepts operation instructions from the second indicator 70 (represented by the user's finger), has a larger display size than the smallest icon image 301 in the first image 200. By increasing the display size of icon image 201 in this way, it becomes easier for the user's finger (represented by the second indicator 70) to select icon image 201, thereby improving user operability.

[0073] Returning to Figure 4, the first control unit 150 determines whether the first image 200 and the second image 300 displayed on the display surface 10 are within a predetermined distance of each other on the display surface 10 (S6). Specifically, the first control unit 150 calculates the distance between the images based on the display coordinates of the first image 200 and the display coordinates of the second image 300 on the display surface 10. Then, the first control unit 150 determines whether the calculated distance is within a preset threshold. Here, the threshold used for the determination is set in advance by the user or the like.

[0074] If the first image 200 and the second image 300 are not within a predetermined distance (S6: No), the first control unit 150 skips S7 and proceeds to S8.

[0075] If the first image 200 and the second image 300 are within a predetermined distance (S6: Yes), the first control unit 150 displays a third image on the display surface 10 that includes a plurality of icon images that accept operation instructions from at least one of the first indicator 50 and the second indicator 70 (S7).

[0076] Specifically, the first control unit 150 reads from the display image information 163 a plurality of icon images that accept operation instructions from at least one of the first indicator 50 and the second indicator 70, along with information such as their display size. The first control unit 150 also reads from the display image information 163 a third image that includes the plurality of icon images, along with information such as its display size and the placement position of the icon images within the third image. Then, based on the information read from the display image information 163, the first control unit 150 generates drawing data for displaying the third image on the display surface 10. Then, the first control unit 150 outputs the generated drawing data to the image generation unit 110. Here, the first control unit 150 may specify the display position on the display surface 10 using the drawing data, such as the vicinity of the first image 200 and the second image 300, which are close together, for example, near an intermediate position between the first image 200 and the second image 300.

[0077] Furthermore, the first control unit 150 erases the first image 200 and the second image 300 that were previously displayed from the display surface 10 when the third image is displayed. As a result, on the display surface 10, it appears as if the first image 200 and the second image 300 have merged to form the third image.

[0078] Figure 8 is an explanatory diagram showing an example of the third image 400. As shown in Figure 8, the third image 400 includes icon images 401 to 407 that accept operation instructions from at least one of the first indicator 50 and the second indicator 70. Icon images 401 to 407 are examples of the third icon image. In this embodiment, all icon images 401 to 407 accept operation instructions from both the first indicator 50 and the second indicator 70. These icon images 401 to 407 may be divided into those that accept operation instructions from the first indicator 50, those that accept operation instructions from the second indicator 70, and those that accept operation instructions from both.

[0079] For example, icon image 401 is an icon image used to instruct the deletion of the third image 400. Note that icon image 401 is the icon image with the smallest area among the icon images 401-407 included in the third image 400. In other words, icon image 401 is an example of the third sub-image.

[0080] Icon image 402 is used to specify tabs for different purposes, such as pens and erasers. In the example shown, icon image 402 is used to specify the pen.

[0081] Icon image 403 is an icon image used to instruct actions such as shrinking and zooming. Icon image 405 is an icon image used to instruct actions such as selecting, undoing, redoing, and deleting.

[0082] Icon images 406 and 407 are representative images of the third image 400, indicating that the third image 400 is a GUI image that accepts operation instructions from at least one of the first indicator 50 and the second indicator 70. These icon images 406 and 407 represent the drag positions when moving the third image 400 using operations such as drag and drop, or when splitting the third image 400 and returning it to the original first image 200 and second image 300.

[0083] For example, a user can move the third image 400 by dragging at least one of the icon images 406 or 407 with the first indicator 50 or the second indicator 70, moving it to a desired position on the display surface 10, and dropping it. Alternatively, a user can drag icon image 407 with the first indicator 50 and icon image 406 with the second indicator 70, then separate the first indicator 50 and the second indicator 70. The user can then drop the dragged icon images 407 and 406 with the first indicator 50 and the second indicator 70, thereby splitting them back into the original first image 200 and second image 300.

[0084] Here, we will compare and explain the smallest icon image 204 in the third image 400 displayed on the display surface 10 with the smallest icon image 201 in the first image 200. The display size of icon image 201 is the same as in Figure 7.

[0085] As shown in the lower part of Figure 8, the smallest icon image 401 in the third image 400 is a rectangle with a width of x3 and a height of y3. When comparing the display sizes of icon image 201 and icon image 401, x3 > x1 and y3 > y1, and x3·y3 > x1·y1. Thus, the icon image 401, which has the smallest display size among the icon images that accept operation instructions from the user's finger (second indicator 70) or the pen (first indicator 50), has a larger display size than the smallest icon image 301 in the first image 200. In this way, by increasing the display size of icon image 401, it becomes easier for the user's finger (second indicator 70) to select icon image 401, thereby improving user operability.

[0086] Furthermore, the first control unit 150 may extract common operation instruction icon images from the icon images 201 to 204 of the first image 200 and the icon images 301 to 303 of the second image 300, and use these as the icon images when displaying the third image 400. The multiple icons 401 to 407 include icon images corresponding to at least one of the multiple icon images 201 to 205 and icon images corresponding to at least one of the multiple icon images 301 to 304.

[0087] Specifically, the first control unit 150 includes icon image 401 in the third image 400, since icon images 201 and 301 for the close operation are common operation instructions. Similarly, the first control unit 150 includes icon image 405 in the third image 400, since icon images 204 and 303 for operations such as select, cancel, redo, and delete are common operation instructions. As a result, the number of multiple icon images 401 to 405 is less than the sum of the number of icon images 201 to 204 and the number of icon images 301 to 303.

[0088] Furthermore, the first control unit 150 may determine which icon images 402 to 404 to include in the third image 400 from among the icon images 202 and 203 of the first image 200 and the icon image 302 of the second image 300, based on the history information 164. Specifically, the first control unit 150 prioritizes selecting those that have been operated on most recently or those that have been operated on many times, based on the history information 164.

[0089] For example, among the tabs for each function such as pen, highlighter, shape, and eraser in icon image 202, let's assume that the most recently operated or most frequently operated options are the pen and eraser. In this case, as shown in Figure 8, the first control unit 150 selects the pen and eraser from the pen, highlighter, shape, and eraser options and sets them as icon image 402. Furthermore, for the icon images used to instruct actions such as reducing, enlarging, 1:1 magnification, and partial enlargement, the first control unit 150 selects the most recently operated or most frequently operated options, namely reducing and enlarging, and sets them as icon image 403. Additionally, for the icon images used to instruct pen tip color and thickness, the first control unit 150 selects the three elements for each, in order of most recently operated or most frequently operated, and sets them as icon image 404.

[0090] Returning to Figure 4, the first control unit 150 determines whether or not a predetermined operation has been performed by the first indicator 50 or the second indicator 70 to move the first image 200, the second image 300, or the third image 400 (S8).

[0091] Here, in the predetermined operations for moving the first image 200, the second image 300, or the third image 400, the movement of the first image 200 is performed by dragging and dropping the icon image 205 that represents the first image 200. The movement of the second image 300 is performed by dragging and dropping the icon image 304 that represents the second image 300. The movement of the third image 400 is performed by dragging and dropping the icon images 406 and 407 that represent the third image 400. In addition, to split the third image 400 and return it to the original first image 200 and second image 300, the icon images 406 and 407 are dragged and then separated before being dropped. The projector 100 can fix the first image 200, the second image 300, or the third image 400 at any position on the display surface 10 by performing predetermined operations such as dragging and dropping on the icon images 205, 304, 406, and 407.

[0092] The above-described movement operations are merely examples, and other operations may also be performed. For example, the first control unit 150 may move the first image 200 in response to the movement of the first indicator 50 on the display surface 10. Similarly, the first control unit 150 may move the second image 300 in response to the movement of the second indicator 70 on the display surface 10. Furthermore, the first control unit 150 may use a flicking motion by the first indicator 50 or the second indicator 70 in a predetermined direction near the first image 200, the second image 300, or the third image 400 as the operation for moving each image. In addition, the first control unit 150 may use a flicking motion that spreads the third image 400 outward from the center as the operation to divide 400 and return it to the original first image 200 and second image 300.

[0093] If there is no predetermined operation for moving the first image 200, the second image 300, or the third image 400 (S9: No), the first control unit 150 skips S9 and returns to the previous step. If there is a predetermined operation for moving the first image 200, the second image 300, or the third image 400 (S9: Yes), the first control unit 150 moves the target image on the display surface 10 (S9) and returns to the previous step.

[0094] Figure 9 is an explanatory diagram illustrating the movement of the second image 300 on the display surface 10. As shown in Figure 9, the user drags the icon image 301, which is a representative of the second image 300, using the second indicator 70, which is the user's finger. Then, the user moves the second image 300 by moving the second indicator 70 to the desired position on the display surface 10 and dropping it.

[0095] Furthermore, the first control unit 150 may configure the composite display of the third image 400 when the first image 200 and the second image 300 are close together, by compositing them in stages according to the distance.

[0096] Figure 10 is an explanatory diagram illustrating the display of the third image 400, which is created by combining the first image 200 and the second image 300. In the example in Figure 10, the process of the second image 300 approaching the first image 200 and the display of the third image 400 is shown stepwise from top to bottom. Conversely, the process of separating the third image 400 into the first image 200 and the second image 300 is shown from bottom to top.

[0097] As shown in Figure 10, when the first image 200 and the second image 300 approach a predetermined distance from each other on the display surface 10, the first control unit 150 displays, for example, a third image 400 at an intermediate position between the first image 200 and the second image 300. At this time, the first control unit 150 displays the third image 400 as a temporary display and does not accept operations using icon images 401 to 407. In the illustrated example, the temporary display is represented by a dotted line. That is, while the third image 400 is a temporary display, the first control unit 150 accepts operations on the first image 200 or the second image 300.

[0098] Next, as the first image 200 and the second image 300 move closer together, the first control unit 150 accepts operations using icon images 401 to 407 of the third image 400. At this time, the first control unit 150 displays the first image 200 and the second image 300 as temporary displays and does not accept operations on the first image 200 or the second image 300.

[0099] Next, when the first image 200 and the second image 300 are closer together, the first control unit 150 erases the first image 200 and the second image 300, which are being displayed temporarily, and displays only the third image 400 on the display surface 10.

[0100] By displaying the composite process in this manner, users can easily recognize that the first image 200 and the second image 300 are combined to display the third image 400. Furthermore, users can easily recognize that the first image 200 and the second image 300 disappear when the third image 400 is displayed.

[0101] Conversely, let's explain the case where the first image 200 and the second image 300 are displayed separately from the third image 400. First, let's assume that in the third image 400, the second image 300 is separated from the third image 400 by dragging the icon image 406 to the lower right relative to the icon image 407.

[0102] Based on this separation operation, the first control unit 150 displays the first image 200 and the second image 300. At this time, the first control unit 150 displays the first image 200 and the second image 300 as temporary displays and does not accept operations on the first image 200 or the second image 300. In other words, while the first image 200 and the second image 300 are temporary displays, the first control unit 150 accepts operations on the third image 400.

[0103] Next, if the separation operation continues, the first control unit 150 will accept operations on the first image 200 and the second image 300. At this time, the first control unit 150 will display the third image 400 as a temporary display and will not accept operations on the third image 400.

[0104] Next, the first control unit 150 erases the temporary third image 400 when the first image 200 and the second image 300 are further apart, and displays only the first image 200 and the second image 300 on the display surface 10.

[0105] By displaying the separation of this process, users can easily recognize that the first image 200 and the second image 300 are separated from the third image 400. Furthermore, users can easily recognize that the third image 400 disappears when the first image 200 and the second image 300 are separated.

[0106] [5. Effects of the Embodiment] As described above, the projector 100 includes a first control unit 150 which includes a processor 165. The first control unit 150 determines whether a first indicator that indicates a position on the display surface 10 is an indicator pen. If the first control unit 150 determines that the first indicator is an indicator pen, it displays a first image 200 on the display surface 10 which includes icon images 201 to 205 that accept operation instructions from the indicator pen. The first control unit 150 determines whether a second indicator that indicates a position on the display surface is a user's finger. If the first control unit 150 determines that the second indicator is a user's finger, it displays a second image 300 on the display surface 10 which includes icon images 301 to 304 that accept operation instructions from the user's finger. The second small image, which is one of the icon images 301 to 304 and has the smallest display size, has a larger display size than the first small image, which is one of the icon images 201 to 205 and has the smallest display size.

[0107] Therefore, the projector 100 can display a smaller icon image 301 that accepts operation instructions from the second finger 70 of the user, larger than the smallest icon image 201 that accepts operation instructions from the first finger 50 of the pen. As a result, the projector 100 can improve the operability of the user's finger. For example, the projector 100 can suppress errors when the user operates with their finger. In addition, in the projection system 1 using the projector 100, the detection accuracy of the indicator may deteriorate when the calibration accuracy is low, and errors in selection tend to occur when the user operates with their finger. Even when the detection accuracy of the indicator deteriorates in this way, the projector 100 can prevent the deterioration of the operability of the user's finger.

[0108] Furthermore, the first control unit 150 displays a third image 400 if the position of the first image 200 and the position of the second image 300 on the display surface 10 are within a predetermined distance. This third image 400 includes icon images 401 to 407 that accept operation instructions from the first indicator 50 and the second indicator 70. Icon images 401 to 407 include an icon image corresponding to at least one of the icon images 201 to 205 and an icon image corresponding to at least one of the icon images 301 to 304.

[0109] By displaying this third image 400, the projector 100 only requires one GUI for providing operation instructions to at least one of the first indicator 50 and the second indicator 70. Therefore, compared to the case where the first image 200 and the second image 300 are displayed on the display surface 10, the projector 100 can simplify the content displayed on the display surface 10 and make it easier for the user to see.

[0110] Furthermore, displaying the third image 400 involves combining the first image 200 and the second image 300 to display the third image 400.

[0111] Therefore, in the projector 100, the projector 100 and the first image 200 are combined to form a single GUI third image 400, which allows for a simplified display on the display surface 10.

[0112] Furthermore, the third smallest icon image, which is one of the icon images 401-407 and has the smallest display size, has a larger display size than the first smallest icon image, which is one of the icon images 201-205 and has the smallest display size.

[0113] Therefore, the projector 100 can display the smallest icon image 401 in the third image 400 at a larger size than the smallest icon image 201 that accepts operation instructions from the first indicator 50 of the pen. As a result, the projector 100 can improve the operability of the icon images in the third image 400 with the user's finger.

[0114] Furthermore, the number of icon images 401-407 is less than the sum of the number of icon images 201-205 and the number of icon images 301-304.

[0115] In this way, by reducing the number of icon images 401 to 407, the projector 100 can simplify the display content within the third image 400.

[0116] Additionally, icon images 401-407 include icon images that correspond to common operation instructions found in icon images 201-205 and icon images 301-304.

[0117] In this way, by including icon images corresponding to common operation instructions in icon images 401 to 407, the projector 100 can efficiently reduce the number of icon images 401 to 407 within the third image 400.

[0118] Furthermore, the first control unit 150 further performs the operation instruction history information 164 to determine which icon images to include in icon images 401 to 407 from among icon images 201 to 205 and icon images 301 to 304.

[0119] Therefore, the projector 100 can use icon images 401 to 407 that correspond to the operation history. For example, the projector 100 can include the icon image used immediately before in icon images 401 to 407. This saves the user the trouble of searching for the desired icon image when using the same operation repeatedly.

[0120] Furthermore, the first control unit 150 further performs the action of displaying a representative image corresponding to the first image 200, the second image 300, or the third image 400 on the display surface 10. For example, the first control unit 150 displays an icon image 205 as a representative image for the first image 200. The first control unit 150 also displays an icon image 304 as a representative image for the second image 300. The first control unit 150 also displays icon images 406 and 407 as representative images for the third image 400. Based on a predetermined operation by at least one of the first indicator 50 and the second indicator 70 on this representative image, the first control unit 150 further performs the action of moving the first image 200, the second image 300, or the third image 400 to a predetermined position on the display surface 10.

[0121] Therefore, the user can move the first image 200, the second image 300, or the third image 400 to a desired position on the display surface 10 by performing a predetermined operation on the representative image.

[0122] [6. Other Embodiments] The embodiments described above are preferred embodiments of the present invention. However, the invention is not limited to these embodiments, and various modifications can be made without departing from the spirit of the invention.

[0123] For example, in the embodiment described above, the projector 100 was configured to include an illumination unit 133 and an imaging unit 135. However, at least one of the illumination unit 133 and the imaging unit 135 may be provided outside the projector 100 as a separate light-emitting device. Also, some of the functions of the first control unit 150 may be mounted on a separate device provided outside the projector 100. When these components are provided outside the projector 100, they may be independent devices, or they may be a device that includes two or more of these components.

[0124] Furthermore, although the above-described embodiment explained the projector 100 as a liquid crystal projector using a transmissive liquid crystal panel, it may also be a projector using a reflective liquid crystal panel or a digital mirror device.

[0125] Furthermore, the determination of the positions of the first indicator 50 and the second indicator 70 on the display surface 10 is not limited to methods such as detecting light emission from the tip 51 of the first indicator 50 or the reflection of the second indicator 70 to detection light emitted along the display surface 10. For example, the projector 100 is equipped with a stereo camera that images a projection area 20 corresponding to the display surface 10. The projector 100 determines the positions of the first indicator 50 and the second indicator 70 on the display surface 10 based on a plurality of stereo images captured by this stereo camera. More specifically, the projector 100 identifies the first indicator 50 and the second indicator 70 included in the plurality of stereo images using known image recognition technology. Then, the projector 100 determines the positions of the tip 51 of the identified first indicator 50 and the fingertip of the second indicator 70 based on the difference between the plurality of stereo images. In this case, the first indicator 50, which functions as an indicator pen, does not need to emit light at its tip 51; for example, the tip 51 may be a pointed rod or the like.

[0126] Furthermore, although projection system 1 was given as an example of a display system in the embodiments described above, the display system is not limited to projection system 1. For example, the display system of this disclosure may be applied to a display device equipped with a stylus as an example of an instruction pen and a touch panel capable of detecting the touch position of a user's finger. Examples of such display devices include electronic whiteboard devices used in meetings or presentations, and tablet terminals.

[0127] Furthermore, the movement of the first image 200 in response to an operation by the first indicator 50 and the second image 300 in response to an operation by the second indicator 70 is not limited to operations on the representative image described above. For example, the projector 100 may move the first image 200 and the second image 300 based on the positions of the first indicator 50 and the second indicator 70 detected on the display surface 10.

[0128] For example, the projector 100 may move the display position of the first image 200 in accordance with the position of the first indicator 50 detected on the display surface 10. Similarly, the projector 100 may move the display position of the second image 300 in accordance with the position of the second indicator 70 detected on the display surface 10. As an example, if the position of the first indicator 50 is in the upper left area of ​​the display surface 10, the projector 100 may move the first image 200 into that area. Similarly, if the position of the second indicator 70 is in the lower right area of ​​the display surface 10, the projector 100 may move the second image 300 into that area. This allows the projector 100 to easily perform operations using the first image 200 or the second image 300 without the user having to move the first image 200 or the second image 300 themselves at the position of the first indicator 50 or the second indicator 70.

[0129] Furthermore, each functional unit of the first indicator 50 shown in Figure 3 represents a functional configuration realized through the cooperation of hardware and software, and the specific implementation form is not particularly limited. Therefore, it is not necessarily required that hardware corresponding to each functional unit be implemented individually, and it is certainly possible to have a configuration in which a single processor executes a program to realize the functions of multiple functional units. In addition, some of the functions realized by software in the above embodiment may be realized by hardware, and some of the functions realized by hardware may be realized by software.

[0130] Furthermore, for example, in the configuration of the first indicator 50 shown in Figure 3, at least a portion of the receiving unit 53, the second control unit 58, the power supply unit 57, and the tip switch 54 may be composed of integrated circuits or other digital circuits. Also, in the configuration of the first indicator 50, at least a portion of each part may include analog circuits. Integrated circuits include LSIs, ASICs, and PLDs. PLDs include, for example, FPGAs. Each of the above parts may be a combination of a processor and an integrated circuit. Such combinations are called, for example, MCUs, SoCs, system LSIs, chipsets, etc.

[0131] Furthermore, in the configuration of the projector 100 shown in Figure 3, at least a portion of the transmitting unit 131, the illuminating unit 133, and the first control unit 150 may be made up of integrated circuits (ICs) or other digital circuits, and at least a portion of each part may include analog circuits. Alternatively, each of the above parts may be made up of a combination of a processor and an integrated circuit.

[0132] Furthermore, the application program 161 executed by the processor 165 to realize the above-described display method may be provided on a recording medium readable by the processor 165. The recording medium readable by the processor 165 may be an optical recording medium such as a DVD, a USB memory, a semiconductor memory device such as an SSD, etc. Also, the application program 161 may be provided or distributed in the form of a transmission medium, stored on a computer connected to a network such as the Internet, and provided or distributed by downloading it over the network.

[0133] Furthermore, the processing units in the flowchart shown in Figure 4 are divided according to their main processing content in order to facilitate understanding of the processing of the first control unit 150, and the present invention is not limited by the way the processing units are divided or named as shown in the flowchart of Figure 4. In addition, the processing of the first control unit 150 can be further divided into more processing units depending on the processing content, or it can be divided so that one processing unit includes even more processing. Moreover, the processing order in the flowchart above is not limited to the example shown.

[0134] [7. Summary of this disclosure] A summary of this disclosure is provided below.

[0135] (Note 1) A display method comprising: a processor determining whether a first indicator that indicates a position on a display surface is an indicator pen; if the first indicator is determined to be the indicator pen, displaying a first image on the display surface that includes a plurality of first icon images that accept operation instructions from the indicator pen; determining whether a second indicator that indicates a position on the display surface is a user's finger; and if the second indicator is determined to be the user's finger, displaying a second image on the display surface that includes a plurality of second icon images that accept operation instructions from the user's finger, wherein the second small image, which is one of the plurality of second icon images and has the smallest display size, has a larger display size than the first small image, which is one of the plurality of first icon images and has the smallest display size.

[0136] This allows the smallest first icon image among multiple first icon images that accept operation instructions from a pointer pen to be displayed at a larger size than the smallest second icon image among multiple second icon images that accept operation instructions from the user's finger, thereby improving the user's ability to operate with their finger.

[0137] (Note 2) The display method according to Appendix 1, wherein, when the position of the first image on the display surface and the position of the second image are within a predetermined distance, the processor further displays a third image including a plurality of third icon images that accept operation instructions from the pointer pen and the user's finger, the plurality of third icon images including an icon image corresponding to at least one of the plurality of first icon images and an icon image corresponding to at least one of the plurality of second icon images.

[0138] This allows for a single image to be used to receive input from both the pointer pen and the user's finger. Consequently, the content displayed on the screen is simplified and easier for the user to see.

[0139] (Note 3) Displaying the third image is the method of displaying the third image by combining the first image and the second image, as described in Appendix 2.

[0140] This results in a single third image created by combining the first and second images. Consequently, the content displayed on the screen is simplified, making it easier for the user to view.

[0141] (Note 4) The display method according to Appendix 2 or 3, wherein the third small image, which is one of the plurality of third icon images and has the smallest display size, has a larger display size than the first small image, which is one of the plurality of first icon images and has the smallest display size.

[0142] This improves the user's ability to interact with multiple third-party icon images using their fingers.

[0143] (Note 5) The display method described in any one of the appendices 2 to 4, wherein the number of the plurality of third icon images is less than the sum of the number of the plurality of first icon images and the number of the plurality of second icon images.

[0144] This reduces the number of third-party icon images, resulting in a simpler and more user-friendly display for the third-party images.

[0145] (Note 6) The display method according to any one of Appendix 2 to 5, wherein the plurality of third icon images include icon images corresponding to common operation instructions in the plurality of first icon images and the plurality of second icon images.

[0146] This allows for an efficient reduction in the number of third-party icon images required.

[0147] (Note 7) The display method according to any one of the appendices 2 to 6, wherein the processor further determines which icon images to include in the plurality of third icon images from among the plurality of first icon images and the plurality of second icon images based on the history information of operation instructions.

[0148] This means that the icon images included in the multiple third icon images, selected from multiple first icon images and multiple second icon images, will correspond to the history of operation instructions, such as those used for the same operation consecutively. Therefore, users are saved the trouble of searching for frequently used icon images, such as those used for the same operation consecutively.

[0149] (Note 8) The display method according to any one of Appendix 2 to 7, wherein the processor further performs the following actions: displaying a representative image corresponding to the first image, the second image, or the third image on the display surface; and moving the first image, the second image, or the third image to a predetermined position on the display surface based on a predetermined action performed on the representative image by at least one of the pointing pen and the user's finger.

[0150] This allows the user to move the first image, second image, or third image to a desired position on the display surface by performing a predetermined operation on the representative image.

[0151] (Note 9) A display system comprising a processor that performs the following: determining whether a first indicator that indicates a position on a display surface is an indicator pen; if the first indicator is determined to be the indicator pen, displaying a first image on the display surface that includes a plurality of first icon images that accept operation instructions from the indicator pen; determining whether a second indicator that indicates a position on the display surface is a user's finger; and if the second indicator is determined to be the user's finger, displaying a second image on the display surface that includes a plurality of second icon images that accept operation instructions from the user's finger, wherein the second small image, which is one of the plurality of second icon images and has the smallest display size, has a larger display size than the first small image, which is one of the plurality of first icon images and has the smallest display size.

[0152] This allows the smallest first icon image among multiple first icon images that accept operation instructions from a pointer pen to be displayed at a larger size than the smallest second icon image among multiple second icon images that accept operation instructions from the user's finger, thereby improving the user's ability to operate with their finger.

[0153] (Note 10) A display program that causes a processor to perform the following actions: determine whether a first indicator that indicates a position on a display surface is an indicator pen; if the first indicator is determined to be the indicator pen, display a first image on the display surface that includes a plurality of first icon images that accept operation instructions from the indicator pen; determine whether a second indicator that indicates a position on the display surface is a user's finger; if the second indicator is determined to be the user's finger, display a second image on the display surface that includes a plurality of second icon images that accept operation instructions from the user's finger; wherein the second small image, which is one of the plurality of second icon images and has the smallest display size, has a larger display size than the first small image, which is one of the plurality of first icon images and has the smallest display size.

[0154] This allows the smallest first icon image among multiple first icon images that accept operation instructions from a pointer pen to be displayed at a larger size than the smallest second icon image among multiple second icon images that accept operation instructions from the user's finger, thereby improving the user's ability to operate with their finger. [Explanation of symbols]

[0155] 1…Projection system, 5…Remote control, 10…Display surface, 20…Projection area, 50…First indicator, 51…Tip, 52…Shaft, 53…Receiver, 54…Tip switch, 55…Light-emitting unit, 57…Power supply unit, 58…Second control unit, 70…Second indicator, 100…Projector, 110…Image generation unit, 115…Image storage unit, 120…Projection unit, 121…Light source, 123…Optical modulation device, 125…Optical unit, 131…Transmitter, 133…Irradiation unit, 135…Imaging unit, 140…Light receiving unit, 150…First control unit, 160…Storage unit, 161…Application program, 163…Display image information, 164…History information, 165…Processor, 170 ...signal light, 180...second light, 190...first light, 200...first image, 201...icon image, 202...icon image, 203...icon image, 204...icon image, 205...icon image, 300...second image, 301...icon image, 302...icon image, 303...icon image, 304...icon image, 400...third image, 401...icon image, 402...icon image, 403...icon image, 404...icon image, 405...icon image, 406...icon image, 407...icon image, x1...width, x2...width, x3...width, X1...width, X2...width, y1...height, y2...height, y3...height, Y1...height, Y2...height.

Claims

1. To determine whether the first indicator that indicates the position on the display surface is an indicator pen, When the first indicator is determined to be the indicator pen, a first image including a plurality of first icon images that accept operation instructions from the indicator pen is displayed on the display surface. The second indicator that indicates the position on the display surface is determined to be the user's finger or not, When the second indicator is determined to be the user's finger, a second image including a plurality of second icon images that accept operation instructions from the user's finger is displayed on the display surface. The processor executes this, The second small image, which is one of the plurality of second icon images and has the smallest display size, has a larger display size than the first small image, which is one of the plurality of first icon images and has the smallest display size. Display method.

2. If the position of the first image and the position of the second image on the display surface are within a predetermined distance, the processor further performs the operation of displaying a third image which includes a plurality of third icon images that accept operation instructions from the pointer pen and the user's finger. The plurality of third icon images include an icon image corresponding to at least one of the plurality of first icon images and an icon image corresponding to at least one of the plurality of second icon images. The display method according to claim 1.

3. Displaying the third image means combining the first image and the second image to display the third image. The display method according to claim 2.

4. The third small image, which is one of the plurality of third icon images and has the smallest display size, has a larger display size than the first small image, which is one of the plurality of first icon images and has the smallest display size. The display method according to claim 2 or 3.

5. The number of the aforementioned plurality of third icon images is less than the sum of the number of the aforementioned plurality of first icon images and the number of the aforementioned plurality of second icon images. The display method according to claim 2 or 3.

6. The plurality of third icon images include icon images that correspond to common operation instructions in the plurality of first icon images and the plurality of second icon images. The display method according to claim 5.

7. The processor further performs the operation of determining which icon images to include in the plurality of third icon images from among the plurality of first icon images and the plurality of second icon images, based on the history information of operation instructions. The display method according to claim 2 or 3.

8. A representative image corresponding to the first image, the second image, or the third image is displayed on the display surface. The processor further performs the following actions based on a predetermined action performed on the representative image by at least one of the pointing pen and the user's finger: moving the first image, the second image, or the third image to a predetermined position on the display surface. The display method according to claim 2 or 3.

9. To determine whether the first indicator that indicates the position on the display surface is an indicator pen, When the first indicator is determined to be the indicator pen, a first image including a plurality of first icon images that accept operation instructions from the indicator pen is displayed on the display surface. The second indicator that indicates the position on the display surface is determined to be the user's finger or not, When the second indicator is determined to be the user's finger, a second image including a plurality of second icon images that accept operation instructions from the user's finger is displayed on the display surface. Includes a processor that runs The second small image, which is one of the plurality of second icon images and has the smallest display size, has a larger display size than the first small image, which is one of the plurality of first icon images and has the smallest display size. Display system.

10. To determine whether the first indicator that indicates the position on the display surface is an indicator pen, When the first indicator is determined to be the indicator pen, a first image including a plurality of first icon images that accept operation instructions from the indicator pen is displayed on the display surface. The second indicator that indicates the position on the display surface is determined to be the user's finger or not, When the second indicator is determined to be the user's finger, a second image including a plurality of second icon images that accept operation instructions from the user's finger is displayed on the display surface. Let the processor execute it, The second small image, which is one of the plurality of second icon images and has the smallest display size, has a larger display size than the first small image, which is one of the plurality of first icon images and has the smallest display size. Display program.

Images