Display method and display system

Abstract

To provide a technique that can display an image from which a user can easily imagine an image projected in a room or the like.SOLUTION: A display method includes: acquiring, in a real space having an object area including a surface and a display, an object image indicating the object area; and in a virtual space having a virtual surface located at a first position corresponding to the position of the surface in the real space and corresponding to the surface and a virtual projector, when displaying, on the display, a first display image being an image projected on the virtual surface from the virtual projector seen from a second position corresponding to the position of the display in the real space, displaying, on the display, a first simulation image obtained by superimposing the first display image in a size within a display on the object image regardless of the position of the display.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present invention relates to a display method and a display system. [Background technology]

[0002] Patent Document 1 discloses an information processing device that displays a simulation image of how a projected image projected from a projector will look. The simulation image shows a room in which the projector is located and a screen on which the projected image will be displayed. When a value that causes the screen to extend beyond the room is input, the information processing device displays a simulation image showing the screen extending beyond the room. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 179272 Summary of the Invention [Problem to be solved by the invention]

[0004] When the information processing device described in Patent Document 1 displays a simulation image showing a screen that extends beyond the room, the projected image in the simulation image may also extend beyond the room. A situation in which a projected image extends beyond the room does not actually occur. For this reason, even when a user looks at the simulation image, it may be difficult to imagine the projected image projected in an actual room or the like. In this case, the user needs to re-enter values, and the convenience of the information processing device described in Patent Document 1 is low. [Means for solving the problem]

[0005] One aspect of a display method according to the present invention includes obtaining a target image indicating a target area in a real space having a target area including a surface and a display; and in a virtual space having a virtual surface located at a first position corresponding to a position of the surface in the real space and corresponding to the surface, and a virtual projector, when displaying a first display image on the display, the first display image being an image projected onto the virtual surface from the virtual projector as viewed from a second position corresponding to the position of the display in the real space, displaying a first simulation image on the display in which the first display image of a size that fits on the display is superimposed on the target image, regardless of the position of the display.

[0006] One aspect of the display system of the present invention includes a camera, a display, and at least one processing unit, wherein the at least one processing unit acquires, using the camera, a target image indicating a target area in a real space having a target area including a surface and the display; and when displaying a first display image on the display in a virtual space located at a first position corresponding to a position of the surface in the real space and having a virtual surface corresponding to the surface and a virtual projector, an image projected onto the virtual surface from the virtual projector is an image viewed from a second position corresponding to the position of the display in the real space, the first simulation image being superimposed on the target image by the first display image of a size that fits on the display regardless of the position of the display. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an information processing device 1. [Diagram 2] 2 is a diagram showing a front surface 1a of the information processing device 1. FIG. [Diagram 3] 2 is a diagram showing a rear surface 1b of the information processing device 1. FIG. [Figure 4] FIG. 2 is a diagram showing an example of a target image H1. [Diagram 5]FIG. 1 illustrates an example of an information processing device 1. [Figure 6] FIG. 2 is a diagram showing an example of a virtual space VS. [Figure 7] FIG. 13 is a diagram showing an example of a second composite image Y2. [Figure 8] FIG. 11 is a diagram for explaining an example of size a1. [Figure 9] 13 is a diagram showing an example of adjustment of a distance Dp between a virtual surface C3 and a virtual projector C4. FIG. [Figure 10] 13 is a flowchart for explaining recognition of a wall E1. [Figure 11] 11 is a flowchart for explaining the display of a first composite image Y1 and a second composite image Y2. [Figure 12] FIG. 13 is a diagram showing a state before a virtual projector C4 is brought closer to a virtual surface C3. [Figure 13] FIG. 13 shows the situation after a virtual projector C4 is moved closer to a virtual surface C3. [Figure 14] 1 is a diagram showing an example of an icon i displayed on the touch panel 12. FIG. [Figure 15] FIG. 13 is a diagram showing an example of a first guide image t1. [Figure 16] FIG. 1 is a diagram showing an example of an information processing device 1 that displays a target image H1. [Figure 17] FIG. 13 is a diagram showing an example of an image u1. [Figure 18] FIG. 13 is a diagram showing an example of an image u2. [Figure 19] A figure showing an example of an image u3. [Figure 20] FIG. 1 is a diagram showing a situation in which information processing device 1 is shaken by a user. [Figure 21] FIG. 13 is a diagram showing an example of a second composite image Y2. [Figure 22] FIG. 13 is a diagram showing another example of the second composite image Y2. [Figure 23] FIG. 13 is a diagram showing yet another example of the second composite image Y2. [Figure 24] FIG. 13 is a diagram showing yet another example of the second composite image Y2. [Diagram 25]FIG. 13 is a diagram showing yet another example of the second composite image Y2. [Figure 26] A figure showing an example of image u5. [Figure 27] A figure showing another example of image u5. [Figure 28] A figure showing yet another example of image u5. [Figure 29] A figure showing yet another example of image u5. [Diagram 30] A figure showing yet another example of image u5. [Diagram 31] FIG. 13 is a diagram showing an example of a menu image v4. [Diagram 32] FIG. 13 is a diagram showing an example of an image candidate v8. [Diagram 33] FIG. 13 is a diagram showing an example of a second composite image Y2. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] A: First embodiment A1: Overview of information processing device 1 FIG. 1 is a diagram showing an information processing device 1. The information processing device 1 is a smartphone. The information processing device 1 is not limited to a smartphone, and may be, for example, a tablet with a camera, a notebook PC (Personal Computer) with a camera, or a notebook PC connected to a camera. The information processing device 1 is an example of a display system. The information processing device 1 is located in a real space RS.

[0009] The real space RS includes a projector 2, a wall E1, a ceiling E2, and a floor E3 in addition to the information processing device 1. The position of the projector 2 in the real space RS is not limited to the position shown in FIG.

[0010] The wall E1 is a vertical surface. The wall E1 is not limited to a vertical surface, but may be any surface that intersects with a horizontal plane. The wall E1 is an inner wall of a building. The wall E1 is not limited to an inner wall of a building, but may be, for example, an outer wall of a building. At least a portion of the wall E1 is an example of a surface. The surface is not limited to at least a portion of the wall E1, but may be, for example, at least a portion of the ceiling E2, at least a portion of the floor E3, a screen, a whiteboard, or a door. The surface is included in the target region TR.

[0011] The target region TR is included in the real space RS. The position of the target region TR in the real space RS is not limited to the position shown in FIG.

[0012] The projector 2 projects a projection image F1 onto a wall E1 using light. The information processing device 1 displays a first composite image Y1 relating to how the projection image F1 appears. The first composite image Y1 is an example of a first simulation image.

[0013] The information processing device 1 includes a front surface 1a, a rear surface 1b, a camera 11, and a touch panel 12. Fig. 2 is a diagram showing the front surface 1a of the information processing device 1. Fig. 3 is a diagram showing the rear surface 1b of the information processing device 1.

[0014] The camera 11 is located on the rear surface 1b of the information processing device 1. The camera 11 captures an image of an imaging area. The imaging area of ​​the camera 11 moves in accordance with the movement of the information processing device 1.

[0015] The imaging area of ​​the camera 11 is used as the target area TR. Therefore, the target area TR moves in accordance with the movement of the information processing device 1. The camera 11 captures the target area TR in a situation where the projector 2 is not projecting the projection image F1, thereby generating a target image H1 indicating the target area TR. The target image H1 indicating the target area TR means an image indicating an object existing in the target area TR.

[0016] 4 is a diagram showing an example of a target image H1. The target image H1 shows a wall E1, a ceiling E2, and a floor E3.

[0017] 2, the touch panel 12 is located on the front surface 1a of the information processing device 1. The touch panel 12 is an example of a display. The touch panel 12 displays a first composite image Y1.

[0018] The first composite image Y1 is an image obtained by superimposing a sample image J1, which is sized to fit on the touch panel 12, on the target image H1. The sample image J1 represents, for example, the projected image F1. The sample image J1 may be an image different from the projected image F1, for example, an image obtained by changing the color of the projected image F1 to a single color. The sample image J1 has a preset transmittance. The transmittance of the sample image J1 may be changeable.

[0019] The first composite image Y1 includes a projector image L1. The projector image L1 is an image showing a projector. The shape of the projector shown by the projector image L1 is the same as the shape of the projector 2. The shape of the projector shown by the projector image L1 may be different from the shape of the projector 2. The projector image L1 has a preset transmittance. The transmittance of the projector image L1 may be changeable.

[0020] The first composite image Y1 further includes a path image L2. The path image L2 is an image showing a path of light used when the projector 2 projects the projection image F1. The path image L2 is also an image showing a path of light virtually used when the virtual projector C4 projects an image. The virtual projector C4 is a virtual projector corresponding to the projector 2. The path image L2 has a preset transmittance. The transmittance of the path image L2 may be changeable.

[0021] The first composite image Y1 does not have to include at least one of the projector image L1 and the path image L2.

[0022] A2: An example of the information processing device 1 5 is a diagram showing an example of the information processing device 1. The information processing device 1 includes a camera 11, a touch panel 12, a motion sensor 13, a storage device 14, and a processing device 15.

[0023] The camera 11 includes an imaging lens 111 and an image sensor 112 .

[0024] The imaging lens 111 forms an optical image on the image sensor 112. The imaging lens 111 forms on the image sensor 112 a target image H1 representing a target region TR.

[0025] The image sensor 112 is a CCD (Charge Coupled Device) image sensor. The image sensor 112 is not limited to a CCD image sensor, and may be, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The image sensor 112 generates imaging data k based on an optical image formed on the image sensor 112. For example, the image sensor 112 generates imaging data kt indicating the target image H1 based on the target image H1 formed by the imaging lens 111. The imaging data kt is an example of the imaging data k.

[0026] The touch panel 12 includes a display 121 and an input device 122. The display 121 displays various images. The input device 122 receives various instructions and the like.

[0027] The motion sensor 13 includes an acceleration sensor and a gyro sensor. The motion sensor 13 detects the motion of the information processing device 1. For example, the motion sensor 13 detects the motion of the information processing device 1 moved by a user. The motion of the information processing device 1 is represented by at least the moving distance of the information processing device 1, the rotation amount of the information processing device 1, and the orientation of the information processing device 1. The motion sensor 13 generates motion data m indicating the motion of the information processing device 1.

[0028] The storage device 14 is a recording medium readable by the processing device 15. The storage device 14 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), and an EEPROM (Electrically Erasable Programmable Read Only Memory). The volatile memory is, for example, a RAM (Random Access Memory). The storage device 14 stores a program P1 and various data. The program P1 is, for example, an application program. The program P1 is provided to the information processing device 1 from a server (not shown). The program P1 may be stored in advance in the storage device 14.

[0029] The processing device 15 is configured with one or more central processing units (CPUs). The one or more CPUs are an example of one or more processors. The processor is an example of a processing unit. Each of the CPU and the processor is an example of a computer.

[0030] The processing device 15 reads the program P1 from the storage device 14. The processing device 15 functions as an acquisition unit 151, a recognition unit 152, and an operation control unit 153 by executing the program P1.

[0031] The processing device 15 may function as the acquisition unit 151 and the operation control unit 153 by executing the program P1, and function as the recognition unit 152 by executing a program different from the program P1. In this case, the program different from the program P1 is stored in the storage device 14, and the processing device 15 reads the program different from the program P1 from the storage device 14.

[0032] Each of the acquisition unit 151, the recognition unit 152, and the operation control unit 153 may be realized by a circuit such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array).

[0033] The acquisition unit 151 acquires a target image H1 indicating a target region TR. For example, the acquisition unit 151 acquires the target image H1 by acquiring imaging data kt indicating the target image H1 from the camera 11. The acquisition unit 151 also acquires motion data m indicating the motion of the information processing device 1 from the motion sensor 13.

[0034] The recognition unit 152 acquires the imaging data kt and the motion data m from the acquisition unit 151. The recognition unit 152 performs three-dimensional measurement of an object present in the target region TR based on the imaging data kt and the motion data m. The recognition unit 152 determines the position of the camera 11 in the real space RS based on the motion data m. The recognition unit 152 performs three-dimensional measurement by using the position of the camera 11 in the real space RS and the imaging data kt.

[0035] In a situation where the information processing device 1 is moved from a first point to a second point while the camera 11 is capturing an image of the wall E1, the recognition unit 152 executes three-dimensional measurement as follows.

[0036] The recognition unit 152 acquires the motion data ms from the acquisition unit 151. The motion data ms is motion data m generated by the motion sensor 13 in a situation where the information processing device 1 is moved from a first point to a second point while the camera 11 captures an image of the wall E1. The recognition unit 152 determines the distance from the first point to the second point as the base line length based on the motion data ms. The base line length is also referred to as the length of the baseline.

[0037] The recognition unit 152 acquires the first imaging data k1 and the second imaging data k2 from the acquisition unit 151. The first imaging data k1 is imaging data kt generated by the camera 11 when the information processing device 1 is located at a first point. The second imaging data k2 is imaging data kt generated by the camera 11 when the information processing device 1 is located at a second point. Each of the first imaging data k1 and the second imaging data k2 shows at least the wall E1.

[0038] The recognition unit 152 performs three-dimensional measurement by performing triangulation using the base line length, the first imaging data k1, and the second imaging data k2.

[0039] The result of the three-dimensional measurement represents the shape of an object present in the target region TR using three-dimensional coordinates. The recognition unit 152 recognizes a wall E1 based on the result of the three-dimensional measurement. For example, the recognition unit 152 recognizes a vertical plane as a wall E1 based on the result of the three-dimensional measurement. The recognition unit 152 determines a distance n from the information processing device 1 to the wall E1 based on the result of the three-dimensional measurement.

[0040] The operation control unit 153 controls the operation of the information processing device 1. The operation control unit 153 provides image data r indicating an image to the touch panel 12, thereby causing the touch panel 12 to display the image indicated by the image data r.

[0041] The operation control unit 153 causes the touch panel 12 to display the first composite image Y1. The operation control unit 153 generates first image data r1 based on the result of the three-dimensional measurement, the imaging data kt, the motion data m indicating the motion of the information processing device 1, and the specifications of the virtual projector C4. The first image data r1 indicates the first composite image Y1. The first image data r1 is an example of image data r.

[0042] The operation control unit 153 determines an image in which the sample image J1, the projector image L1, and the path image L2 are superimposed on the target image H1 as a first composite image Y1.

[0043] The operation control unit 153 uses the three-dimensional virtual space VS to determine the sample images J1, etc. Fig. 6 is a diagram showing an example of the virtual space VS.

[0044] The action control unit 153 reproduces the arrangement of objects in the real space RS by using the virtual space VS.

[0045] The motion control unit 153 sets a first position C1 in the virtual space VS by using the result of the three-dimensional measurement of the wall E1. The first position C1 in the virtual space VS corresponds to the position of the wall E1 in the real space RS.

[0046] The motion control unit 153 determines the shape of the virtual surface C3 based on the result of the three-dimensional measurement of the wall E1. The shape of the virtual surface C3 is the same as the shape indicated by the result of the three-dimensional measurement of the wall E1. The virtual surface C3 is a surface corresponding to the wall E1. The motion control unit 153 places the virtual surface C3 at the first position C1.

[0047] The operation control unit 153 sets a second position C2 in the virtual space VS based on the position of the camera 11 in the real space RS. The second position C2 in the virtual space VS corresponds to the position of the camera 11 in the real space RS. The camera 11 is located in the information processing device 1 together with the touch panel 12. Therefore, the second position C2 in the virtual space VS corresponds to the position of the camera 11 in the real space RS and also corresponds to the position of the touch panel 12 in the real space RS.

[0048] The operation control unit 153 places the virtual projector C4 at the second position C2. Therefore, in the virtual space VS, the relative position of the virtual projector C4 with respect to the second position C2 is fixed. In the virtual space VS, the state in which the relative position of the virtual projector C4 with respect to the second position C2 is fixed is not limited to the state in which the virtual projector C4 is located at the second position C2. For example, in the virtual space VS, the relative position of the virtual projector C4 with respect to the second position C2 may be fixed in a state in which the virtual projector C4 is located at a position different from the second position C2.

[0049] The second position C2 changes in response to a change in the position in real space RS of the touch panel 12. For this reason, in a situation in which the relative position of the virtual projector C4 with respect to the second position C2 is fixed in the virtual space VS, when the position of the touch panel 12 changes in real space RS, the position of the virtual projector C4 changes in the virtual space VS.

[0050] Virtual projector C4 is a virtual projector corresponding to projector 2. The specifications of virtual projector C4 are the same as those of projector 2. The specifications of projector 2 are described in program P1. Therefore, the operation control unit 153 recognizes in advance the specifications of projector 2 and the specifications of virtual projector C4. Note that the specifications of projector 2 include the angle of view of projector 2.

[0051] The operation control unit 153 causes the optical axis direction of the projection lens of the virtual projector C4 relative to the virtual surface C3 to coincide with the optical axis direction of the imaging lens 111 relative to the wall E1. The operation control unit 153 determines the optical axis direction of the imaging lens 111 relative to the wall E1 based on the recognition result of the wall E1 and the motion data m indicating the motion of the information processing device 1.

[0052] The operation control unit 153 places the image v21 on the virtual surface C3. The image v21 is an image displayed on the virtual surface C3 in a situation where the virtual projector C4 projects a rectangular first image onto the virtual surface C3. The rectangular first image is, for example, a rectangular monochromatic image. The rectangular first image is not limited to a rectangular monochromatic image. The rectangular first image may be any image as long as it is a rectangular image. When the optical axis of the projection lens of the virtual projector C4 is inclined with respect to the normal to the virtual surface C3, the image v21 has a keystone distortion. The keystone distortion will be described later. For this reason, the image v21 may not match the rectangular first image. However, the image v21 is an image based on the rectangular first image.

[0053] An image of the image v21 displayed on the virtual surface C3 viewed from the second position C2 is referred to as a “screen image v2.” The screen image v2 is an example of each of the first display image and the second display image.

[0054] The first display image is an image displayed on a virtual plane C3 located at a first position C1 in the virtual space VS when the image is projected from a virtual projector C4 onto the virtual plane C3, as viewed from a second position C2.

[0055] The second display image is an image displayed on the virtual surface C3 viewed from the second position C2 in a situation where an image is projected onto the virtual surface C3 located at the first position C1 from a virtual projector C4 whose relative position with respect to the second position C2 in the virtual space VS is fixed.

[0056] The image v21 is an image showing the area in which the image J1a (not shown) is displayed, and functions as a screen for the image J1a.

[0057] The image J1a is an image displayed on the virtual surface C3 in a situation where the virtual projector C4 projects a rectangular second image onto the virtual surface C3. The rectangular second image is, for example, a rectangular image selected by a user. The rectangular second image is not limited to a rectangular image selected by a user. The rectangular second image may be any image as long as it is a rectangular image. If the optical axis of the projection lens of the virtual projector C4 is inclined with respect to the normal to the virtual surface C3, the image J1a has a keystone distortion. For this reason, the image J1a may not match the rectangular second image. However, the image J1a is an image based on the rectangular second image.

[0058] The sample image J1 is an image of the image J1a displayed on the virtual plane C3 viewed from the second position C2. The sample image J1 is an example of each of the first display image and the second display image.

[0059] The position of the image v21 on the virtual plane C3 is fixed in response to a position fixing instruction from the user. Until the position fixing instruction is received from the user, the operation control unit 153 determines the position of the image v21 on the virtual plane C3 based on the position of the intersection between the virtual plane C3 and the optical axis of the projection lens of the virtual projector C4. For example, the operation control unit 153 aligns the position of the center of the image v21 on the virtual plane C3 with the position of the intersection between the virtual plane C3 and the optical axis of the projection lens of the virtual projector C4. The position of the center of the image v21 is, for example, the position of the intersection of the diagonal lines of the image v21.

[0060] When the operation control unit 153 causes the screen image v2 to be displayed on the touch panel 12, the operation control unit 153 causes the touch panel 12 to display the second composite image Y2 regardless of the position of the touch panel 12 in the real space RS.

[0061] 7 is a diagram showing an example of the second composite image Y2. The second composite image Y2 is an example of the first simulation image. The second composite image Y2 is an image in which a screen image v2 having a size that fits within the touch panel 12 is superimposed on the target image H1.

[0062] The size of the screen image v2 that fits within the touch panel 12 is equal to or smaller than the size a1. The size a1 is the size of the image v21 displayed on the virtual surface C3 when viewed from the second position C2.

[0063] Fig. 8 is a diagram for explaining an example of the size a1. In Fig. 8, the size and shape of the screen image v2 are the same as the size and shape of the image v21. Fig. 8 shows an example of the size a1, a virtual projector C4, a viewing angle C4a of the virtual projector C4, a screen image v2, a virtual surface C3, a virtual camera C5, a viewing angle C5a of the virtual camera C5, a target image H1, and a touch panel 12.

[0064] Virtual camera C5 is a virtual camera corresponding to camera 11. The specifications of virtual camera C5 are the same as those of camera 11. The specifications of camera 11 include the angle of view of camera 11. Therefore, the angle of view C5a of virtual camera C5 is equal to the angle of view of camera 11. The specifications of camera 11 are written in program P1. Therefore, operation control unit 153 recognizes in advance the specifications of camera 11 and the specifications of virtual camera C5.

[0065] The virtual camera C5 is located at the second position C2, similar to the virtual projector C4. The position of the optical axis C5b of the virtual camera C5 coincides with the position of the optical axis C4b of the virtual projector C4. The angle of view C4a of the virtual projector C4 is equal to the angle of view of the projector 2. Therefore, the relationship between the angle of view C5a of the virtual camera C5 and the angle of view C4a of the virtual projector C4 is equal to the relationship between the angle of view of the camera 11 and the angle of view of the projector 2.

[0066] The size of the image v21 and the size a1 of the screen image v2 depend on the angle of view C4a of the virtual projector C4. The size of the target image H1 depends on the angle of view of the camera 11. The angle of view of the camera 11 is equal to the angle of view C5a of the virtual camera C5. Therefore, the size of the target image H1 depends on the angle of view C5a of the virtual camera C5. The size of the target image H1 is set to a size that fits within the touch panel 12.

[0067] When the screen image v2 is superimposed on the target image H1 in a situation where the angle of view C4a of the virtual projector C4 is larger than the angle of view C5a of the virtual camera C5, the screen image v2 does not fit within the target image H1. When the screen image v2 does not fit within the target image H1, the part of the screen image v2 that is not superimposed on the target image H1 may not be displayed on the touch panel 12. For example, as shown in FIG. 8, when the screen image v2 does not fit within the target image H1 in a situation where the horizontal length of the target image H1 matches the horizontal length of the touch panel 12, the part of the screen image v2 that is not superimposed on the target image H1 is not displayed on the touch panel 12. Even if the user sees the touch panel 12 that does not display the entire screen image v2, it is difficult to imagine the projection image F1 that is actually projected.

[0068] Therefore, when the operation control unit 153 causes the screen image v2 to be displayed on the touch panel 12, the operation control unit 153 causes the second composite image Y2 to be displayed on the touch panel 12 regardless of the position of the touch panel 12. The second composite image Y2 is an image in which the screen image v2, which is sized to fit on the touch panel 12, is superimposed on the target image H1, as shown in FIG.

[0069] The operation control unit 153 adjusts the size of the screen image v2 to a size that fits within the touch panel 12 by adjusting the distance Dp between the virtual plane C3 and the virtual projector C4, as shown in FIG.

[0070] For example, the operation control unit 153 adjusts the distance Dp between the virtual plane C3 and the virtual projector C4 based on the distance between the touch panel 12 and the wall E1, thereby adjusting the size of the screen image v2 to a size that fits within the touch panel 12. Note that since the touch panel 12 is located on the information processing device 1, the operation control unit 153 uses the distance n from the information processing device 1 to the wall E1 as the distance between the touch panel 12 and the wall E1.

[0071] As an example, the operation control unit 153 adjusts the distance between the virtual plane C3 and the virtual projector C4 based on the distance between the touch panel 12 and the wall E1 and the angle of view C4a of the virtual projector C4, thereby adjusting the size of the screen image v2 to a size that fits within the touch panel 12. Fig. 9 shows an example in which the size of the screen image v2 is adjusted by moving the virtual projector C4 closer to the virtual plane C3 by these methods. Details will be described later with reference to Figs. 12 and 13.

[0072] The operation control unit 153 provides the touch panel 12 with second image data r2 indicating the second composite image Y2, thereby causing the touch panel 12 to display the second composite image Y2.

[0073] Furthermore, the operation control unit 153 changes the screen image v2 to a sample image J1 in response to, for example, an instruction from the user.

[0074] When the operation control unit 153 causes the sample image J1 to be displayed on the touch panel 12, the operation control unit 153 causes the first composite image Y1 to be displayed on the touch panel 12 regardless of the position of the touch panel 12. The first composite image Y1 is an image in which the sample image J1, which has a size that fits on the touch panel 12, is superimposed on the target image H1. For example, the operation control unit 153 adjusts the size of the sample image J1 to a size that fits on the touch panel 12 using a method similar to the method for adjusting the size of the screen image v2.

[0075] The operation control unit 153 provides the first image data r1 representing the first composite image Y1 to the touch panel 12, thereby causing the touch panel 12 to display the first composite image Y1.

[0076] A3: Recognition of wall E1 FIG. 10 is a flowchart for explaining the operation of recognizing the wall E1.

[0077] When the touch panel 12 receives a start instruction from a user, in step S101, the processing device 15 starts executing the program P1, which is an application program.

[0078] Next, in step S102, the operation control unit 153 causes the camera 11 to start capturing an image of the target area TR. The camera 11 captures an image of the target area TR to generate captured image data kt.

[0079] Next, in step S103, the motion control unit 153 operates the motion sensor 13. The motion sensor 13 generates motion data m indicating the motion of the information processing device 1.

[0080] Next, in step S104, the acquisition unit 151 starts acquiring the imaging data kt and the motion data m.

[0081] Next, in step S105, the operation control unit 153 causes the recognition unit 152 to recognize the wall E1.

[0082] In step S105, the recognition unit 152 performs three-dimensional measurement of an object present in the target region TR based on the imaging data kt and the motion data m acquired by the acquisition unit 151 during the scanning situation.

[0083] The scanning situation means a situation in which the information processing device 1 is moved from a first point to a second point while the camera 11 is capturing an image of the wall E1. The first point is, for example, the position of the information processing device 1 at the start of the scanning situation. The second point is, for example, the position of the information processing device 1 at the end of the scanning situation. The imaging data kt acquired by the acquisition unit 151 in the scanning situation is the first imaging data k1 and the second imaging data k2. The first imaging data k1 is the imaging data kt generated by the camera 11 when the information processing device 1 is located at the first point. The second imaging data k2 is the imaging data kt generated by the camera 11 when the information processing device 1 is located at the second point. The motion data m acquired by the acquisition unit 151 in the scanning situation is the motion data ms. The motion data ms is the motion data m generated by the motion sensor 13 in a situation in which the information processing device 1 is moved from the first point to the second point while the camera 11 is capturing an image of the wall E1.

[0084] The recognition unit 152 determines the distance from the first point to the second point as a base line length based on the motion data ms. The recognition unit 152 performs three-dimensional measurement by performing triangulation using the base line length, the first imaging data k1, and the second imaging data k2.

[0085] Next, the recognition unit 152 recognizes the wall E1 based on the result of the three-dimensional measurement. For example, the recognition unit 152 recognizes a vertical plane as the wall E1 based on the result of the three-dimensional measurement.

[0086] A4: Display of the first composite image Y1 and the second composite image Y2 Fig. 11 is a flow chart for explaining the operation of displaying the first composite image Y1 and the second composite image Y2. The operation shown in Fig. 11 is executed in a situation where the wall E1 is recognized.

[0087] In step S201, the operation control unit 153 causes the recognition unit 152 to determine the distance n from the information processing device 1 to the wall E1.

[0088] In step S201, the recognition unit 152 first acquires motion data m indicating the motion of the information processing device 1 from the acquisition unit 151. Next, the recognition unit 152 determines the position of the information processing device 1 in the real space RS, i.e., the position of the camera 11 in the real space RS, based on the motion data m. Next, the recognition unit 152 determines the distance n from the information processing device 1 to the wall E1 based on the result of the three-dimensional measurement and the position of the camera 11. For example, the recognition unit 152 determines the distance from the camera 11 to the wall E1 as the distance n from the information processing device 1 to the wall E1.

[0089] Next, in step S202, the operation control unit 153 generates a virtual space VS.

[0090] Next, in step S203, the operation control unit 153 sets a first position C1 and a second position C2 in the virtual space VS.

[0091] In step S203, the motion control unit 153 first sets a first position C1 in the virtual space VS by using the result of the three-dimensional measurement of the wall E1. The first position C1 in the virtual space VS corresponds to the position of the wall E1 in the real space RS. Next, the motion control unit 153 sets a second position C2 in the virtual space VS based on the position of the camera 11 in the real space RS. The second position C2 in the virtual space VS corresponds to the position of the camera 11 in the real space RS.

[0092] Next, in step S204, the operation control unit 153 places a virtual surface C3 in the virtual space VS.

[0093] In step S204, the operation control unit 153 first matches the shape of the virtual surface C3 with the shape of the wall E1 based on the result of the three-dimensional measurement of the wall E1. Then, the operation control unit 153 places the virtual surface C3 at the first position C1.

[0094] Next, in step S205, the operation control unit 153 places the virtual projector C4 at the second position C2.

[0095] In step S205, the operation control unit 153 places the virtual projector C4 at the second position C2, thereby fixing the relative position of the virtual projector C4 with respect to the second position C2. Next, the operation control unit 153 determines the orientation of the optical axis of the imaging lens 111 with respect to the wall E1, based on the recognition result of the wall E1 and the motion data m indicating the motion of the information processing device 1. Next, the operation control unit 153 matches the orientation of the optical axis of the projection lens of the virtual projector C4 with respect to the virtual plane C3 with the orientation of the optical axis of the imaging lens 111 with respect to the wall E1.

[0096] Next, in step S206, the operation control unit 153 places the image v21 on the virtual plane C3.

[0097] In step S206, the operation control unit 153 adjusts the position of the center of the image v21 on the virtual plane C3 to the position of the intersection between the virtual plane C3 and the optical axis of the projection lens of the virtual projector C4. The position of the center of the image v21 is not limited to the position of the intersection between the virtual plane C3 and the optical axis of the projection lens of the virtual projector C4, but may be any position based on the position of the intersection. Next, the operation control unit 153 determines the correspondence between the distance n from the information processing device 1 to the wall E1 and the size of the image v21 based on the angle of view of the projector 2. The operation control unit 153 determines the size of the image v21 based on the correspondence between the distance n and the size of the image v21 and the determination result of the distance n by the recognition unit 152. For example, the operation control unit 153 increases the size of the image v21 in response to an increase in the distance n. Next, the operation control unit 153 sets a path of the projection light from the virtual projector C4 to the image v21 in the virtual space VS.

[0098] Next, in step S207, the operation control unit 153 places the virtual camera C5 at the second position C2.

[0099] In step S207, the operation control unit 153 places the virtual camera C5 at the second position C2, thereby fixing the relative position of the virtual camera C5 with respect to the second position C2. Next, the operation control unit 153 causes the orientation of the optical axis of the imaging lens of the virtual camera C5 with respect to the virtual plane C3 to coincide with the orientation of the optical axis of the imaging lens 111 with respect to the wall E1.

[0100] Subsequently, in step S208, the operation control unit 153 determines whether or not the angle of view C4a of the virtual projector C4 is equal to or larger than the angle of view C5a of the virtual camera C5.

[0101] If the operation control unit 153 determines that the angle of view C4a of the virtual projector C4 is equal to or larger than the angle of view C5a of the virtual camera C5, the operation control unit 153 adjusts the size of the image v21 in step S209. If the operation control unit 153 determines that the angle of view C4a of the virtual projector C4 is smaller than the angle of view C5a of the virtual camera C5, step S209 is skipped.

[0102] In step S209, the operation control unit 153 adjusts the distance between the virtual plane C3 and the virtual projector C4 based on the distance between the touch panel 12 and the wall E1, thereby adjusting the size of the image v21 to a size that fits within the touch panel 12. For example, the operation control unit 153 adjusts the distance between the virtual plane C3 and the virtual projector C4 based on the distance between the touch panel 12 and the wall E1 by using the following formulas 1 and 2. Specifically, the operation control unit 153 adjusts the size of the image v21 by moving the virtual projector C4 closer to the virtual plane C3.

[0103] FIG. 12 is a diagram showing a situation before the virtual projector C4 is brought closer to the virtual surface C3. In the situation shown in FIG. 12, the distance Dp from the virtual projector C4 to the virtual surface C3 is equal to the distance n2 from the virtual camera C5 to the virtual surface C3. Note that the operation control unit 153 uses the distance n from the information processing device 1 to the wall E1 as the distance n2 from the virtual camera C5 to the virtual surface C3. Since the information processing device 1 includes the touch panel 12, the distance n from the information processing device 1 to the wall E1 means the distance between the touch panel 12 and the wall E1. Therefore, the distance n2 from the virtual camera C5 to the virtual surface C3 means the distance between the touch panel 12 and the wall E1.

[0104] When the distance n2 from the virtual camera C5 to the virtual plane C3 is the distance n from the information processing device 1 to the wall E1, the width H1w of the target image H1 is expressed by Equation 1. H1w=tan(C5a / 2)*n2*2 ··· Formula 1

[0105] In a situation where the angle of view C4a of the virtual projector C4 is greater than or equal to the angle of view C5a of the virtual camera C5, if the distance Dp from the virtual projector C4 to the virtual surface C3 is equal to the distance n2 from the virtual camera C5 to the virtual surface C3, the width v2w of the screen image v2 is greater than or equal to the width H1w of the target image H1.

[0106] FIG. 13 is a diagram showing the situation after bringing the virtual projector C4 closer to the virtual plane C3. In the situation shown in FIG. 13, the distance Dp from the virtual projector C4 to the virtual plane C3 is set to the distance n11. The distance n11 is smaller than the distance n2 from the virtual camera C5 to the virtual plane C3.

[0107] The operation control unit 153 sets the distance Dp from the virtual projector C4 to the virtual plane C3 to the distance n11 so that the width v2w1 of the screen image v2 becomes smaller than the width H1w of the target image H1. When the width v2w1 of the screen image v2 is smaller than the width H1w of the target image H1, as shown in FIG. 13, a gap is generated between the edge of the screen image v2 and the edge of the touch panel 12. For this reason, the edge of the screen image v2 becomes easier for the user to recognize. Therefore, the user can easily recognize the size of the screen image v2.

[0108] The operation control unit 153 determines the distance n11 by using Equation 2. n11 = H1w * α * b ··· Equation 2 α = (Dp) / (v2wa), b = 0.98. Dp is the distance from the virtual projector C4 to the virtual plane C3. v2wa is the width of the screen image v2 when the distance from the virtual projector C4 to the virtual plane C3 is the distance Dp. α is uniquely determined by the viewing angle C4a of the virtual projector C4. That is, α is a fixed value depending on the specifications of the projector 2. b is a coefficient for making the width of the screen image v2 smaller than the width H1w of the target image H1.

[0109] b is not limited to 0.98, and any value that satisfies the relationship 0 < b < 1 may be used. For example, the value of b may be 0.9.

[0110] The width of the screen image v2 increases in accordance with an increase in the angle formed between the optical axis of the projection lens of the virtual projector C4 and the normal line of the virtual plane C3. When the angle formed between the optical axis of the projection lens of the virtual projector C4 and the normal line of the virtual plane C3 is 0 degrees, the operation control unit 153 may use 0.98 as the value of b. The operation control unit 153 may decrease the value of b within the range satisfying the relationship 0 < b < 1 in accordance with an increase in the angle formed between the optical axis of the projection lens of the virtual projector C4 and the normal line of the virtual plane C3. In this case, even if the angle formed between the optical axis of the projection lens of the virtual projector C4 and the normal line of the virtual plane C3 increases, the screen image v2 can be contained within the touch panel 12.

[0111] The operation control unit 153 sets the distance Dp to the distance n11 by moving the virtual projector C4 closer to the virtual plane C3 along the optical axis of the projection lens of the virtual projector C4. By setting the distance Dp to the distance n11, the operation control unit 153 makes the width of the screen image v2 smaller than the width H1w of the target image H1.

[0112] Returning to the description with reference to FIG. 11. Subsequently, in step S210, the operation control unit 153 determines the second composite image Y2.

[0113] In step S210, the operation control unit 153 first deletes the virtual plane C3 from the virtual space VS while leaving the image v21, the virtual projector C4, the path of the projection light from the virtual projector C4 toward the image v21, and the virtual camera C5 in the virtual space VS.

[0114] Subsequently, the operation control unit 153 determines the image obtained when the virtual camera C5 performs imaging as the first virtual image.

[0115] The first virtual image has transparency. The first virtual image includes an image of the image v21 on the virtual plane C3 as seen from the second position C2. In the first virtual image, the image of the image v21 on the virtual plane C3 as seen from the second position C2 serves as an example of the screen image v2.

[0116] The first virtual image further includes an image showing the virtual projector C4. In the first virtual image, the image showing the virtual projector C4 is an example of the projector image L1. Note that both the virtual camera C5 and the virtual projector C4 are located at the second position C2. For this reason, for example, when the virtual camera C5 is larger than the virtual projector C4, the virtual projector C4 may be located inside the virtual camera C5. In this case, the first virtual image obtained when the virtual camera C5 performs imaging does not include an image showing the virtual projector C4. When the first virtual image does not include an image showing the virtual projector C4, the operation control unit 153 adds the projector image L1 to the first virtual image. The position where the projector image L1 is added to the first virtual image is, for example, the lower part in the first virtual image.

[0117] The first virtual image further includes an image showing the path of the projection light from the virtual projector C4 to the screen image v2. In the first virtual image, the image showing the path of the projection light from the virtual projector C4 to the screen image v2 is an example of a path image L2.

[0118] Next, the operation control unit 153 determines a second composite image Y2 by superimposing the first virtual image on the target image H1.

[0119] Next, in step S211, the operation control section 153 generates second image data r2 representing the second composite image Y2.

[0120] Next, in step S212, the operation control unit 153 provides the touch panel 12 with second image data r2, thereby causing the touch panel 12 to display the second composite image Y2.

[0121] Next, in step S213, the motion control unit 153 determines the image J1a. In step S213, the motion control unit 153 determines the image J1a in response to, for example, an instruction from the user. The shape of the image J1a is the same as the shape of the image v21.

[0122] Next, in step S214, the operation control unit 153 changes the image v21 to the image J1a.

[0123] Next, in step S215, the operation control unit 153 determines the first composite image Y1.

[0124] In step S215, the operation control unit 153 determines the image obtained when the virtual camera C5 performs image capture as the second virtual image.

[0125] The second virtual image has transparency. The second virtual image includes an image of the image J1a on the virtual plane C3 viewed from the second position C2. The image of the image J1a on the virtual plane C3 viewed from the second position C2 is the sample image J1.

[0126] The second virtual image further includes an image showing the virtual projector C4. In the second virtual image, the image showing the virtual projector C4 is an example of a projector image L1. Note that, for example, if the second virtual image does not include an image showing the virtual projector C4 because the virtual camera C5 is larger than the virtual projector C4, the operation control unit 153 adds the projector image L1 to the second virtual image. The position where the projector image L1 is added to the second virtual image is, for example, the lower part in the second virtual image.

[0127] The second virtual image further includes an image showing the path of the projection light from the virtual projector C4 to the image J1a. In the second virtual image, the image showing the path of the projection light from the virtual projector C4 to the image J1a is an example of a path image L2.

[0128] Next, the operation control unit 153 determines a first composite image Y1 by superimposing the second virtual image on the target image H1.

[0129] Next, in step S216, the operation control unit 153 generates first image data r1 representing the first composite image Y1.

[0130] Next, in step S217, the operation control unit 153 provides the first image data r1 to the touch panel 12, thereby causing the touch panel 12 to display the first composite image Y1.

[0131] A5: An example of operation Next, an example of the above-mentioned operation will be described. Note that when the above-mentioned operation is executed, the projector 2 may or may not be present in a building having the wall E1. In step S101, the processing device 15 starts execution of the program P1. Step S101 is executed when the touch panel 12 receives a start instruction from the user. The start instruction is, for example, a tap on the icon i of the program P1 displayed on the touch panel 12. FIG. 14 is a diagram showing an example of the icon i displayed on the touch panel 12.

[0132] When the icon i is tapped, the processing device 15 reads the program P1 from the storage device 14. Then, the processing device 15 executes the program P1.

[0133] Until the processing device 15 executes the program P1, the processing device 15 causes a splash screen to be displayed on the touch panel 12. When the processing device 15 executes the program P1, the operation control unit 153 causes the touch panel 12 to display a first guide image t1.

[0134] 15 is a diagram showing an example of a first guide image t1. The first guide image t1 shows an overview of functions of the information processing device 1 that are realized by executing the program P1.

[0135] For example, the first guide image t1 shown in Fig. 15 shows a projector d1 making the comment "I'll try installing a projector in my room using AR." AR stands for Augmented Reality.

[0136] The comment indicated by the first guide image t1 is not limited to "I'll try installing a projector in my room using AR" and can be changed as appropriate. The first guide image t1 does not have to show the projector d1. The first guide image t1 may show an object different from the projector d1, such as an animal, instead of the projector d1.

[0137] Next, in step S102, the camera 11 captures an image of the target region TR to generate imaging data kt. Next, in step S103, the motion sensor 13 generates motion data m. Next, in step S104, the acquisition unit 151 acquires the imaging data kt and the motion data m.

[0138] After completion of step S104, the operation control unit 153 may acquire the imaging data kt from the acquisition unit 151. In this case, the operation control unit 153 provides the imaging data kt as image data r to the touch panel 12, thereby causing the touch panel 12 to display the target image H1. Fig. 16 is a diagram showing an example of the information processing device 1 displaying the target image H1.

[0139] Next, in step S105, the operation control unit 153 causes the recognition unit 152 to recognize the wall E1.

[0140] In step S105, the operation control unit 153 first causes the touch panel 12 to display the image u1.

[0141] 17 is a diagram showing an example of an image u1. The image u1 is an image in which a second guide image t2 is superimposed on a target image H1. The second guide image t2 shows a projector d1 making a comment, "Nice to meet you! Let's try using the projector!"

[0142] The comment shown by the second guide image t2 is not limited to "Nice to meet you! Let's try using the projector!" and can be changed as appropriate. The second guide image t2 does not have to show the projector d1. The second guide image t2 may show an object different from the projector d1, such as an animal, instead of the projector d1.

[0143] Next, the operation control unit 153 causes the touch panel 12 to display the image u2.

[0144] 18 is a diagram showing an example of an image u2. The image u2 is an image in which a third guide image t3 and a button v1 are superimposed on a target image H1. The third guide image t3 is an image that prompts the user to generate a scanning situation. The scanning situation means a situation in which the information processing device 1 is moved from a first point to a second point while the camera 11 captures an image of a wall E1. The button v1 is a button for receiving an input for starting the scanning situation.

[0145] The third guide image t3 shows a projector d1 saying, "First, press the button where you want to capture the image and shake your smartphone."

[0146] The comment shown by the third guide image t3 is not limited to "First, press the button where you want to take a picture and shake your smartphone," but can be changed as appropriate as long as it encourages the user to generate a scanning situation. The third guide image t3 does not need to show the projector d1. The third guide image t3 may show an object different from the projector d1, such as an animal, instead of the projector d1. The form of the button v1 is not limited to the form shown in FIG. 18, but can be changed as appropriate.

[0147] Following the comment in the third guide image t3, for example, the user presses the button v1 while the wall E1 is displayed on the touch panel 12, and then shakes the information processing device 1.

[0148] When the touch panel 12 detects a tap on the button v1, the operation control unit 153 causes the touch panel 12 to display an image u3.

[0149] 19 is a diagram showing an example of an image u3. The image u3 is an image obtained by superimposing a fourth guide image t4 on the target image H1. The fourth guide image t4 shows the projector d1 making the comment "I will scan the wall surface."

[0150] The comment shown by the fourth guide image t4 is not limited to "I will scan the wall surface" and can be changed as appropriate. The fourth guide image t4 does not have to show the projector d1. The fourth guide image t4 may show an object different from the projector d1, for example, an animal, instead of the projector d1.

[0151] 20 is a diagram showing a situation in which a user shakes the information processing device 1. When the user shakes the information processing device 1, a scan situation occurs.

[0152] In the scanning state, the recognition unit 152 acquires the first imaging data k1, the second imaging data k2, and the motion data ms.

[0153] The recognition unit 152 recognizes the wall E1 based on the first imaging data k1, the second imaging data k2, and the motion data ms.

[0154] Next, in step S201, the operation control unit 153 causes the recognition unit 152 to determine the distance n from the information processing device 1 to the wall E1.

[0155] Next, in step S202, the operation control unit 153 generates a virtual space VS.

[0156] Next, in step S203, the operation control unit 153 sets a first position C1 and a second position C2 in the virtual space VS.

[0157] Next, in step S204, the operation control unit 153 places a virtual surface C3 in the virtual space VS.

[0158] Next, in step S205, the operation control unit 153 places the virtual projector C4 in the virtual space VS.

[0159] Next, in step S206, the operation control unit 153 places the image v21 on the virtual plane C3.

[0160] In step S206, the operation control unit 153 first aligns the position of the center of the image v21 on the virtual plane C3 with the position of the intersection between the virtual plane C3 and the optical axis of the projection lens of the virtual projector C4.

[0161] Next, the operation control unit 153 sets a path of projection light from the virtual projector C4 toward the image v21 in the virtual space VS.

[0162] Next, in step S207, the operation control unit 153 sets up a virtual camera C5 at the second position C2. The position of the optical axis of the imaging lens of the virtual camera C5 coincides with the position of the optical axis of the projection lens of the virtual projector C4.

[0163] Subsequently, in step S208, the operation control unit 153 determines whether or not the angle of view C4a of the virtual projector C4 is equal to or larger than the angle of view C5a of the virtual camera C5.

[0164] If the operation control unit 153 determines that the angle of view C4a of the virtual projector C4 is equal to or larger than the angle of view C5a of the virtual camera C5, the operation control unit 153 adjusts the size of the image v21 in step S209. If the operation control unit 153 determines that the angle of view C4a of the virtual projector C4 is smaller than the angle of view C5a of the virtual camera C5, step S209 is skipped.

[0165] In step S209, the operation control unit 153 adjusts the size of the image v21 by moving the virtual projector C4 closer to the virtual plane C3.

[0166] Next, in step S210, the operation control unit 153 determines a second composite image Y2.

[0167] Next, in step S211, the operation control section 153 generates second image data r2 representing the second composite image Y2.

[0168] Next, in step S212, the operation control unit 153 provides the touch panel 12 with second image data r2, thereby causing the touch panel 12 to display the second composite image Y2.

[0169] Fig. 21 is a diagram showing an example of the second composite image Y2. In the second composite image Y2 shown in Fig. 21, in addition to the first virtual image, a fifth guide image t5 is superimposed on the target image H1. The first virtual image is an image obtained by the virtual camera C5 performing imaging.

[0170] In the second composite image Y2, the position of the screen image v2 with respect to the wall E1 changes in response to a change in the position of the touch panel 12 in the real space RS and a change in the orientation of the touch panel 12 in the real space RS. The touch panel 12 is mounted on the information processing device 1. Therefore, a change in the position of the touch panel 12 in the real space RS means a change in the position of the information processing device 1 in the real space RS. Furthermore, a change in the orientation of the touch panel 12 in the real space RS means a change in the orientation of the information processing device 1 in the real space RS. Therefore, by changing both the position and the orientation of the information processing device 1, the user can adjust the position of the screen image v2 with a feeling as if the information processing device 1 were the projector 2.

[0171] Furthermore, the portion of the wall E1 shown in the target image H1 is changed in response to a change in the position of the touch panel 12 in the real space RS and a change in the orientation of the touch panel 12 in the real space RS.

[0172] Therefore, when either the position of the touch panel 12 in the real space RS or the orientation of the touch panel 12 in the real space RS changes, the portion of the wall E1 shown in the target image H1 of the second composite image Y2 changes, whereas the position of the projector image L1 in the second composite image Y2 does not change. Therefore, by looking at the second composite image Y2 displayed on the touch panel 12, the user can adjust the position of the screen image v2 on the wall E1 with the sensation that the projector 2 is present at the position of the information processing device 1.

[0173] The screen image v2 includes an operation button v3. The operation button v3 is used to fix the position of the screen image v2 with respect to the wall E1. More specifically, the operation button v3 is used by the user to input a position setting instruction.

[0174] The form of the operation button v3 is not limited to the form shown in Fig. 21 and can be changed as appropriate. The color of the screen image v2 having the operation button v3 is gray. The color of the screen image v2 having the operation button v3 is not limited to gray and can be changed as appropriate.

[0175] The fifth guide image t5 is an image that prompts the user to perform an operation to fix the position of the screen image v2 relative to the wall E1. The fifth guide image t5 shows the projector d1 making the comment "Once the location of the screen has been decided, press the operation button."

[0176] The comment shown by the fifth guide image t5 is not limited to "Once the location of the screen is decided, press the operation button", but can be changed as appropriate as long as it encourages the user to perform an operation to fix the position of the screen image v2. The fifth guide image t5 does not need to show the projector d1. The fifth guide image t5 may show an object different from the projector d1, such as an animal, instead of the projector d1.

[0177] The user checks the second composite image Y2 while changing the position of the information processing device 1. FIG. 22 is a diagram showing an example of the second composite image Y2 displayed on the information processing device 1 when the position of the information processing device 1 becomes closer to the wall E1 than the position of the information processing device 1 displaying the second composite image Y2 shown in FIG. 21. In FIG. 22, the fifth guide image t5 is omitted. The closer the information processing device 1 is to the wall E1, the smaller the ratio of the size of the screen image v2 to the size of the wall E1 becomes. The size of the screen image v2 shown in FIG. 22 is smaller than the size of the screen image v2 shown in FIG. 21. Note that the size of the screen image v2 displayed in the second composite image Y2 does not need to be changed.

[0178] The operation control unit 153 may superimpose an image showing the projector d1 issuing the comment "It will get smaller if you get closer" on the second composite image Y2 in order to inform the user of a method for reducing the ratio of the size of the screen image v2 to the size of the wall E1. The comment "It will get smaller if you get closer" is an example of a first operation comment indicating an operation for reducing the ratio of the size of the screen image v2 to the size of the wall E1.

[0179] The first operation comment is not limited to the comment "It will get smaller if you get closer" and can be changed as appropriate. If the first operation comment is displayed, the projector d1 that issues the first operation comment does not need to be displayed. The object that issues the first operation comment is not limited to the projector d1, and may be, for example, an object different from the projector d1, such as an animal.

[0180] 23 is a diagram showing an example of the second composite image Y2 displayed on the information processing device 1 when the position of the information processing device 1 is farther from the wall E1 than the position of the information processing device 1 displaying the second composite image Y2 shown in FIG. 21. In FIG. 23, the fifth guide image t5 is omitted. The farther the information processing device 1 is from the wall E1, the larger the ratio of the size of the screen image v2 to the size of the wall E1. The size of the screen image v2 shown in FIG. 23 is larger than the size of the screen image v2 shown in FIG. 21. Note that the size of the screen image v2 displayed in the second composite image Y2 does not need to be changed.

[0181] The operation control unit 153 may superimpose an image showing the projector d1 issuing the comment "It will get bigger if you move away" on the second composite image Y2 to inform the user of a method for increasing the ratio of the size of the screen image v2 to the size of the wall E1. The comment "It will get bigger if you move away" is an example of a second operation comment indicating an operation for increasing the ratio of the size of the screen image v2 to the size of the wall E1.

[0182] The second operation comment is not limited to the comment "It will get bigger if you move away" and can be changed as appropriate. If the second operation comment is displayed, the projector d1 that issues the second operation comment does not need to be displayed. The object that issues the second operation comment is not limited to the projector d1, and may be, for example, an object other than the projector, such as an animal.

[0183] The operation control unit 153 may change the transmittance of the screen image v2 in the second composite image Y2 according to the distance n from the information processing device 1 to the wall E1. For example, the operation control unit 153 increases the transmittance of the screen image v2 in the second composite image Y2 as the distance n increases. In this case, the visibility of the screen image v2 in the second composite image Y2 decreases as the distance n increases. Therefore, the operation control unit 153 can simulate the phenomenon in which the visibility of the projection image F1 on the wall E1 decreases as the distance from the wall E1 to the projector 2 increases.

[0184] FIG. 24 is a diagram showing an example of the second composite image Y2 displayed on the information processing device 1 when the optical axis of the imaging lens 111 is inclined with respect to the normal line of the wall E1. In this case, the screen image v2 has a distortion according to the inclination of the optical axis of the imaging lens 111 with respect to the normal line of the wall E1. This distortion is called a keystone distortion. When the projector 2 has a distortion correction function for correcting the keystone distortion, the operation control unit 153 corrects the keystone distortion of the screen image v2 by a distortion correction function equivalent to the distortion correction function of the projector 2. FIG. 25 is a diagram showing an example of the second composite image Y2 having the screen image v2 with the keystone distortion corrected as shown in FIG. 24. In FIG. 24 and FIG. 25, the fifth guide image t5 is omitted.

[0185] When the touch panel 12 detects a tap on the operation button v3, the operation control unit 153 fixes the screen image v2 at the position where the screen image v2 was displayed when the operation button v3 was tapped. Moreover, the operation control unit 153 fixes the image v21 at the position where the image v21 was displayed when the operation button v3 was tapped.

[0186] Next, the operation control unit 153 updates the second composite image Y2 to an image u5. For example, the operation control unit 153 executes the following operations on the second composite image Y2: deleting the operation button v3, changing the color of the screen image v2 from gray to blue, and adding a sixth guide image t6, thereby updating the second composite image Y2 to an image u5. The color of the screen image v2 after the change is not limited to blue and can be changed as appropriate. The image u5 is an example of a first simulation image.

[0187] 26 is a diagram showing an example of the image u5. A sixth guide image t6 in the image u5 is an image that prompts the user to decide on an image to be displayed on the screen image v2.

[0188] In FIG. 26, the sixth guide image t6 shows projector d1 making the comment "Tap the screen to project whatever you like onto the screen."

[0189] The comment shown by the sixth guide image t6 is not limited to "Tap the screen to project whatever you like onto the screen," but can be changed as appropriate as long as it prompts the user to decide on an image to be displayed on the screen image v2. The sixth guide image t6 does not have to show the projector d1. The sixth guide image t6 may show an object different from the projector d1, such as an animal, instead of the projector d1.

[0190] The user can check the fixed screen image v2 by looking at the image u5 while moving the information processing device 1. FIG. 27 is a diagram showing an example of the image u5 displayed on the information processing device 1 when the position of the information processing device 1 is closer to the wall E1 than the position of the information processing device 1 displaying the image u5 shown in FIG. 26. In FIG. 27, the sixth guide image t6 is omitted. Even in a situation where the position of the screen image v2 is fixed, the ratio of the size of the screen image v2 to the size of the wall E1 decreases as the distance between the information processing device 1 and the wall E1 decreases. The size of the screen image v2 shown in FIG. 27 is smaller than the size of the screen image v2 shown in FIG. 26. The size of the screen image v2 may be constant.

[0191] The operation control unit 153 may superimpose an image showing the projector d1 issuing a first operation comment such as "It will get smaller if you get closer" on the image u5. If the first operation comment is displayed, the projector d1 issuing the first operation comment does not need to be displayed. The object issuing the first operation comment is not limited to the projector d1, and may be, for example, an object different from the projector d1, such as an animal.

[0192] FIG. 28 is a diagram showing an example of an image u5 displayed on the information processing device 1 when the position of the information processing device 1 is farther from the wall E1 than the position of the information processing device 1 displaying the image u5 shown in FIG. 26. In FIG. 28, the sixth guide image t6 is omitted. Even in a situation where the position of the screen image v2 is fixed, the ratio of the size of the screen image v2 to the size of the wall E1 increases as the distance between the information processing device 1 and the wall E1 increases. The size of the screen image v2 shown in FIG. 28 is larger than the size of the screen image v2 shown in FIG. 26. The size of the screen image v2 may be constant.

[0193] The operation control unit 153 may superimpose an image showing the projector d1 issuing a second operation comment such as "It will get bigger if you move away" on the image u5. If the second operation comment is displayed, the projector d1 issuing the second operation comment does not need to be displayed. The object issuing the second operation comment is not limited to the projector d1, and may be, for example, an object different from the projector d1, such as an animal.

[0194] The operation control unit 153 may change the transmittance of the screen image v2 in the image u5 according to the distance n from the information processing device 1 to the wall E1. For example, the operation control unit 153 increases the transmittance of the screen image v2 in the image u5 as the distance n increases.

[0195] FIG. 29 is a diagram showing an example of an image u5 displayed on the information processing device 1 when the optical axis of the imaging lens 111 is inclined with respect to the normal line of the wall E1. In this case, the screen image v2 has a keystone distortion according to the inclination of the optical axis of the imaging lens 111 with respect to the normal line of the wall E1. When the projector 2 has a distortion correction function for correcting the keystone distortion, the operation control unit 153 corrects the keystone distortion of the screen image v2 by a distortion correction function equivalent to the distortion correction function of the projector 2. FIG. 30 is a diagram showing an example of an image u5 having the screen image v2 with the keystone distortion corrected shown in FIG. 29. In FIG. 29 and FIG. 30, the sixth guide image t6 is omitted.

[0196] The user can determine an image to be displayed on the screen image v2 by operating the information processing device 1 in accordance with the sixth guide image t6.

[0197] When the touch panel 12 detects a tap on the screen image v2 in the image u5, the operation control unit 153 causes the touch panel 12 to display the menu image v4.

[0198] 31 is a diagram showing an example of the menu image v4. The menu image v4 includes a selection button v5.

[0199] The selection button v5 is used to determine the image to be displayed on the screen image v2, that is, the sample image J1.

[0200] When the touch panel 12 detects a tap on the selection button v5, the operation control unit 153 causes the touch panel 12 to display an image v81.

[0201] 32 is a diagram showing an example of an image v81. The image v81 shows an image candidate v8 to be displayed on the screen image v2. The image candidate v8 is an image corresponding to the projection image F1 projected from the projector 2. For example, the image candidate v8 is an image showing the projection image F1 projected from the projector 2. The image candidate v8 is, for example, a photograph image indicated by photograph data. The image candidate v8 may also be an image of a document indicated by document data.

[0202] The user taps one image candidate v8 to be used as the sample image J1. When the touch panel 12 detects the tap on the image candidate v8, the operation control unit 153 determines the tapped image candidate v8 as the rectangular second image.

[0203] Next, in step S213, the operation control unit 153 determines the image J1a by matching the shape and size of the rectangular second image to the shape and size of the image v21.

[0204] Next, in step S214, the operation control unit 153 changes the image v21 to the image J1a.

[0205] Next, in step S215, the operation control unit 153 determines the first composite image Y1.

[0206] Next, in step S216, the operation control unit 153 generates first image data r1 representing the first composite image Y1.

[0207] Next, in step S217, the operation control unit 153 provides the first image data r1 to the touch panel 12, thereby causing the touch panel 12 to display the first composite image Y1.

[0208] In addition, when the operation control unit 153 changes the position of the virtual projector C4 to a position different from the second position C2, the operation control unit 153 may include, in the first composite image Y1 and the second composite image Y2, an image indicating that the position of the virtual projector C4 has been adjusted. The image indicating that the position of the virtual projector C4 has been adjusted is, for example, an image indicating a comment such as "Image after position adjustment." The image indicating that the position of the virtual projector C4 has been adjusted is not limited to an image indicating the comment "Image with position adjusted," and can be changed as appropriate.

[0209] A6: Summary of the first embodiment The display method and information processing device 1 according to the first embodiment include the following aspects.

[0210] The acquisition unit 151 acquires a target image H1 showing a target region TR including a wall E1. When the operation control unit 153 displays the sample image J1 on the touch panel 12, the operation control unit 153 displays on the touch panel 12 a first composite image Y1 obtained by superimposing the sample image J1, which is of a size that fits on the touch panel 12, on the target image H1, regardless of the position of the touch panel 12. The sample image J1 is an image of an image displayed on a virtual surface C3 viewed from a second position C2 in a situation in which an image is projected from a virtual projector C4 onto a virtual surface C3 located at a first position C1 in a virtual space VS. The first position C1 is a position in the virtual space VS that corresponds to the position of the wall E1 in the real space RS. The second position C2 is a position in the virtual space VS that corresponds to the position of the touch panel 12 in the real space RS.

[0211] According to this embodiment, the first composite image Y1, in which the sample image J1 that fits on the touch panel 12 is superimposed on the target image H1, is displayed on the touch panel 12 regardless of the position of the touch panel 12. Therefore, the user can see the first composite image Y1 including the sample image J1 that fits on the touch panel 12, regardless of the position of the touch panel 12. By seeing the first composite image Y1, the user can easily imagine the state in which the projector 2 projects the projection image F1. Therefore, a convenient information processing device 1 can be provided.

[0212] The operation control unit 153 adjusts the distance between the virtual surface C3 and the virtual projector C4, thereby adjusting the size of the sample image J1 to a size that fits on the touch panel 12. According to this embodiment, the size of the sample image J1 can be adjusted to a size that fits on the touch panel 12 without the user having to move the information processing device 1 in the real space RS in order to adjust the size of the sample image J1.

[0213] The operation control unit 153 adjusts the distance between the virtual plane C3 and the virtual projector C4 based on the distance between the touch panel 12 and the wall E1, thereby adjusting the size of the sample image J1 to a size that fits within the touch panel 12. According to this embodiment, the size of the sample image J1 can be adjusted to a size that fits within the touch panel 12 based on the distance between the touch panel 12 and the wall E1.

[0214] The operation control unit 153 adjusts the distance between the virtual plane C3 and the virtual projector C4 based on the distance between the touch panel 12 and the wall E1 and the angle of view C4a of the virtual projector C4, thereby adjusting the size of the sample image J1 to a size that fits within the touch panel 12. According to this embodiment, the size of the sample image J1 can be adjusted to a size that fits within the touch panel 12 based on the distance between the touch panel 12 and the wall E1 and the angle of view C4a of the virtual projector C4.

[0215] The first composite image Y1 includes a projector image L1, which is an image showing the projector 2. The projector image L1 is located in a portion of the first composite image Y1 that corresponds to the second position C2. According to this embodiment, the user can easily imagine the state in which the projector 2 projects the projection image F1 by looking at the state in which the projector shown in the projector image L1 projects the sample image J1.

[0216] B: Variation The following are examples of modified aspects of the above-described embodiment. Two or more aspects selected from the following examples may be combined as appropriate to the extent that they are not mutually inconsistent.

[0217] B1: First modified example In the first embodiment, the operation control unit 153 may adjust the size of the screen image v2 and the size of the sample image J1 to a size that fits within the touch panel 12 by adjusting the angle of view C4a of the virtual projector.

[0218] For example, when the angle of view C4a of the virtual projector C4 is equal to or larger than the angle of view C5a of the virtual camera C5, the operation control unit 153 changes the angle of view C4a of the virtual projector C4 to less than the angle of view C5a of the virtual camera C5, thereby adjusting the size of the screen image v2 and the size of the sample image J1 to a size that fits within the touch panel 12. Also, for example, the operation control unit 153 may change the value of the angle of view C4a included in the specifications of the virtual projector C4 to a value less than the angle of view C5a of the camera 11, i.e., a value less than the angle of view C5a of the virtual camera C5.

[0219] According to the first modification, the virtual projector C4 is not moved, Screen Image v 2 and the size of the sample image J1 are adjusted to fit on the touch panel 12. In addition, the process of calculating the position after the movement of the virtual projector C4 can be omitted.

[0220] The operation control unit 153 may use a method of using a model of a virtual projector having an angle of view less than the angle of view C5a of the camera 11 as the virtual projector C4, as a method of adjusting the angle of view C4a of the virtual projector C4. For example, the operation control unit 153 uses a method of changing the model of the virtual projector C4 from a first virtual projector having an angle of view equal to or greater than the angle of view C5a of the camera 11 to a second virtual projector having an angle of view less than the angle of view C5a of the camera 11. Specifically, the operation control unit 153 refers to the specifications of a plurality of models of projectors stored in the program P1, and selects a model of a projector having an angle of view less than the angle of view C5a of the camera 11 as the virtual projector C4. In this case, the size of the screen image v2 and the size of the sample image J1 can be adjusted to a size that fits within the touch panel 12 by changing the virtual projector C4.

[0221] The operation control unit 153 may display the name of the actual projector corresponding to the first virtual projector and the name of the actual projector corresponding to the second virtual projector on the touch panel 12. In this case, the user can easily recognize the projectors simulated by the information processing device 1.

[0222] The operation control unit 153 may use a method of adjusting the angle of view of a virtual projection lens of the virtual projector C4 as a method of adjusting the angle of view C4a of the virtual projector C4. In this case, the operation control unit 153 adjusts the size of the screen image v2 and the size of the sample image J1 to a size that fits within the touch panel 12 by adjusting the angle of view of the virtual projection lens.

[0223] For example, when the angle of view of the virtual projection lens is equal to or larger than the angle of view C5a of the virtual camera C5, the operation control unit 153 changes the angle of view of the virtual projection lens to less than the angle of view C5a of the virtual camera C5, thereby adjusting the size of the screen image v2 and the size of the sample image J1 to a size that fits within the touch panel 12. Also, for example, the operation control unit 153 may change the value of the angle of view of the virtual projection lens included in the specifications of the virtual projector C4 to a value less than the angle of view C5a of the camera 11, i.e., a value less than the angle of view C5a of the virtual camera C5.

[0224] In this case, without moving the virtual projector C4, the size of the screen image v2 and the size of the sample image J1 can be adjusted to a size that fits within the touch panel 12. Also, the process of calculating the position of the virtual projector C4 after it is moved can be omitted.

[0225] The operation control unit 153 may use a virtual projection lens having an angle of view less than the angle of view C5a of the camera 11 as the virtual projection lens of the virtual projector C4 as a method of adjusting the angle of view of the virtual projection lens. For example, the operation control unit 153 may use a method of changing the virtual projection lens from a first virtual projection lens having an angle of view equal to or greater than the angle of view C5a of the camera 11 to a second virtual projection lens having an angle of view less than the angle of view C5a of the camera 11. Also, for example, the operation control unit 153 may change the value of the angle of view of the virtual projection lens included in the specifications of the virtual projector C4 from a value equal to or greater than the angle of view C5a of the camera 11 to a value less than the angle of view C5a of the camera 11. In this case, the size of the screen image v2 and the size of the sample image J1 can be adjusted to a size that fits within the touch panel 12 by changing the virtual projection lens.

[0226] The operation control unit 153 may display the name of an actual projection lens corresponding to the first virtual projection lens and the name of an actual projection lens corresponding to the second virtual projection lens on the touch panel 12. In this case, the user can easily recognize the projection lens used in the simulation by the information processing device 1.

[0227] B2: Second variant In the first embodiment and the first modified example, the operation control unit 153 may include an additional image L3, which is an image based on the distance between the position of the virtual projector C4 and the second position C2, in at least one of the first composite image Y1 and the second composite image Y2. The additional image L3 is an example of an image indicating the distance between the position of the virtual projector C4 and the second position C2.

[0228] FIG. 33 is a diagram showing an example of the second composite image Y2 including the additional image L3. The additional image L3 shows a hand holding the projector shown in the projector image L1. Therefore, the user can check how the projected image F1 looks, with the sensation of moving the projector 2 in his / her hand. The additional image L3 may be different from the image showing the hand holding the projector shown in the projector image L1. For example, the additional image L3 may be an image showing the distance between the position of the virtual projector C4 and the second position C2. Specifically, the additional image L3 may be a scale line showing the distance or a line segment having a length according to the distance. When the scale line or line segment is used as the additional image L3, for example, the additional image L3 may be displayed between the position of the virtual projector C4 and the second position C2.

[0229] The operation control unit 153 determines the position of the added image L3 in the second composite image Y2 based on the distance between the position of the virtual projector C4 and the second position C2. For example, the operation control unit 153 increases the distance between the end of the added image L3 on the virtual projector C4 side or the screen image v2 side and the second position C2 in the second composite image Y2 in accordance with an increase in the distance between the position of the virtual projector C4 and the second position C2. Therefore, the added image L3 makes it easier for the user to recognize the relationship between the position of the projector indicated by the projector image L1 and the position of the screen image v2.

[0230] B3: Third variant In the first embodiment and the first to second modified examples, when the operation control unit 153 receives an input instructing to switch the display while the touch panel 12 is displaying the first composite image Y1 or the second composite image Y2, the operation control unit 153 may execute the operations shown in FIG. 11 except for step S209.

[0231] In this case, the operation control unit 153 causes the touch panel 12 to display a simulation image. The simulation image is an example of a second simulation image. The simulation image is an image obtained by superimposing a display image on a target image H1. The display image is an example of a second display image. The display image is an image obtained by viewing an image displayed on the virtual plane C3 from the second position C2 in a situation in which an image is projected onto the virtual plane C3 located at the first position C1 from a virtual projector C4 whose relative position with respect to the second position C2 is fixed. In this case, the size of the display image is not adjusted. The simulation image may be an image obtained by superimposing a projector image L1 and a path image L2 on the target image H1 in addition to the display image.

[0232] According to the third modified example, when the angle of view C4a of the virtual projector C4 is equal to or larger than the angle of view C5a of the virtual camera C5, the simulation image shows a state in which the displayed image does not fit within the touch panel 12. For this reason, by looking at the simulation image, the user can easily recognize that the first composite image Y1 and the second composite image Y2 are adjusted images.

[0233] B4: Fourth variant In the first embodiment and the first to third modified examples, when the touch panel 12 receives an instruction from the user to fix the virtual projector C4 after displaying at least one of the first composite image Y1 and the second composite image Y2 on the touch panel 12, the operation control unit 153 may fix the position of the virtual projector C4 in the virtual space VS.

[0234] For example, the operation control unit 153 fixes the virtual projector C4 at the position of the virtual projector C4 when the touch panel 12 receives an instruction from the user to fix the virtual projector C4.

[0235] In this case, the operation control unit 153 displays the superimposed image on the touch panel 12. The superimposed image is also referred to as a third simulation image. The superimposed image is an image obtained by superimposing the third virtual image on the target image H1. The third virtual image is also referred to as a third display image. The third virtual image is an image displayed on the virtual plane C3 in a situation where the virtual projector C4 whose position is fixed in the virtual space VS projects an image onto the virtual plane C3, as viewed from the second position C2. The virtual projector C4 whose position is fixed in the virtual space VS means the virtual projector C4 whose absolute position is fixed in the virtual space VS. Note that, in a situation where the superimposed image is displayed, if the information processing device 1 is again located at the position of the information processing device 1 when the touch panel 12 receives an instruction from the user to fix the virtual projector C4, the operation control unit 153 may fix the virtual projector C4 to the second position C2 again.

[0236] According to the fourth modified example, in addition to the first composite image Y1 and the second composite image Y2, a superimposed image is displayed, which further assists the user in checking the installation position of the projector 2.

[0237] B5: Fifth variant In the first embodiment and the first to fourth modified examples, the camera 11, the touch panel 12, and the processing device 15 may be separate from each other. In the first embodiment and the first to fourth modified examples, the camera 11 and the touch panel 12 may be separate from the information processing device 1. In the first embodiment and the first to fourth modified examples, the camera 11 may be separate from the information processing device 1. In the first embodiment and the first to fourth modified examples, the touch panel 12 may be separate from the information processing device 1. In the first embodiment and the first to fourth modified examples, the display 121 and the input device 122 may be separate from each other. [Explanation of symbols]

[0238] 1...information processing device, 2...projector, 11...camera, 12...touch panel, 13...motion sensor, 14...storage device, 15...processing device, 111...imaging lens, 112...image sensor, 121...display, 122...input device, 151...acquisition unit, 152...recognition unit, 153...operation control unit.

Claims

1. a display that displays a target area in a real space including a surface; acquiring a target image; a virtual surface located at a first position corresponding to a position of the surface in the real space; In a virtual space having a virtual surface and a virtual projector, The image projected onto the virtual surface from the display is adjusted to correspond to the position of the display in the real space. When a first display image, which is an image seen from a second position, is displayed on the display, The first display image is a size that fits on the display regardless of the position of the display. A first simulation image in which the above-mentioned is superimposed on the target image is displayed on the display. thing, Including, how to display.

2. The distance between the virtual surface and the virtual projector is adjusted to change the first display. Resizing the image to fit on the display; The display method of claim 1 , further comprising:

3. The virtual surface and the virtual projector are connected based on the distance between the display and the surface. By adjusting the distance, the size of the first display image can be adjusted so that it fits on the display. Adjust the size to fit your needs. The display method of claim 1 further comprising:

4. Based on the distance between the display and the surface and the angle of view of the virtual projector, By adjusting the distance between the virtual surface and the virtual projector, the first display Resizing the image to fit on the display; The display method of claim 1 further comprising:

5. The size of the first display image is adjusted by adjusting the angle of view of the virtual projector. , adjusting the size to fit the display; The display method of claim 1 , further comprising:

6. capturing the target image includes capturing the target image using a camera; The adjusting step may include adjusting the virtual projector to have an angle of view that is less than the angle of view of the camera. By using a virtual projector having a model having the above-mentioned [0033] The display method according to claim 5.

7. the virtual projector includes a virtual projection lens that projects an image onto the virtual surface; By adjusting the angle of view of the virtual projection lens, the size of the first display image can be adjusted to the size of the front Adjust the size so that it fits the display. The display method of claim 1 , further comprising:

8. capturing the target image includes capturing the target image using a camera; The adjusting step includes adjusting the virtual projection lens to have an angle of view less than the angle of view of the camera. and adjusting the field angle of the virtual projection lens by using a virtual projection lens having a field angle of the virtual projection lens. include, The display method according to claim 7.

9. The first simulation image is a projector image that shows a projector. Including the statue, The projector image is a first simulation image. The virtual space is located at a portion corresponding to the position of the vector in the virtual space. The display method according to any one of claims 1 to 8.

10. The first simulation image is a projection image of the virtual projector and the second position. Includes an image showing distances, The display method according to any one of claims 1 to 8.

11. The position of the image showing the distance in the first simulation image is determining the distance between the vector position and the second position; The display method of claim 10 further comprising:

12. The first simulation image is a projector image which is an image showing a projector. Including the statue, The projector image is a first simulation image. the location of the vector in the virtual space; The image showing the distance shows a hand holding the projector shown in the projector image. show, The display method according to claim 10 or 11.

13. When the first simulation image is displayed on the display, the display is switched. When an input instructing a change is received, the relative position with respect to the second position is fixed. an image projected onto the virtual surface from the virtual projector as viewed from the second position; A second simulation image is generated by superimposing a second display image on the target image. Showing up in play, The display method according to claim 1 , further comprising:

14. A camera and A display and At least one processing unit, The at least one processing unit A target area including a surface and the display. acquiring an image of a target using the camera; a virtual surface located at a first position corresponding to a position of the surface in the real space; In a virtual space having a virtual surface and a virtual projector, The image projected onto the virtual surface from the display is adjusted to correspond to the position of the display in the real space. When a first display image, which is an image seen from a second position, is displayed on the display, The first display image is a size that fits on the display regardless of the position of the display. A first simulation image in which the above-mentioned is superimposed on the target image is displayed on the display. And, A display system that performs

Images