Display method and display system

Abstract

The present invention provides a display method and a display system, which display a simulated image that can easily imagine the actual visibility of a projected image. The display method includes: obtaining an object image, where the object image represents an object area including a surface; displaying a simulated image on a display, where the simulated image is obtained by overlapping a display image corresponding to an image projected by a projector onto the surface with the object image; displaying the object image included in the simulated image on the display with a brightness based on first information specifying the brightness of the object area; and displaying the display image included in the simulated image on the display with a brightness based on second information specifying the brightness of the display image.

Description

Technical Field

[0001] The present invention relates to a display method and a display system. Background Art

[0002] Patent Document 1 discloses an information processing apparatus that displays a simulated image related to the appearance of a projected image projected from a projector. The simulated image represents an image of a room in which the projector is located and the projected image. The information processing apparatus simulates the brightness of the projected image by changing the transmittance of the projected image.

[0003] Patent Document 1: International Publication No. 2017 / 179272

[0004] The visibility of a projected image is sometimes not determined solely by the brightness of the projected image. Therefore, even if a user views the simulated image displayed by the information processing apparatus described in Patent Document 1, it is difficult to imagine the actual visibility of the projected image. Summary of the Invention

[0005] One aspect of the display method of the present invention includes: obtaining an object image representing an object area including a surface; displaying a simulated image on a display, the simulated image being obtained by overlapping a display image corresponding to an image projected by a projector onto the surface with the object image; displaying the object image included in the simulated image on the display with a brightness based on first information specifying the brightness of the object area; and displaying the display image included in the simulated image on the display with a brightness based on second information specifying the brightness of the display image.

[0006] One aspect of the display system of the present invention includes: a camera; a display; and at least one processing unit that performs: obtaining an object image using the camera, the object image representing an object area including a surface; causing the display to display a simulated image obtained by overlapping a display image corresponding to an image projected by a projector onto the surface with the object image; causing the object image included in the simulated image to be displayed on the display with a brightness based on first information specifying the brightness of the object area; and causing the display image included in the simulated image to be displayed on the display with a brightness based on second information specifying the brightness of the display image. Brief Description of the Drawings

[0007] Figure 1 It is a diagram showing the information processing apparatus 1.

[0008] Figure 2 It is a diagram showing the front surface 1a of the information processing apparatus 1.

[0009] Figure 3 This is a diagram showing the back surface 1b of the information processing apparatus 1.

[0010] Figure 4 This is a diagram showing an example of the object image H1.

[0011] Figure 5 This is a diagram showing an example of the information processing apparatus 1.

[0012] Figure 6 This is a diagram showing an example of the virtual space VS.

[0013] Figure 7 This is a flowchart for explaining the recognition of the wall E1.

[0014] Figure 8 This is a flowchart for explaining the display of the first simulation image G1.

[0015] Figure 9 This is a flowchart for explaining the operation of changing the brightness of the first simulation image G1.

[0016] Figure 10 This is a diagram showing an example of the icon i displayed on the touch panel 12.

[0017] Figure 11 This is a diagram showing an example of the first guide image t1.

[0018] Figure 12 This is a diagram showing an example of the information processing apparatus 1 displaying the object image H1.

[0019] Figure 13 This is a diagram showing an example of the image u1.

[0020] Figure 14 This is a diagram showing an example of the image u2.

[0021] Figure 15 This is a diagram showing an example of the image u3.

[0022] Figure 16 This is a diagram showing the state where the user shakes the information processing apparatus 1.

[0023] Figure 17 This is a diagram showing an example of the image u4.

[0024] Figure 18 This is a diagram showing another example of the image u4.

[0025] Figure 19 This is a diagram showing yet another example of the image u4.

[0026] Figure 20 This is a diagram showing yet another example of the image u4.

[0027] Figure 21 It is a diagram showing another example of the image u4.

[0028] Figure 22 It is a diagram showing an example of the image u5.

[0029] Figure 23 It is a diagram showing another example of the image u5.

[0030] Figure 24 It is a diagram showing another example of the image u5.

[0031] Figure 25 It is a diagram showing another example of the image u5.

[0032] Figure 26 It is a diagram showing another example of the image u5.

[0033] Figure 27 It is a diagram showing an example of the menu image v4.

[0034] Figure 28 It is a diagram showing an example of the candidate v8 of the image.

[0035] Figure 29 It is a diagram showing an example of the image u6.

[0036] Figure 30 It is a diagram showing another example of the image u6.

[0037] Figure 31 It is a diagram showing an example of the image u7.

[0038] Figure 32 It is a diagram showing another example of the image u7.

[0039] Figure 33 It is a diagram showing an example of the mask v16.

[0040] Figure 34 It is a diagram showing an example of the image u8.

[0041] Explanation of reference numerals

[0042] 1: Information processing device; 2: Projector; 11: Camera; 12: Touch panel; 13: Motion sensor; 14: Storage device; 15: Processing device; 111: Shooting lens; 112: Image sensor; 121: Display; 122: Input device; 151: Acquisition unit; 152: Recognition unit; 153: Action control unit. Detailed implementation manners

[0043] A: First implementation manner

[0044] A1: Outline of the information processing device 1

[0045] Figure 1 This is a diagram showing the information processing device 1. The information processing device 1 is a smartphone. The information processing device 1 is not limited to a smartphone, and for example, it can also be a tablet computer with a camera, a notebook PC (Personal Computer) with a camera, or a notebook PC connected to a camera. The information processing device 1 is an example of a display system. The information processing device 1 is located in the real space RS.

[0046] In addition to the information processing device 1, the real space RS also includes a projector 2, a wall E1, a ceiling E2, and a floor E3. The position of the projector 2 in the real space RS is not limited to Figure 1 the position shown and can be appropriately changed.

[0047] The wall E1 is a vertical plane. The wall E1 is not limited to a vertical plane and can be any plane that intersects with the horizontal plane. The wall E1 is the inner wall of a building. The wall E1 is not limited to the inner wall of a building and can also be, for example, the outer wall of a building. At least a part of the wall E1 is an example of a surface. The surface is not limited to at least a part of the wall E1 and can also be, for example, at least a part of the ceiling E2, at least a part of the floor E3, a screen, a whiteboard, or a door. The surface is included in the object area TR.

[0048] The object area TR is included in the real space RS. The position of the object area TR in the real space RS is not limited to Figure 1 the position shown and can be appropriately changed.

[0049] The projector 2 projects a projection image F1 onto the wall E1 using light. The information processing device 1 displays a first simulated image G1 related to the appearance of the projection image F1.

[0050] The information processing device 1 includes a front surface 1a, a back surface 1b, a camera 11, and a touch panel 12. Figure 2 This is a diagram showing the front surface 1a of the information processing device 1. Figure 3 This is a diagram showing the back surface 1b of the information processing device 1.

[0051] The camera 11 is located on the back surface 1b of the information processing device 1. The camera 11 captures the shooting area. The shooting area of the camera 11 moves corresponding to the movement of the information processing device 1.

[0052] The imaging area of the camera 11 is used as the object area TR. Therefore, the object area TR moves corresponding to the movement of the information processing device 1. The camera 11 generates an object image H1 representing the object area TR by imaging the object area TR in a state where the projection image F1 is not projected by the projector 2. The object image H1 representing the object area TR is an image representing the objects existing in the object area TR.

[0053] Figure 4 It is a diagram showing an example of the object image H1. The object image H1 represents the wall E1, the ceiling E2, and the floor E3.

[0054] As Figure 2 shown, the touch panel 12 is located on the front surface 1a of the information processing device 1. The touch panel 12 is an example of a display. The touch panel 12 displays the first simulated image G1. The first simulated image G1 is an example of a simulated image.

[0055] The first simulated image G1 is an image obtained by overlapping the sample image J1 on the object image H1. The sample image J1 is an example of a displayed image. The aspect ratio of the sample image J1 is equal to the aspect ratio of the projection image F1. The sample image J1 is an image corresponding to the projection image F1. The sample image J1 represents, for example, the projection image F1. The sample image J1 may also be an image different from the projection image F1, for example, an image obtained by changing the color of the projection image F1 to monochrome. The sample image J1 has a preset transmittance. The transmittance of the sample image J1 can also be changed.

[0056] The first simulated image G1 includes the projector image L1. The projector image L1 is an image representing the projector. The shape of the projector represented by the projector image L1 is the same as the shape of the projector 2. The shape of the projector represented by the projector image L1 may also be different from the shape of the projector 2. The projector image L1 has a preset transmittance. The transmittance of the projector image L1 can also be changed.

[0057] The first simulated image G1 also includes the path image L2. The path image L2 is an image representing the path of the light used when the projector 2 projects the projection image F1. The path image L2 is also an image representing the path of the light virtually used when the virtual projector C4 corresponding to the projector 2 projects an image corresponding to the projection image F1. The virtual projector C4 will be described later. The path image L2 has a preset transmittance. The transmittance of the path image L2 can also be changed.

[0058] The first simulated image G1 may not include at least one of the projector image L1 and the path image L2.

[0059] A2: An example of the information processing device 1

[0060] Figure 5 This is a diagram showing an example of the information processing apparatus 1. The information processing apparatus 1 includes a camera 11, a touch panel 12, a motion sensor 13, a storage device 14, and a processing device 15.

[0061] The camera 11 includes a photographing lens 111 and an image sensor 112.

[0062] The photographing lens 111 forms an optical image on the image sensor 112. The photographing lens 111 images an object image H1 representing the object area TR on the image sensor 112.

[0063] The image sensor 112 is a CCD (Charge Coupled Device) image sensor. The image sensor 112 is not limited to a CCD image sensor, and may be, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The image sensor 112 generates photographing data k based on the optical image formed on the image sensor 112. For example, the image sensor 112 generates photographing data kt representing the object image H1 based on the object image H1 formed by the photographing lens 111. The photographing data kt is an example of the photographing data k.

[0064] The touch panel 12 includes a display 121 and an input device 122. The display 121 displays various images. The input device 122 receives various instructions and the like.

[0065] The motion sensor 13 includes an acceleration sensor and a gyro sensor. The motion sensor 13 detects the motion of the information processing apparatus 1. For example, the motion sensor 13 detects the motion of the information processing apparatus 1 moved by the user. The motion of the information processing apparatus 1 is represented by at least the moving distance of the information processing apparatus 1, the amount of rotation of the information processing apparatus 1, and the orientation of the information processing apparatus 1. The motion sensor 13 generates motion data m representing the motion of the information processing apparatus 1.

[0066] The storage device 14 is a recording medium readable by the processing device 15. The storage device 14 includes, for example, a non-volatile memory and a volatile memory. The non-volatile memory is, for example, a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read Only Memory). The volatile memory is, for example, a RAM (Random Access Memory). The storage device 14 stores the program P1 and various data. The program P1 is, for example, an application program. The program P1 is provided to the information processing device 1 from a server (not shown). The program P1 may also be pre-stored in the storage device 14.

[0067] The processing device 15 is composed of one or more CPUs (Central Processing Unit). One or more CPUs are an example of one or more processors. A processor is an example of a processing unit. A CPU and a processor are each an example of a computer.

[0068] The processing device 15 reads the program P1 from the storage device 14. By executing the program P1, the processing device 15 functions as an acquisition unit 151, an identification unit 152, and an operation control unit 153.

[0069] The processing device 15 may function as the acquisition unit 151 and the operation control unit 153 by executing the program P1, and function as the identification unit 152 by executing a program other than the program P1. In this case, a program different from the program P1 is stored in the storage device 14, and the processing device 15 reads the program different from the program P1 from the storage device 14.

[0070] The acquisition unit 151, the identification unit 152, and the operation control unit 153 may also be implemented by circuits such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array).

[0071] The acquisition unit 151 acquires the object image H1 representing the object area TR. For example, the acquisition unit 151 acquires the object image H1 by acquiring the shooting data kt representing the object image H1 from the camera 11. In addition, the acquisition unit 151 acquires the motion data m from the motion sensor 13.

[0072] The recognition unit 152 acquires the shooting data kt and the motion data m from the acquisition unit 151. The recognition unit 152 performs three-dimensional measurement on the object existing in the object area TR based on the shooting data kt and the motion data m.

[0073] In the situation where the information processing device 1 moves from the first point to the second point while the camera 11 shoots the wall E1, the recognition unit 152 performs three-dimensional measurement as follows.

[0074] The recognition unit 152 acquires the motion data ms from the acquisition unit 151. The motion data ms is the motion data m generated by the motion sensor 13 in the situation where the information processing device 1 moves from the first point to the second point while the camera 11 shoots the wall E1. The recognition unit 152 determines the distance from the first point to the second point as the baseline length based on the motion data ms. The baseline length is also referred to as the length of the baseline.

[0075] The recognition unit 152 acquires the first shooting data k1 and the second shooting data k2 from the acquisition unit 151. The first shooting data k1 is the shooting data kt generated by the camera 11 when the information processing device 1 is at the first point. The second shooting data k2 is the shooting data kt generated by the camera 11 when the information processing device 1 is at the second point. The first shooting data k1 and the second shooting data k2 each represent at least the wall E1.

[0076] The recognition unit 152 performs three-dimensional measurement by performing triangulation using the baseline length, the first shooting data k1, and the second shooting data k2.

[0077] The result of the three-dimensional measurement represents the shape of the object existing in the object area TR using three-dimensional coordinates. The position of the camera 11 in the real space RS is used as the reference position for the three-dimensional measurement. The recognition unit 152 identifies the wall E1 based on the result of the three-dimensional measurement. For example, the recognition unit 152 identifies the vertical plane as the wall E1 based on the result of the three-dimensional measurement. The recognition unit 152 determines the distance n from the information processing device 1 to the wall E1 based on the result of the three-dimensional measurement.

[0078] The motion control unit 153 controls the motion of the information processing device 1. The motion control unit 153 causes the touch panel 12 to display the image represented by the image data r by providing the image data r representing the image to the touch panel 12.

[0079] The motion control unit 153 causes the touch panel 12 to display the first simulated image G1. The motion control unit 153 generates simulated image data r1 based on the result of three-dimensional measurement and the captured data kt. The simulated image data r1 is an example of the image data r. The simulated image data r1 represents the first simulated image G1.

[0080] For example, the motion control unit 153 determines the size q of the sample image J1 based on the distance n determined according to the result of three-dimensional measurement. The distance n is the distance from the information processing device 1 to the wall E1. The size q of the sample image J1 represents the horizontal length and the vertical length of the sample image J1. The motion control unit 153 increases the size q, for example, corresponding to an increase in the distance n. The motion control unit 153 determines the correspondence relationship between the distance n and the size q according to the field of view angle of the projector 2. The field of view angle of the projector 2 is described in the program P1. Therefore, the motion control unit 153 pre-identifies the field of view angle of the projector 2.

[0081] The motion control unit 153 determines the image obtained by overlapping the sample image J1 of size q, the projector image L1, and the path image L2 on the object image H1 as the first simulated image G1.

[0082] The motion control unit 153 determines the sample image J1 using the three-dimensional virtual space VS. Figure 6 It is a diagram showing an example of the virtual space VS.

[0083] The motion control unit 153 reproduces the configuration of the objects in the real space RS by using the virtual space VS.

[0084] The motion control unit 153 sets the first position C1 in the virtual space VS by using the result of three-dimensional measurement for the wall E1. The first position C1 in the virtual space VS corresponds to the position of the wall E1 in the real space RS.

[0085] The motion control unit 153 determines the shape of the virtual plane C3 based on the result of three-dimensional measurement for the wall E1. The virtual plane C3 has the same shape as the wall E1. The virtual plane C3 is the plane corresponding to the wall E1. The motion control unit 153 arranges the virtual plane C3 at the first position C1.

[0086] The motion control unit 153 sets the second position C2 in the virtual space VS based on the position of the camera 11 in the real space RS. The second position C2 in the virtual space VS corresponds to the position of the camera 11 in the real space RS. The camera 11 is located in the information processing device 1 together with the touch panel 12. Therefore, the second position C2 in the virtual space VS corresponds to the position of the camera 11 in the real space RS and also corresponds to the position of the touch panel 12 in the real space RS.

[0087] The motion control unit 153 arranges the virtual projector C4 at the second position C2. Accordingly, in the virtual space VS, the relative position of the virtual projector C4 with respect to the second position C2 is fixed. The state where the relative position of the virtual projector C4 with respect to the second position C2 is fixed in the virtual space VS is not limited to the state where the virtual projector C4 is located at the second position C2. For example, in the virtual space VS, the relative position of the virtual projector C4 with respect to the second position C2 may also be fixed in a state where the virtual projector C4 is located at a position different from the second position C2.

[0088] The second position C2 changes corresponding to the change in the position of the touch panel 12 in the real space RS. Accordingly, in a situation where the relative position of the virtual projector C4 with respect to the second position C2 is fixed in the virtual space VS, when the position of the touch panel 12 in the real space RS changes, the position of the virtual projector C4 in the virtual space VS changes.

[0089] The virtual projector C4 is a projector corresponding to the projector 2. The specifications of the virtual projector C4 are the same as those of the projector 2. The specifications of the projector 2 are described in the program P1. Accordingly, the motion control unit 153 identifies the specifications of the projector 2 in advance.

[0090] The motion control unit 153 makes the orientation of the optical axis of the projection lens of the virtual projector C4 with respect to the virtual plane C3 coincide with the orientation of the optical axis of the imaging lens 111 with respect to the wall E1. In addition, the motion control unit 153 determines the orientation of the optical axis of the imaging lens 111 with respect to the wall E1 based on the recognition result of the wall E1 and the motion data m.

[0091] The motion control unit 153 arranges the screen image v2 on the virtual plane C3. The screen image v2 is an image obtained by observing, from the second position C2, the image displayed on the virtual plane C3 when the virtual projector C4 projects an image onto the virtual plane C3. The screen image v2 is another example of a displayed image. The screen image v2 is an image representing the area where the sample image J1 is displayed. The screen image v2 functions as a screen for the sample image J1. The size of the screen image v2 is equal to the size q of the sample image J1. The motion control unit 153 determines the size of the screen image v2 by the same method as the method for determining the size q of the sample image J1. The screen image v2 is an image corresponding to the projected image F1.

[0092] The position of the screen image v2 on the virtual plane C3 is fixed according to an instruction from the user. During the period before receiving an instruction from the user, the motion control unit 153 determines the position of the screen image v2 on the virtual plane C3 based on the intersection position of the virtual plane C3 and the optical axis of the projection lens of the virtual projector C4. For example, the motion control unit 153 makes the center position of the screen image v2 on the virtual plane C3 coincide with the intersection position of the virtual plane C3 and the optical axis of the projection lens of the virtual projector C4. The center position of the screen image v2 is, for example, the intersection position of the diagonals in the screen image v2.

[0093] The motion control unit 153 determines the first simulated image G1 by changing the screen image v2 to the sample image J1.

[0094] The sample image J1 is an image obtained by observing, from the second position C2, the image displayed on the virtual plane C3 in a situation where an image is projected from the virtual projector C4 whose relative position with respect to the second position C2 is fixed to the virtual plane C3.

[0095] In the virtual space VS, when the position of the touch panel 12 changes in the real space RS in a situation where the relative position of the virtual projector C4 with respect to the second position C2 is fixed, the position of the viewpoint for observing the image displayed on the virtual plane C3 changes in addition to the position of the virtual projector C4 in the virtual space VS.

[0096] The motion control unit 153 generates simulation image data r1 representing the first simulated image G1.

[0097] The motion control unit 153 causes the touch panel 12 to display the first simulated image G1 by providing the simulation image data r1 to the touch panel 12.

[0098] The motion control unit 153 causes the touch panel 12 to display the object image H1 included in the first simulated image G1 and the projector image L1 included in the first simulated image G1 with the brightness of the first information based on the brightness of the specified object area TR. For example, the motion control unit 153 changes the brightness of the object image H1 included in the first simulated image G1 and the brightness of the projector image L1 included in the first simulated image G1 to the brightness based on the first information. Additionally, the brightness of the projector image L1 may also be fixed.

[0099] The motion control unit 153 causes the touch panel 12 to display the sample image J1 included in the first simulated image G1 and the path image L2 included in the first simulated image G1 with the brightness of the second information based on the brightness of the specified sample image J1. For example, the motion control unit 153 changes the brightness of the sample image J1 included in the first simulated image G1 and the brightness of the path image L2 included in the first simulated image G1 to the brightness based on the second information. The second information is also information specifying the brightness of the light used when the virtual projector C4 projects an image. In addition, the brightness of the path image L2 may be fixed.

[0100] The first information is input to the touch panel 12 by the user, for example. The second information is input to the touch panel 12 by the user, for example. The motion control unit 153 may generate the second information based on the first information input to the touch panel 12. The motion control unit 153 may also generate the first information based on the second information input to the touch panel 12.

[0101] A3: Identification of the wall E1

[0102] Figure 7 It is a flowchart for explaining the operation of identifying the wall E1.

[0103] When the touch panel 12 receives a start instruction from the user, in step S101, the processing device 15 starts the execution of the program P1 as an application program.

[0104] Next, in step S102, the motion control unit 153 causes the camera 11 to start photographing the object area TR. The camera 11 generates photographing data kt by photographing the object area TR.

[0105] Next, in step S103, the motion control unit 153 causes the motion sensor 13 to operate. The motion sensor 13 generates motion data m.

[0106] Next, in step S104, the acquisition unit 151 starts acquiring the photographing data kt and the motion data m.

[0107] Next, in step S105, the motion control unit 153 causes the identification unit 152 to identify the wall E1.

[0108] In step S105, the identification unit 152 performs three-dimensional measurement on the object existing in the object area TR based on the photographing data kt and the motion data m acquired by the acquisition unit 151 in the scanning state.

[0109] The scanning state refers to a state where the information processing device 1 is moved from the first point to the second point while the wall E1 is photographed by the camera 11. The first point is, for example, the position of the information processing device 1 at the start time of the scanning state. The second point is, for example, the position of the information processing device 1 at the end time of the scanning state. The photographed data kt acquired by the acquisition unit 151 in the scanning state is the first photographed data k1 and the second photographed data k2. The first photographed data k1 is the photographed data kt generated by the camera 11 when the information processing device 1 is at the first point. The second photographed data k2 is the photographed data kt generated by the camera 11 when the information processing device 1 is at the second point. The motion data m acquired by the acquisition unit 151 in the scanning state is the motion data ms. The motion data ms is the motion data m generated by the motion sensor 13 in a state where the information processing device 1 is moved from the first point to the second point while the camera 11 photographs the wall E1.

[0110] The recognition unit 152 determines the distance from the first point to the second point as the baseline length based on the motion data ms. The recognition unit 152 performs three-dimensional measurement by executing triangulation using the baseline length, the first photographed data k1, and the second photographed data k2.

[0111] Next, the recognition unit 152 recognizes the wall E1 based on the result of the three-dimensional measurement. For example, the recognition unit 152 recognizes a vertical plane as the wall E1 based on the result of the three-dimensional measurement.

[0112] A4: Display of the first simulated image G1

[0113] Figure 8 It is a flowchart for explaining the operation of displaying the first simulated image G1. Figure 8 The operation shown is executed in a state where the wall E1 is recognized.

[0114] In step S201, the motion control unit 153 causes the recognition unit 152 to determine the distance n from the information processing device 1 to the wall E1.

[0115] In step S201, the recognition unit 152 first acquires the motion data m from the acquisition unit 151. Next, the recognition unit 152 determines the position of the information processing device 1 in the real space RS, that is, the position of the camera 11 in the real space RS, based on the motion data m. Next, the recognition unit 152 determines the distance n from the information processing device 1 to the wall E1 based on the result of the three-dimensional measurement and the position of the information processing device 1 in the real space RS.

[0116] Next, in step S202, the motion control unit 153 generates the virtual space VS.

[0117] Next, in step S203, the motion control unit 153 sets the first position C1 and the second position C2 in the virtual space VS.

[0118] In step S203, the motion control unit 153 first sets the first position C1 in the virtual space VS by using the three-dimensional measurement result for the wall E1. The first position C1 in the virtual space VS corresponds to the position of the wall E1 in the real space RS. Next, the motion control unit 153 sets the second position C2 in the virtual space VS by using the position of the camera 11 in the real space RS. The second position C2 in the virtual space VS corresponds to the position of the camera 11 in the real space RS.

[0119] Next, in step S204, the motion control unit 153 configures the virtual plane C3 in the virtual space VS.

[0120] In step S204, the motion control unit 153 first makes the shape of the virtual plane C3 coincide with the shape of the wall E1 based on the result of the three-dimensional measurement for the wall E1. Next, the motion control unit 153 configures the virtual plane C3 at the first position C1.

[0121] Next, in step S205, the motion control unit 153 configures the virtual projector C4 at the second position C2.

[0122] In step S205, by configuring the virtual projector C4 at the second position C2, the motion control unit 153 fixes the relative position of the virtual projector C4 with respect to the second position C2. Next, the motion control unit 153 determines the orientation of the optical axis of the photographing lens 111 with respect to the wall E1 based on the recognition result of the wall E1 and the motion data m. Next, the motion control unit 153 makes the orientation of the optical axis of the projection lens of the virtual projector C4 with respect to the virtual plane C3 coincide with the orientation of the optical axis of the photographing lens 111 with respect to the wall E1.

[0123] Next, in step S206, the motion control unit 153 configures the screen image v2 on the virtual plane C3.

[0124] In step S206, the motion control unit 153 aligns the center position of the screen image v2 in the virtual plane C3 with the intersection position of the virtual plane C3 and the optical axis of the projection lens of the virtual projector C4. Additionally, the center position of the screen image v2 on the virtual plane C3 is not limited to the intersection position of the virtual plane C3 and the optical axis of the projection lens of the virtual projector C4, as long as it is a position based on this intersection position. Next, the motion control unit 153 determines the size of the screen image v2 based on the determination result of the distance n. The motion control unit 153 increases the size of the screen image v2 corresponding to the increase in the distance n. The motion control unit 153 determines the correspondence relationship between the distance n and the size of the screen image v2 according to the field of view angle of the projector 2. Next, the motion control unit 153 sets the path of the projection light from the virtual projector C4 towards the screen image v2 in the virtual space VS. Next, when the touch panel 12 receives a position setting instruction from the user, the motion control unit 153 fixes the screen image v2 at the position of the screen image v2 when the position setting instruction is received.

[0125] Next, in step S207, the motion control unit 153 determines the original image of the sample image J1. In step S207, first, the motion control unit 153 determines the image displayed on the screen image v2 of the virtual plane C3 in the situation where an image is projected from the virtual projector C4 fixed at the relative position with respect to the second position C2 to the screen image v2 of the virtual plane C3 as the original image of the sample image J1. Additionally, the size of the original image of the sample image J1 is equal to the size of the screen image v2.

[0126] Next, in step S208, the motion control unit 153 determines the first simulation image G1.

[0127] In step S208, the motion control unit 153 first changes the screen image v2 to the original image of the sample image J1 in the virtual space VS. Next, the motion control unit 153 sets a virtual camera having the same specifications as the specifications of the camera 11 at the second position C2. The optical axis position of the shooting lens of the virtual camera coincides with the optical axis position of the projection lens of the virtual projector C4.

[0128] Next, the motion control unit 153 retains the original image of the sample image J1, the virtual projector C4, and the path of the projection light from the virtual projector C4 towards the original image of the sample image J1 in the virtual space VS, and deletes the virtual plane C3 from the virtual space VS.

[0129] Next, the motion control unit 153 determines the image obtained when the virtual camera performs shooting as the first image.

[0130] The first image is transmissive. The first image includes an image obtained by observing the original image of the sample image J1 from the second position C2. In the first image, the image obtained by observing the original image of the sample image J1 from the second position C2 becomes the sample image J1.

[0131] The first image also includes an image representing the virtual projector C4. In the first image, the image representing the virtual projector C4 becomes an example of the projector image L1.

[0132] The first image also includes an image representing the path of the projection light from the virtual projector C4 toward the original image of the sample image J1. In the first image, the image representing the path of the projection light from the virtual projector C4 toward the original image of the sample image J1 becomes an example of the path image L2.

[0133] Next, the motion control unit 153 determines the first simulated image G1 by overlapping the first image on the object image H1. In addition, the motion control unit 153 determines the brightness of the sample image J1 in the first simulated image G1 based on the first default value that specifies the brightness at the start of the display of the sample image J1. The motion control unit 153 determines the brightness of the object image H1 in the first simulated image G1 based on the second default value representing the brightness at the start of the display of the object image H1. The motion control unit 153 determines the brightness of the projector image L1 in the first simulated image G1 based on the third default value representing the brightness at the start of the display of the projector image L1. The motion control unit 153 determines the brightness of the path image L2 in the first simulated image G1 based on the fourth default value representing the brightness at the start of the display of the path image L2.

[0134] Next, in step S209, the motion control unit 153 generates simulated image data r1 representing the first simulated image G1.

[0135] Next, in step S210, the motion control unit 153 causes the touch panel 12 to display the first simulated image G1 by providing the simulated image data r1 to the touch panel 12.

[0136] In this way, when the relative position of the virtual projector C4 with respect to the second position C2 in the virtual space VS is fixed, the motion control unit 153 displays the first simulated image G1 obtained by overlapping the sample image J1 on the object image H1 on the touch panel 12. The sample image J1 is an image obtained by observing an image displayed on the virtual plane C3 in a state where the virtual projector C4 that projects an image onto the virtual plane C3 is fixed in the relative position with respect to the second position C2 from the second position C2.

[0137] A5: Operation of changing the brightness of the first simulated image G1

[0138] The visibility of the projected image F1 projected from the projector 2 depends on the brightness of the projected image F1 and the brightness of the object area TR. For example, when the object area TR is darkened while maintaining the brightness of the projected image F1, the visibility of the projected image F1 is improved. The visibility of the projected image F1 is represented as the visibility of the sample image J1 in the first simulated image G1. Therefore, in the first simulated image G1, it is possible to adjust both the brightness of the sample image J1 and the brightness of the object image H1 so that the visibility of the projected image F1 can be confirmed using the first simulated image G1.

[0139] Figure 9 is a flowchart for explaining the operation of changing the brightness of the first simulated image G1. Figure 9 The operation shown is repeated in the situation where the first simulated image G1 is displayed on the touch panel 12.

[0140] In the situation where the first simulated image G1 is displayed on the touch panel 12, the operation control unit 153 determines in step S301 whether the touch panel 12 has received the first information from the user. When the determination in step S301 is "yes", that is, when the operation control unit 153 determines that the touch panel 12 has received the first information from the user, the operation control unit 153 executes step S302. In addition, the first information is information specifying the brightness of the object area TR.

[0141] In step S302, the operation control unit 153 changes the brightness of the object image H1 included in the first simulated image G1 and the brightness of the projector image L1 included in the first simulated image G1 to the brightness based on the first information. In addition, when the determination in step S301 is "no", that is, when the operation control unit 153 determines that the touch panel 12 has not received the first information from the user, step S302 is skipped.

[0142] Next, in step S303, the operation control unit 153 determines whether the touch panel 12 has received the second information from the user. When the determination in step S303 is "yes", that is, when the operation control unit 153 determines that the touch panel 12 has received the second information from the user, the operation control unit 153 executes step S304. In addition, the second information is information specifying the brightness of the light used when the virtual projector C4 projects an image, and further, is information specifying the brightness of the sample image J1.

[0143] In step S304, the operation control unit 153 changes the brightness of the sample image J1 included in the first simulated image G1 and the brightness of the path image L2 included in the first simulated image G1 to the brightness based on the second information. In addition, when the determination in step S303 is "no", that is, when the operation control unit 153 determines that the touch panel 12 has not received the second information from the user, step S304 is skipped.

[0144] Regarding each brightness included in the subsequent image u4, it can also be changed by the same method as the method for changing the brightness of the first simulation image G1. In this case, the screen image v2 is used instead of the sample image J1.

[0145] Regarding each brightness included in the subsequent image u5, it can also be changed by the same method as the method for changing the brightness of the first simulation image G1. In this case, the screen image v2 is used instead of the sample image J1.

[0146] A6: An example of the operation

[0147] Next, an example of the above operation will be described. In step S101, the processing device 15 starts executing the program P1. When the touch panel 12 receives a start instruction from the user, step S101 is executed. The start instruction is, for example, a click on the icon i of the program P1 displayed on the touch panel 12. Figure 10 FIG. is an example of the icon i displayed on the touch panel 12.

[0148] When the icon i is clicked, the processing device 15 reads the program P1 from the storage device 14. Next, the processing device 15 executes the program P1.

[0149] The processing device 15 causes the touch panel 12 to display a start screen until the program P1 is executed. When the processing device 15 executes the program P1, the motion control unit 153 causes the touch panel 12 to display the first guidance image t1.

[0150] Figure 11 FIG. is an example of the first guidance image t1. The first guidance image t1 shows a summary of the functions of the information processing device 1 achieved by executing the program P1.

[0151] For example, Figure 11 the first guidance image t1 shown indicates a projector d1 that issues a comment such as "Let's try to set up the projector in my room using AR". AR represents Augmented Reality, that is, augmented reality.

[0152] The comment shown in the first guidance image t1 is not limited to the comment such as "Let's try to set up the projector in my room using AR", and can be changed as appropriate. The first guidance image t1 may not show the projector d1. The first guidance image t1 may show an object different from the projector d1, such as an animal, instead of the projector d1.

[0153] Next, in step S102, the camera 11 generates captured data kt by photographing the subject area TR. Next, in step S103, the motion sensor 13 generates motion data m. Next, in step S104, the acquisition unit 151 acquires the captured data kt and the motion data m.

[0154] Alternatively, after step S104 is completed, the motion control unit 153 acquires the captured data kt from the acquisition unit 151. In this case, the motion control unit 153 provides the captured data kt as image data r to the touch panel 12, thereby causing the touch panel 12 to display the object image H1. Figure 12 FIG. is an example of the information processing apparatus 1 that displays the object image H1.

[0155] Next, in step S105, the motion control unit 153 causes the recognition unit 152 to recognize the wall E1.

[0156] In step S105, the motion control unit 153 first causes the touch panel 12 to display the image u1.

[0157] Figure 13 FIG. is an example of the image u1. The image u1 is an image obtained by overlapping the second guide image t2 on the object image H1. The second guide image t2 represents the projector d1 that emits a comment such as "Nice to meet you. Let's start trying to use the projector!"

[0158] The comment shown in the second guide image t2 is not limited to the comment such as "Nice to meet you. Let's start trying to use the projector!" and can be appropriately changed. The second guide image t2 may not show the projector d1. The second guide image t2 may show an object different from the projector d1, such as an animal, instead of the projector d1.

[0159] Next, the motion control unit 153 causes the touch panel 12 to display the image u2.

[0160] Figure 14 FIG. is an example of the image u2. The image u2 is an image obtained by overlapping the third guide image t3 and the button v1 on the object image H1. The third guide image t3 is an image that prompts the user to generate a scanning state. The scanning state refers to a state in which the information processing apparatus 1 is moved from the first point to the second point while the camera 11 photographs the wall E1. The button v1 is a button for accepting the start input of the scanning state.

[0161] The third guide image t3 represents the projector d1 that emits a comment such as "First, press the button and shake the smartphone where you want to project."

[0162] The annotation shown in the third guidance image t3 is not limited to an annotation such as "First, press the button and shake the smartphone where you want to project". As long as it is an annotation that prompts the user to generate a scanning state, it can be appropriately changed. The third guidance image t3 may not show the projector d1. The third guidance image t3 may also show an object different from the projector d1, such as an animal, instead of the projector d1. The form of the button v1 is not limited to Figure 14 the form shown, and can be appropriately changed.

[0163] The user follows the annotation of the third guidance image t3. For example, when the wall E1 is displayed on the touch panel 12, the user presses the button v1 and then shakes the information processing device 1.

[0164] When the touch panel 12 detects a click on the button v1, the motion control unit 153 causes the touch panel 12 to display the image u3.

[0165] Figure 15 is a diagram showing an example of the image u3. The image u3 is an image obtained by overlapping the fourth guidance image t4 on the object image H1. The fourth guidance image t4 shows the projector d1 that emits an annotation such as "Scan the wall".

[0166] The annotation shown in the fourth guidance image t4 is not limited to an annotation such as "Scan the wall", and can be appropriately changed. The fourth guidance image t4 may not show the projector d1. The fourth guidance image t4 may also show an object different from the projector d1, such as an animal, instead of the projector d1.

[0167] Figure 16 is a diagram showing the state of the user shaking the information processing device 1. When the user shakes the information processing device 1, a scanning state is generated.

[0168] In the scanning state, the recognition unit 152 acquires the first captured data k1, the second captured data k2, and the motion data ms.

[0169] The recognition unit 152 identifies the wall E1 based on the first captured data k1, the second captured data k2, and the motion data ms.

[0170] Next, in step S201, the motion control unit 153 causes the recognition unit 152 to determine the distance n from the information processing device 1 to the wall E1.

[0171] Next, in step S202, the motion control unit 153 generates the virtual space VS.

[0172] Next, in step S203, the motion control unit 153 sets the first position C1 and the second position C2 in the virtual space VS.

[0173] Next, in step S204, the motion control unit 153 configures the virtual plane C3 in the virtual space VS.

[0174] Next, in step S205, the motion control unit 153 configures the virtual projector C4 in the virtual space VS.

[0175] Next, in step S206, the motion control unit 153 configures the screen image v2 on the virtual plane C3.

[0176] In step S206, the motion control unit 153 first makes the center position of the screen image v2 on the virtual plane C3 coincide with the intersection position of the optical axis of the projection lens of the virtual plane C3 and the virtual projector C4.

[0177] Next, the motion control unit 153 sets the path of the projection light from the virtual projector C4 toward the screen image v2 in the virtual space VS.

[0178] Next, the motion control unit 153 sets a virtual camera having the same specifications as those of the camera 11 at the second position C2. The optical axis position of the imaging lens of the virtual camera coincides with the optical axis position of the projection lens of the virtual projector C4.

[0179] Next, the motion control unit 153 retains the screen image v2, the virtual projector C4, and the path of the projection light from the virtual projector C4 toward the original image of the sample image J1 in the virtual space VS, and deletes the virtual plane C3 from the virtual space VS.

[0180] Next, the motion control unit 153 determines the image obtained when the virtual camera performs shooting as the second image.

[0181] The second image has transparency. The second image includes the image obtained by observing the screen image v2 from the second position C2. In the second image, the image obtained by observing the screen image v2 from the second position C2 becomes another example of the display image.

[0182] The second image also includes an image representing the virtual projector C4. In the second image, the image representing the virtual projector C4 is another example of the projector image L1.

[0183] The second image also includes an image representing the path of the projection light from the virtual projector C4 toward the screen image v2. In the second image, the image representing the path of the projection light from the virtual projector C4 toward the screen image v2 becomes another example of the path image L2.

[0184] Next, the motion control unit 153 generates an image u4 by overlapping the second image and the fifth guide image t5 on the object image H1. The image u4 is the second simulation image. The image u4 and the second simulation image are each another example of a simulation image. Next, the motion control unit 153 causes the touch panel 12 to display the image u4.

[0185] Figure 17 FIG. shows an example of the image u4. In the image u4, the position of the screen image v2 relative to the wall E1 changes corresponding to the position change of the touch panel 12 in the real space RS and the orientation change of the touch panel 12 in the real space RS. The touch panel 12 is mounted on the information processing device 1. Therefore, the position change of the touch panel 12 in the real space RS means the position change of the information processing device 1 in the real space RS. In addition, the orientation change of the touch panel 12 in the real space RS means the orientation change of the information processing device 1 in the real space RS. Therefore, by changing the position and orientation of the information processing device 1 respectively, the user can adjust the position of the screen image v2 as if the information processing device 1 were a projector 2.

[0186] In addition, corresponding to the position change of the touch panel 12 in the real space RS and the orientation change of the touch panel 12 in the real space RS, the part of the wall E1 shown in the object image H1 is changed respectively.

[0187] Therefore, in the case where any one of the position change of the touch panel 12 in the real space RS and the orientation change of the touch panel 12 in the real space RS occurs, the part of the wall E1 shown in the object image H1 of the image u4 is changed. In contrast, the position of the projector image L1 in the image u4 is not changed. Therefore, by observing the image u4 displayed on the touch panel 12, the user can adjust the position of the screen image v2 on the wall E1 as if the projector 2 were present at the position of the information processing device 1.

[0188] The screen image v2 includes an operation button v3. The operation button v3 is used to fix the position of the screen image v2 relative to the wall E1. Further, the operation button v3 is used by the user to input a position setting instruction.

[0189] The form of the operation button v3 is not limited to Figure 17 the form shown, and can be changed appropriately. The color of the screen image v2 having the operation button v3 is gray. The color of the screen image v2 having the operation button v3 is not limited to gray and can be changed appropriately.

[0190] The fifth guiding image t5 is an image that prompts the user to perform an operation to fix the position of the fixed screen image v2 relative to the wall E1. The fifth guiding image t5 shows a projector d1 that emits a comment such as "If the location of the screen is determined, press the operation button."

[0191] The comment shown in the fifth guiding image t5 is not limited to the comment such as "If the location of the screen is determined, press the operation button." As long as it is a comment that prompts the user to perform an operation to fix the position of the fixed screen image v2, it can be appropriately changed. The fifth guiding image t5 may not show the projector d1. The fifth guiding image t5 may also show an object different from the projector d1, such as an animal, instead of the projector d1.

[0192] The user confirms the image u4 while changing the position of the information processing device 1. Figure 18 It shows that when the position of the information processing device 1 is closer to the wall E1 than the position of the information processing device 1 that shows the image u4 Figure 17 shown, it is a diagram showing an example of the image u4 displayed by the information processing device 1. In Figure 18 the fifth guiding image t5 is omitted. The closer the information processing device 1 is to the wall E1, the smaller the ratio of the size of the screen image v2 to the size of the wall E1. Figure 18 The size of the screen image v2 shown Figure 17 is smaller than the size of the screen image v2 shown. In addition, the size of the screen image v2 shown in the image u4 may not be changed.

[0193] In order to notify the user of a method for reducing the ratio of the size of the screen image v2 to the size of the wall E1, the action control unit 153 may also overlap an image of the projector d1 that emits a comment such as "It gets smaller when approaching" on the image u4. The comment such as "It gets smaller when approaching" is an example of a first operation comment indicating an operation for reducing the ratio of the size of the screen image v2 to the size of the wall E1.

[0194] The first operation comment is not limited to the comment such as "It gets smaller when approaching" and can be appropriately changed. As long as the first operation comment is shown, the projector d1 that emits the first operation comment may not be shown. The object that emits the first operation comment is not limited to the projector d1. For example, it may also be an object different from the projector d1, such as an animal.

[0195] Figure 19 It shows that when the position of the information processing device 1 is farther from the wall E1 than the position of the information processing device 1 that shows the image u4 Figure 17 shown, it is a diagram showing an example of the image u4 displayed by the information processing device 1. In Figure 19In this case, the fifth guiding image t5 is omitted. The farther the information processing apparatus 1 is from the wall E1, the larger the ratio of the size of the screen image v2 to the size of the wall E1 becomes. Figure 19 The size ratio of the screen image v2 shown Figure 17 is larger than the size of the screen image v2 shown. Additionally, the size of the screen image v2 shown in the image u4 may also remain unchanged.

[0196] In order to notify the user of the method for increasing the ratio of the size of the screen image v2 to the size of the wall E1, the operation control unit 153 may also superimpose an image of the projector d1 that displays a comment saying "It gets bigger when you move away" on the image u4. The comment "It gets bigger when you move away" is an example of a second operation comment indicating the operation for increasing the ratio of the size of the screen image v2 to the size of the wall E1.

[0197] The second operation comment is not limited to the comment "It gets bigger when you move away" and can be appropriately changed. As long as the second operation comment is shown, the projector d1 that emits the second operation comment may not be shown. The object that emits the second operation comment is not limited to the projector d1. For example, it may be an object different from the projector, such as an animal.

[0198] Additionally, the operation control unit 153 may also change the transmittance of the screen image v2 in the image u4 corresponding to the distance n from the information processing apparatus 1 to the wall E1. For example, the operation control unit 153 increases the transmittance of the screen image v2 in the image u4 corresponding to an increase in the distance n. In this case, as the distance n increases, the visibility of the screen image v2 in the image u4 decreases. Therefore, the operation control unit 153 can simulate the phenomenon that the visibility of the projected image F1 on the wall E1 decreases corresponding to an increase in the distance from the wall E1 to the projector 2.

[0199] Figure 20 FIG. is an example of the image u4 displayed on the information processing apparatus 1 when the optical axis of the photographing lens 111 is inclined with respect to the normal line of the wall E1. In this case, the screen image v2 has a distortion corresponding to the inclination of the optical axis of the photographing lens 111 with respect to the normal line of the wall E1. This distortion is called trapezoidal distortion. When the projector 2 has a distortion correction function for correcting trapezoidal distortion, the operation control unit 153 corrects the trapezoidal distortion of the screen image v2 through a distortion correction function equivalent to the distortion correction function of the projector 2. Figure 21 is a diagram showing an example of the image u4 having the screen image v2 with the Figure 20 shown trapezoidal distortion corrected. In Figure 20 and Figure 21 the fifth guiding image t5 is omitted.

[0200] When the touch panel 12 detects a click on the operation button v3, the motion control unit 153 fixes the screen image v2 at the position where the screen image v2 was displayed when the operation button v3 was clicked.

[0201] Next, the motion control unit 153 updates the image u4 to the image u5. For example, the motion control unit 153 updates the image u4 to the image u5 by performing deletion of the operation button v3 on the image u4, changing the color of the screen image v2 from gray to blue, and adding the sixth guidance image t6. The changed color of the screen image v2 is not limited to blue and can be changed appropriately.

[0202] Figure 22 It is a diagram showing an example of the image u5. The image u5 is another example of a simulated image. The sixth guidance image t6 in the image u5 is an image that prompts the user to decide to display the screen image v2.

[0203] In Figure 22 the sixth guidance image t6 represents a projector d1 that emits a comment "Click on the screen and try to project something you like on the screen".

[0204] The comment shown by the sixth guidance image t6 is not limited to the comment "Click on the screen and try to project something you like on the screen", and can be changed appropriately as long as it is a comment that prompts the user to decide to display on the screen image v2. The sixth guidance image t6 may not show the projector d1. The sixth guidance image t6 may show an object different from the projector d1, such as an animal, instead of the projector d1.

[0205] The user can confirm the fixed screen image v2 by moving the information processing device 1 while viewing the image u5. Figure 23 It is a diagram showing an example of the image u5 displayed by the information processing device 1 when the position of the information processing device 1 is closer to the wall E1 than the position of the information processing device 1 showing the image u5 shown in Figure 22 In Figure 23 the sixth guidance image t6 is omitted. Even when the position of the screen image v2 is fixed, the ratio of the size of the screen image v2 to the size of the wall E1 becomes smaller corresponding to the decrease in the distance between the information processing device 1 and the wall E1. Figure 23 The size of the screen image v2 shown in Figure 22 is smaller than the size of the screen image v2 shown in

[0206] The motion control unit 153 may also superimpose an image of the projector d1 that emits the first operation comment "It gets smaller when approaching" on the image u5. As long as the first operation comment is shown, the projector d1 that emits the first operation comment may not be shown. The object that emits the first operation comment is not limited to the projector d1. For example, it may be an object different from the projector d1, such as an animal.

[0207] Figure 24 Indicates that the position of the information processing device 1 is farther from the wall E1 than the position of the information processing device 1 in the image u5 shown Figure 22 FIG. is an example of the image u5 displayed by the information processing device 1 when the position of the information processing device 1 is farther from the wall E1 than the position of the image u5 shown. In Figure 24 The sixth guide image t6 is omitted. Even when the position of the screen image v2 is fixed, the ratio of the size of the screen image v2 to the size of the wall E1 increases corresponding to the increase in the distance between the information processing device 1 and the wall E1. Figure 24 The size of the screen image v2 shown Figure 22 is larger than the size of the screen image v2 shown. In addition, the size of the screen image v2 shown in the image u5 may also be fixed.

[0208] The motion control unit 153 may also superimpose an image of the projector d1 that emits the second operation comment "It gets larger when moving away" on the image u5. As long as the second operation comment is shown, the projector d1 that emits the second operation comment may not be shown. The object that emits the second operation comment is not limited to the projector d1. For example, it may be an object different from the projector d1, such as an animal.

[0209] In addition, the motion control unit 153 may also change the transmittance of the screen image v2 in the image u5 corresponding to the distance n from the information processing device 1 to the wall E1. For example, the motion control unit 153 increases the transmittance of the screen image v2 in the image u5 corresponding to the increase in the distance n.

[0210] Figure 25 FIG. is an example of the image u5 displayed in the information processing device 1 when the optical axis of the photographing lens 111 is inclined with respect to the normal of the wall E1. In this case, the screen image v2 has a trapezoidal distortion corresponding to the inclination of the optical axis of the photographing lens 111 with respect to the normal of the wall E1. When the projector 2 has a distortion correction function for correcting trapezoidal distortion, the motion control unit 153 corrects the trapezoidal distortion of the screen image v2 by a distortion correction function equivalent to the distortion correction function of the projector 2. Figure 26 Indicates having corrected Figure 25 FIG. is an example of the image u5 of the screen image v2 with the trapezoidal distortion shown. In Figure 25 and Figure 26In this case, the sixth guidance image t6 is omitted.

[0211] By operating the information processing apparatus 1 in accordance with the sixth guidance image t6, the user can determine the image to be displayed as the screen image v2.

[0212] When the touch panel 12 detects a click on the screen image v2 in the image u5, the action control unit 153 causes the touch panel 12 to display the menu image v4.

[0213] Figure 27 This is a diagram showing an example of the menu image v4. The menu image v4 includes a selection button v5, a first adjustment button v6, and a second adjustment button v7.

[0214] The selection button v5 is used to determine the image to be displayed as the screen image v2, that is, the sample image J1. The first adjustment button v6 is used to adjust the brightness of the object image H1. The second adjustment button v7 is used to adjust the brightness of the screen image v2 and the brightness of the sample image J1. Adjusting the brightness of the screen image v2 and adjusting the brightness of the sample image J1 means adjusting the brightness of the light emitted by the virtual projector C4.

[0215] When the touch panel 12 detects a click on the selection button v5, the action control unit 153 causes the touch panel 12 to display the image v81.

[0216] Figure 28 This is a diagram showing an example of the image v81. The image v81 shows candidates v8 for the image to be displayed on the screen image v2. The candidates v8 for the image are images corresponding to the projection image F1 projected from the projector 2. For example, the candidates v8 for the image are images representing the projection image F1 projected from the projector 2. The image candidates v8 are, for example, images of photos represented by photo data. The image candidates v8 can be images of documents represented by document data.

[0217] The user clicks on one of the candidates v8 for the image used as the sample image J1. When the touch panel 12 detects a click on the image candidate v8, the action control unit 153 determines the clicked image candidate v8 as the sample image J1.

[0218] Next, in step S207, the action control unit 153 determines the original image of the sample image J1. In step S207, the action control unit 153 determines the original image of the sample image J1 by changing the size of the sample image J1 to the size of the screen image v2.

[0219] Next, in step S208, the action control unit 153 determines the first simulated image G1.

[0220] In step S208, the motion control unit 153 changes the screen image v2 to the original image of the sample image J1 in the virtual space VS. Next, the motion control unit 153 sets a virtual camera having the same specifications as those of the camera 11 at the second position C2. The optical axis position of the shooting lens of the virtual camera coincides with the optical axis position of the projection lens of the virtual projector C4.

[0221] Next, the motion control unit 153 retains the original image of the sample image J1, the virtual projector C4, and the path of the projection light from the virtual projector C4 toward the original image of the sample image J1 in the virtual space VS, and deletes the virtual plane C3 from the virtual space VS.

[0222] Next, the motion control unit 153 determines the image obtained when the virtual camera performs shooting as the first image.

[0223] Next, the motion control unit 153 determines the first simulated image G1 by overlapping the first image on the object image H1.

[0224] Next, in step S209, the motion control unit 153 generates simulated image data r1 representing the first simulated image G1.

[0225] Next, in step S210, the motion control unit 153 causes the touch panel 12 to display the first simulated image G1 by providing the simulated image data r1 to the touch panel 12.

[0226] In the state where the first simulated image G1 is displayed on the touch panel 12, when the touch panel 12 detects a click on the sample image J1, the motion control unit 153 causes Figure 27 the menu image v4 shown in the figure to be displayed on the touch panel 12.

[0227] When the touch panel 12 detects a click on the first adjustment button v6 in the menu image v4, the motion control unit 153 causes the touch panel 12 to display the image u6.

[0228] Figure 29 is a diagram showing an example of the image u6. The image u6 includes an image u61 and an image u62. The image u61 is an image representing the operation unit v9. The image u62 is an image obtained by overlapping the OK button v10 and the cancel button v11 on the first simulated image G1.

[0229] The operation unit v9 includes an adjustment button v91, a + mark v92, and a - mark v93.

[0230] The adjustment button v91 can move between the + mark v92 and the - mark v93. The movement of the adjustment button v91 is performed by the user sliding the adjustment button v91. The sliding of the adjustment button v91 by the user is received by the touch panel 12. The end position of the sliding of the adjustment button v91 by the user is an example of the first information of the brightness of the specified object area TR. The action of the touch panel 12 receiving the sliding of the adjustment button v91 by the user is an example of step S301.

[0231] Next, in step S302, the motion control unit 153 changes the brightness of the object image H1 included in the first simulation image G1 and the brightness of the projector image L1 included in the first simulation image G1 to the brightness based on the sliding of the adjustment button v91.

[0232] In step S302, the motion control unit 153 first determines the position of the adjustment button v91 after the movement based on the sliding received by the touch panel 12. The motion control unit 153 adjusts the brightness of the object image H1 and the brightness of the projector image L1 according to the position of the adjustment button v91 after the movement.

[0233] The motion control unit 153 brightens the object image H1 corresponding to the decrease in the distance between the position of the adjustment button v91 and the + mark v92. The motion control unit 153 brightens the projector image L1 corresponding to the decrease in the distance between the position of the adjustment button v91 and the + mark v92.

[0234] The motion control unit 153 darkens the object image H1 corresponding to the decrease in the distance between the position of the adjustment button v91 and the - mark v93. The motion control unit 153 darkens the projector image L1 corresponding to the decrease in the distance between the position of the adjustment button v91 and the - mark v93.

[0235] Figure 30 It is a diagram showing an example in which the motion control unit 153 darkens the object image H1 and the projector image L1 corresponding to the decrease in the distance between the position of the adjustment button v91 and the - mark v93.

[0236] The user can adjust the brightness of the object image H1 by performing the sliding of the adjustment button v91.

[0237] The determination button v10 is used to determine the brightness of the object image H1. When the touch panel 12 detects a click on the determination button v10, the motion control unit 153 determines the brightness of the object image H1 and the brightness of the projector image L1 as the brightness based on the position of the adjustment button v91. Next, the motion control unit 153 changes the display on the touch panel 12 from the image u6 to the first simulation image G1.

[0238] The cancel button v11 is used to cancel the brightness adjustment of the object image H1 accompanying the movement of the adjustment button v91. When the touch panel 12 detects a click on the cancel button v11, the action control unit 153 returns the brightness of the object image H1 and the brightness of the projector image L1 to the brightness before the movement of the adjustment button v91. Then, the action control unit 153 changes the display on the touch panel 12 from the image u6 to the first analog image G1.

[0239] When the touch panel 12 Figure 27 detects a click on the second adjustment button v7 in the menu image v4 shown, the action control unit 153 causes the touch panel 12 to display the image u7.

[0240] Figure 31 FIG. is an example showing the image u7. The image u7 includes an image u71 and an image u72. The image u71 is an image showing the operation unit v12. The image u72 is an image obtained by overlapping the OK button v13 and the cancel button v14 on the first analog image G1.

[0241] The operation unit v12 includes an adjustment button v121, a + mark v122, and a - mark v123.

[0242] The adjustment button v121 can move between the + mark v122 and the - mark v123. The movement of the adjustment button v121 is performed by the user's sliding of the adjustment button v121. The user's sliding of the adjustment button v121 is received by the touch panel 12. The end position of the user's sliding of the adjustment button v121 is an example of the second information specifying the brightness of the sample image J1 and the brightness of the screen image v2. The second information is also information specifying the brightness of the light used when the virtual projector C4 projects the projection image F1. The action of the touch panel 12 receiving the user's sliding of the adjustment button v121 is an example of step S303.

[0243] Next, in step S304, the action control unit 153 changes the brightness of the sample image J1 included in the first analog image G1 and the brightness of the path image L2 included in the first analog image G1 to the brightness based on the sliding of the adjustment button v121.

[0244] In step S304, the action control unit 153 first determines the position of the adjustment button v121 based on the detection result of the touch panel 12.

[0245] The action control unit 153 brightens the sample image J1 as the distance between the position of the adjustment button v121 and the + mark v122 decreases. The action control unit 153 may also brighten the path image L2 as the distance between the position of the adjustment button v121 and the + mark v122 decreases.

[0246] When the distance between the position of the adjustment button v121 and the - mark v123 decreases, the operation control unit 153 darkens the sample image J1. The operation control unit 153 may also darken the path image L2 when the distance between the position of the adjustment button v121 and the - mark v123 decreases.

[0247] Figure 32 FIG. is an example showing that the operation control unit 153 darkens the sample image J1 and the path image L2 when the distance between the position of the adjustment button v121 and the - mark v123 decreases.

[0248] The user can adjust the brightness of the sample image J1 and the brightness of the path image L2 by performing a slide on the adjustment button v121. The position of the adjustment button v121 determined by the slide on the adjustment button v121 is an example of the second information.

[0249] The confirmation button v13 is used to confirm the brightness of the sample image J1. When the touch panel 12 detects a click on the confirmation button v13, the operation control unit 153 determines the brightness of the sample image J1 and the brightness of the path image L2 as the brightness based on the position of the adjustment button v121. Then, the operation control unit 153 changes the display on the touch panel 12 from the image u7 to the first simulation image G1.

[0250] The cancel button v14 is used to cancel the brightness adjustment of the sample image J1 accompanying the movement of the adjustment button v121. When the touch panel 12 detects a click on the cancel button v14, the operation control unit 153 returns the brightness of the sample image J1 and the brightness of the path image L2 to the brightness before the movement of the adjustment button v121. Then, the operation control unit 153 changes the display on the touch panel 12 from the image u7 to the first simulation image G1.

[0251] A7: Summary of the First Embodiment

[0252] The display method and the information processing apparatus 1 of the first embodiment include the following modes.

[0253] The acquisition unit 151 acquires the target image H1, and the target image H1 represents the target area TR including the wall E1. The operation control unit 153 causes the touch panel 12 to display the first simulated image G1. The first simulated image G1 is an image obtained by overlapping the sample image J1 corresponding to the projection image F1 projected onto the wall E1 by the projector 2 on the target image H1. The operation control unit 153 causes the touch panel 12 to display the target image H1 included in the first simulated image G1 with the brightness based on the first information of the brightness of the specified target area TR. The operation control unit 153 causes the touch panel 12 to display the sample image J1 included in the first simulated image G1 with the brightness based on the second information of the brightness of the specified sample image J1. According to this method, the brightness of the target image H1 and the brightness of the sample image J1 can be adjusted in the first simulated image G1 according to the first information and the second information. Therefore, by observing the first simulated image G1, the user can easily imagine the actual visibility of the projection image F1.

[0254] The first simulated image G1 includes the projector image L1 as an image representing the projector 2. According to this method, by observing the first simulated image G1, the user can easily imagine the projector 2 that projects the projection image F1.

[0255] The operation control unit 153 causes the touch panel 12 to display the projector image L1 included in the first simulated image G1 with the brightness based on the first information. According to this method, by observing the first simulated image G1, the user can more easily imagine the way the projector 2 that projects the projection image F1.

[0256] The first simulated image G1 includes the path image L2 representing the path of the light used by the projector 2 in the projection of the projection image F1. According to this method, by observing the first simulated image G1, the user can easily imagine the path of the light used by the projector 2 in the projection of the projection image F1.

[0257] The operation control unit 153 causes the touch panel 12 to display the path image L2 included in the first simulated image G1 with the brightness based on the second information. According to this method, by observing the first simulated image G1, the user can more easily imagine the path of the light used by the projector 2 in the projection of the projection image F1.

[0258] B: Variation

[0259] The following exemplifies the variation methods of the above-exemplified embodiments. Two or more methods arbitrarily selected from the following exemplifications can be appropriately combined within a non-conflicting range.

[0260] B1: First variation

[0261] In the first embodiment, the operation control unit 153 may also determine the second information based on the first information.

[0262] For example, when the object image H1 included in the first simulated image G1 brightens according to the first information, the motion control unit 153 determines the second information for darkening the sample image J1 included in the first simulated image G1. Further, when the object image H1 included in the first simulated image G1 darkens according to the first information, the motion control unit 153 determines the second information for brightening the sample image J1 included in the first simulated image G1. In such a case, the motion control unit 153 can suppress a decrease in the contrast between the object image H1 and the sample image J1. Therefore, it is possible to suppress a decrease in the visibility of the sample image J1 caused by a decrease in the contrast between the object image H1 and the sample image J1.

[0263] According to the first modification, the motion control unit 153 can change the brightness of the sample image J1 corresponding to a change in the brightness of the object image H1.

[0264] Further, in the first embodiment and the first modification, the motion control unit 153 may determine the first information based on the second information.

[0265] When the sample image J1 included in the first simulated image G1 brightens according to the second information, the motion control unit 153 determines the first information for darkening the object image H1 included in the first simulated image G1. Further, when the sample image J1 included in the first simulated image G1 darkens according to the second information, the motion control unit 153 determines the first information for brightening the object image H1 included in the first simulated image G1. In such a case, the motion control unit 153 can suppress a decrease in the contrast between the object image H1 and the sample image J1. Therefore, it is possible to suppress a decrease in the visibility of the sample image J1 caused by a decrease in the contrast between the object image H1 and the sample image J1.

[0266] In this case, the motion control unit 153 can change the brightness of the object image H1 corresponding to a change in the brightness of the sample image J1.

[0267] B2: Second modification

[0268] In the first modification, the object image H1 and the sample image J1 may be translucent, respectively.

[0269] When neither the object image H1 nor the sample image J1 is affected by the first information and the second information, that is, when the user does not input the first information and the second information, the motion control unit 153 sets the value of the coefficient w1 for adjusting the brightness of the object image H1 to "0.5". When neither the object image H1 nor the sample image J1 is affected by the first information and the second information, the motion control unit 153 sets the value of the coefficient w2 for adjusting the brightness of the sample image J1 to "0.5".

[0270] The motion control unit 153 changes the brightness of each pixel of the object image H1 by multiplying the brightness of each pixel of the object image H1 by a coefficient w1. The motion control unit 153 changes the brightness of each pixel of the sample image J1 by multiplying the brightness of each pixel of the sample image J1 by a coefficient w2.

[0271] The motion control unit 153 overlaps the sample image J1 with its pixel brightness changed on the object image H1 with its pixel brightness changed. Even if the object image H1 and the sample image J1 are each semi-transparent, the brightness of the portion where the sample image J1 overlaps the object image H1 is prevented from becoming too high.

[0272] When the brightness based on the first information is a brightness that brightens the object image H1, the motion control unit 153 increases the value of the coefficient w1 and decreases the value of the coefficient w2 in such a way that the sum of the value of the coefficient w1 and the value of the coefficient w2 is maintained at "1".

[0273] When the brightness based on the first information is a brightness that darkens the object image H1, the motion control unit 153 decreases the value of the coefficient w1 and increases the value of the coefficient w2 in such a way that the sum of the value of the coefficient w1 and the value of the coefficient w2 is maintained at "1".

[0274] When the brightness based on the second information is a brightness that brightens the sample image J1, the motion control unit 153 decreases the value of the coefficient w1 and increases the value of the coefficient w2 in such a way that the sum of the value of the coefficient w1 and the value of the coefficient w2 is maintained at "1".

[0275] When the brightness based on the second information is a brightness that darkens the sample image J1, the motion control unit 153 increases the value of the coefficient w1 and decreases the value of the coefficient w2 in such a way that the sum of the value of the coefficient w1 and the value of the coefficient w2 is maintained at "1".

[0276] In addition, the changed coefficient w1 corresponds to the first information, and the changed coefficient w2 corresponds to the second information.

[0277] According to the second modification example, even if the object image H1 and the sample image J1 are each semi-transparent, the brightness of the portion where the sample image J1 overlaps the object image H1 can be prevented from becoming too high.

[0278] B3: The third modification example

[0279] In the first embodiment, the first modification to the second modification, the operation control unit 153 may also change the brightness of the path image L2 to a brightness different from the brightness based on the second information when the touch panel 12 receives a change instruction to change the brightness of the path image L2. The change instruction is a click on the projector image L1. The change instruction is not limited to a click on the projector image L1. For example, it may also be a click on the path image L2.

[0280] For example, when the touch panel 12 receives a change instruction, the operation control unit 153 makes the brightness of the path image L2 darker than the brightness based on the second information. Making the brightness of the path image L2 darker than the brightness based on the second information includes making the transmittance of the path image L2 higher than the transmittance of the path image L2 when the brightness is based on the second information. In this case, the display of the path image L2 can be made less obvious. According to the third modification, the brightness of the path image L2 can be changed independently of the brightness of the sample image J1.

[0281] B4: The fourth modification

[0282] In the first embodiment, the first modification to the third modification, the projector 2 may also have an optical zoom lens. In this case, the operation control unit 153 may also change the size of the screen image v2 and the size of the sample image J1 within the range of the zoom characteristics of the optical zoom lens provided in the projector 2.

[0283] For example, when the touch panel 12 receives a pinch in while the screen image v2 is being displayed on the touch panel 12, the operation control unit 153 reduces the size of the screen image v2 within the range of the zoom characteristics of the optical zoom lens.

[0284] When the touch panel 12 receives a pinch out while the screen image v2 is being displayed on the touch panel 12, the operation control unit 153 increases the size of the screen image v2 within the range of the zoom characteristics of the optical zoom lens.

[0285] When the touch panel 12 receives a pinch in while the sample image J1 is being displayed on the touch panel 12, the operation control unit 153 reduces the size of the sample image J1 within the range of the zoom characteristics of the optical zoom lens.

[0286] When the touch panel 12 receives a pinch out while the sample image J1 is being displayed on the touch panel 12, the operation control unit 153 increases the size of the sample image J1 within the range of the zoom characteristics of the optical zoom lens.

[0287] When the projector 2 has a digital zoom function, the operation control unit 153 can also change the size of the screen image v2 and the size of the sample image J1 within the range of the zoom characteristics based on the digital zoom function of the projector 2. For example, the method for changing the size of the screen image v2 when the projector 2 has a digital zoom function is the same as the method for changing the size of the screen image v2 when the projector 2 has an optical zoom lens. For example, the method for changing the size of the sample image J1 when the projector 2 has a digital zoom function is the same as the method for changing the size of the sample image J1 when the projector 2 has an optical zoom lens.

[0288] According to the fourth modification example, when the projector 2 has an optical zoom lens or a digital zoom function, the first analog image G1 corresponding to the zoom function of the projector 2 can be displayed.

[0289] B5: The fifth modification example

[0290] In the first embodiment and the first to fourth modification examples, the projector 2 may also have a lens shift function. In this case, the operation control unit 153 can also change the position of the screen image v2 and the position of the sample image J1 within the range of the shift characteristics based on the lens shift of the projector 2.

[0291] For example, when the touch panel 12 accepts a slide on the screen image v2, the operation control unit 153 moves the screen image v2 according to the slide within the range of the shift characteristics based on the lens shift.

[0292] When the touch panel 12 accepts a slide on the sample image J1, the operation control unit 153 moves the sample image J1 according to the slide within the range of the shift characteristics based on the lens shift.

[0293] According to the fifth modification example, when the projector 2 has a lens shift function, the first analog image G1 corresponding to the lens shift function of the projector 2 can be displayed.

[0294] B6: The sixth modification example

[0295] In the first embodiment and the first to fifth modification examples, the part of the screen image v2 displayed on the touch panel 12 may be restricted. For example, the operation control unit 153 uses a mask v16 to define the part of the screen image v2 displayed on the touch panel 12.

[0296] Figure 33FIG. is an example showing a mask v16. The mask v16 includes a transmissive region v161 and a non-transmissive region v162. The transmittance of the transmissive region v161 is 100%. The transmittance of the non-transmissive region v162 is 0%. The shapes of the transmissive region v161 and the non-transmissive region v162 can be appropriately changed separately. The mask v16 is selected by the user.

[0297] The operation control unit 153 defines the portion of the screen image v2 that is displayed on the touch panel 12 by overlapping the mask v16 on the screen image v2. In this case, only the portion of the screen image v2 that overlaps with the transmissive region v161 is displayed on the touch panel 12.

[0298] The operation control unit 153 may also define the portion of the sample image J1 that is displayed on the touch panel 12 by overlapping the mask v16 on the sample image J1. In this case, only the portion of the sample image J1 that overlaps with the transmissive region v161 is displayed on the touch panel 12.

[0299] According to the sixth modification, it is possible to limit the portion of the screen image v2 that is displayed on the touch panel 12 and the portion of the sample image J1 that is displayed on the touch panel 12. In addition, it is possible to determine the shape of the portion of the screen image v2 that is displayed on the touch panel 12 and the shape of the portion of the sample image J1 that is displayed on the touch panel 12 according to the shape of the non-transmissive region v162.

[0300] B7: Seventh Modification

[0301] In the first embodiment and the first to sixth modifications, the operation control unit 153 may also display at least one of the size of the screen image v2 and the size of the sample image J1 on the touch panel 12. In addition, in the first embodiment and the first to sixth modifications, the operation control unit 153 may also display the distance n from the information processing device 1 to the wall E1 on the touch panel 12.

[0302] Figure 34 FIG. is an example showing an image u8 that displays the size of the screen image v2 and the distance n.

[0303] In Figure 34 αm is shown as the horizontal length of the screen image v2. βm is shown as the vertical length of the screen image v2. γm is shown as the distance n.

[0304] In addition, the display method of the size of the sample image J1 is, for example, the same as the display method of the size of the screen image v2. The display method of the size of the screen image v2, the display method of the distance n, and the display method of the size of the sample image J1 are not limited to Figure 34 the display method shown, and can be appropriately changed.

[0305] According to the seventh modification example, the user can confirm at least one of the size of the screen image v2, the size of the sample image J1, and the distance n by observing the touch panel 12.

[0306] B8: Eighth modification example

[0307] In the first embodiment and the first to seventh modification examples, the projector 2 as the simulation object may also be changed. In this case, the motion control unit 153 generates the first simulation image G1 according to the characteristics of the changed projector 2 as the simulation object.

[0308] In the first embodiment and the first to seventh modification examples, the projector 2 as the simulation object may also be selected from multiple projectors. In this case, the motion control unit 153 generates the first simulation image G1 according to the characteristics of the selected projector 2 as the simulation object.

[0309] B9: Ninth modification example

[0310] In the first embodiment and the first to eighth modification examples, the position of the virtual projector C4 may also be fixed in the virtual space VS.

[0311] B10: Tenth modification example

[0312] In the first embodiment and the first to ninth modification examples, the camera 11, the touch panel 12, and the processing device 15 may also be separated from each other. In the first embodiment and the first to ninth modification examples, the camera 11 and the touch panel 12 may also be separated from the information processing device 1. In the first embodiment and the first to ninth modification examples, the camera 11 may also be separated from the information processing device 1. In the first embodiment and the first to ninth modification examples, the touch panel 12 may also be separated from the information processing device 1. In the first embodiment and the first to ninth modification examples, the display 121 and the input device 122 may also be separated from each other.

Claims

1. A display method, wherein, the display method includes: acquiring an object image using a camera, the object image representing an object area that includes a surface in the real space and does not include a projected image projected by a projector disposed in the real space; displaying a simulated image on a touch panel provided in an information processing device, the simulated image being obtained by overlapping a display image corresponding to the display image projected by the projector onto the surface with the object image; displaying the object image included in the simulated image on the touch panel with a brightness based on first information specifying the brightness of the object area; and displaying the display image included in the simulated image on the touch panel with a brightness based on second information specifying the brightness of the display image.

2. The display method according to claim 1, wherein, the display method further includes: receiving the first information from a user; and determining the second information based on the first information.

3. The display method according to claim 1 or 2, wherein, the display method further includes: receiving the second information from a user; and determining the first information based on the second information.

4. The display method according to claim 1 or 2, wherein, the simulated image includes a projector image, the projector image being an image representing the projector.

5. The display method according to claim 4, wherein, the display method further includes: displaying the projector image included in the simulated image on the touch panel with a brightness based on the first information.

6. The display method according to claim 1 or 2, wherein, the simulated image includes a path image, the path image representing the path of light used by the projector in image projection.

7. The display method according to claim 6, wherein, the display method further includes: displaying the path image included in the simulated image on the touch panel with a brightness based on the second information.

8. The display method according to claim 7, wherein, the display method further includes: when receiving a change instruction to change the brightness of the path image, changing the brightness of the path image to a brightness different from the brightness based on the second information.

9. The display method according to claim 7, wherein, the display method further includes: when receiving a change instruction to change the brightness of the path image, making the brightness of the path image darker than the brightness based on the second information.

10. A display system, wherein, the display system includes: a camera; a touch panel provided in an information processing device; and at least one processing unit, the at least one processing unit performs: acquiring an object image using the camera, the object image representing an object area that includes a surface in the real space and does not include a projected image projected by a projector disposed in the real space; causing the touch panel to display a simulated image, the simulated image being obtained by overlapping a display image corresponding to the display image projected by the projector onto the surface with the object image; Display the object image included in the simulated image on the touch panel with a brightness based on the brightness of the first information designating the brightness of the object area; and Display the display image included in the simulated image on the touch panel with a brightness based on the brightness of the second information designating the brightness of the display image.

Images