Projection method and program

Abstract

To easily adjust the shape of a projection image and maintain the adjusted shape.SOLUTION: A projection method includes: controlling a projector to display a projection image having four corners inside a display area having four corners as an area on a projection surface; acquiring a first picked-up image that is a picked-up image of a range including the display area where the projection image is displayed; executing image processing on the first picked-up image to detect a plurality of first feature points corresponding one by one to four corners of a first image showing the display area included in the first picked-up image; controlling the projector to display pointer images located at the corners of the projection image and to adjust the shape of the projection image; changing the positions of the corners of the projection image based on a user operation to move the pointer images to execute first correction to correct the shape of the projection image; and executing, based on the plurality of first feature points, second correction to maintain the positional relationship between the four corners of the display area and the four corners of the projection image determined through the first correction.SELECTED DRAWING: Figure 15

Description

[Technical field]

[0001] The present invention relates to a projection method and a program. [Background technology]

[0002] Technologies have been developed for correcting the position and shape of an image displayed on a projection surface by a device such as a projector. For example, Patent Document 1 discloses a projector that corrects an image displayed on a projection surface when the projector is displaced from its initial position, thereby displaying the image in the same position as when the image was displayed while the projector was located at its initial position. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-092169 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the correction reference period when the projector described in Patent Document 1 is located at the initial position, if the shape of the image displayed on the projection surface is not the shape desired by the user, the shape may be maintained. For this reason, it is necessary to adjust the shape of the image displayed on the projection surface in the correction reference period to the shape desired by the user in advance. In order to perform this adjustment, the user needs to accurately set the position, direction, angle, etc. of the projector, which is time-consuming for the user. [Means for solving the problem]

[0005] One aspect of the projection method according to the present invention includes controlling a projection device to display a projection image having four corners inside a display area having four corners as an area on a projection surface; acquiring a first captured image which is a captured image of a range including the display area in which the projection image is displayed; detecting a plurality of first feature points corresponding one-to-one to four corners of a first image showing the display area included in the first captured image by executing image processing on the first captured image; displaying a pointer image located at a corner of the projection image and for adjusting a shape of the projection image by controlling the projection device; User Actions and performing a first correction to correct a shape of the projected image by changing positions of corners of the projected image based on the plurality of first feature points, and performing a second correction to maintain a positional relationship between the four corners of the display area and the four corners of the projected image determined by the first correction, based on the plurality of first feature points.

[0006] One aspect of the program of the present invention includes causing a processing device to control a projection device to display a projection image having four corners inside a display area on a projection surface, the display area having four corners, obtain a first captured image representing a result of capturing an image of an area including the display area in which the projection image is displayed, and perform image processing on the first captured image to detect a plurality of first feature points corresponding one-to-one to the four corners of a first image representing the display area included in the first captured image, and control the projection device to display a pointer image located at a corner of the projection image and for adjusting a shape of the projection image, and a pointer for moving the pointer image. User Actions and performing a first correction to correct a shape of the projected image by changing positions of corners of the projected image based on the plurality of first feature points, and performing a second correction to maintain a positional relationship between the four corners of the display area and the four corners of the projected image determined by the first correction, based on the plurality of first feature points. [Brief description of the drawings]

[0007] [Figure 1]10 is a schematic diagram illustrating a state in which a projection image GP1 is displayed in a first period. FIG. [Diagram 2] 13 is a schematic diagram illustrating a state in which a projection image GP2 is displayed in a second period. FIG. [Diagram 3] 13 is a schematic diagram illustrating a state in which a projection image GP2 is displayed in a third period. FIG. [Figure 4] 13 is a schematic diagram illustrating a state in which a projection image GP1 is displayed in a fourth period. FIG. [Diagram 5] 13 is a schematic diagram illustrating a state in which a projection image GP2 is displayed in a fifth period. FIG. [Figure 6] 13 is a schematic diagram illustrating a state in which a projection image GP2 is displayed in a sixth period. FIG. [Figure 7] 1 is a block diagram showing the configuration of a projector 1 according to a first embodiment. [Figure 8] 1 is a block diagram showing a configuration of a storage device 10 according to a first embodiment. [Figure 9] 2 is a schematic diagram illustrating an example of an image represented by the first projection image information 102. FIG. [Figure 10] 2 is a schematic diagram illustrating an example of an image indicated by the first captured image information 105. FIG. [Figure 11] 1 is a schematic diagram illustrating an example of an image indicated by the second captured image information 106. FIG. [Figure 12] 13 is a schematic diagram illustrating an example of an image indicated by the third captured image information 107. FIG. [Figure 13] 13 is a schematic diagram illustrating an example of an image indicated by the fourth captured image information 108. FIG. [Figure 14] FIG. 11 is a schematic diagram for explaining a second correction. [Figure 15] 11 is a flowchart for explaining the operation of the projector 1 in a time period including a first period to a third period. [Figure 16] 13 is a flowchart for explaining the operation of the projector 1 in a time period including a fourth period to a sixth period. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. Note that the dimensions and scale of each part in the drawings may differ from the actual dimensions, and some parts are shown diagrammatically to facilitate understanding. Furthermore, the scope of the present invention is not limited to these embodiments unless otherwise specified in the following description to the effect that the present invention is limited thereto.

[0009] 1. First embodiment In the first embodiment, a projection method and a program according to the present invention will be described using a projector that corrects the shape of an image displayed on a projection surface based on a user's operation and displays the corrected image at a predetermined position on the projection surface. That is, a user can easily adjust the shape of an image displayed on a projection surface without having to accurately set the direction and angle of the projector according to this embodiment.

[0010] 1.1.Projector Overview Hereinafter, an overview of a projector 1 according to a first embodiment will be described with reference to FIGS.

[0011] 1 is a schematic diagram illustrating a state in which a projected image GP1 is displayed in a first period. A projector 1 displays a projected image GP1 by projecting projection light onto an area R2 of a wall surface W1, which is a projection surface. The projected image GP1 includes a plurality of points. The projected image GP1 in the first period may be referred to as a "projected image GP1-1."

[0012] The projector 1 includes an imaging device 14 that captures an image of a range including a predetermined region of the projection surface, and a projection device 16 that projects projection light onto the projection surface. The imaging device 14 includes an imaging lens 142 for collecting light, and an imaging element 140 that generates a captured image by converting the light collected by the imaging lens 142 into an electrical signal. The imaging element 140 includes a plurality of pixels. The projection device 16 includes a light source (not shown), an optical modulator 160 that modulates the light emitted from the light source into projection light for displaying a projection image on the projection surface, and a projection lens 162 that projects the projection light modulated by the optical modulator 160 onto the projection surface. The optical modulator 160 includes a plurality of pixels. The projector 1 displays a projection image on the projection surface by controlling the projection device 16. In this embodiment, the projector 1 displays a projection image on the wall surface W1 by controlling the projection device 16.

[0013] In the first period, the projector 1 controls the imaging device 14 to capture an image of a range including the region R2 where the projection image GP1-1 is displayed. The region R2 includes the region R1. That is, the projector 1 captures an image of a range including the region R1. The region R1 includes a plurality of corners CN1. The plurality of corners CN1 include corners CN1-1, CN1-2, CN1-3, and CN1-4. The region R1 may be, for example, an area surrounded by a frame of a screen installed on the wall surface W1, or an area on the wall surface W1 surrounded by a frame line drawn on a part of the surface of the wall surface W1. In addition, the projector 1 generates information indicating a correspondence relationship between a plurality of pixels included in the light modulator 160 and a plurality of pixels included in the imaging element 140 based on a captured image representing a result of capturing an image of a range including the region R2 where the projection image GP1-1 is displayed and an image projected from the projector 1 when the projection image GP1-1 is displayed.

[0014] The position of the region R1, specifically the positions of the four corners of the region R1, are clearly indicated on the wall W1. In Fig. 1, the position of the region R1 is illustrated by a thick straight line, but is not limited to this. For example, the position of the region R1 may be indicated by attaching a marker indicating the position of the corner of the region R1 to each of the four corners of the region R1. In Fig. 1, the thick straight line indicating the position of the region R1 corresponds to a frame line drawn on the frame of the screen or a part of the surface of the wall W1.

[0015] FIG. 2 is a schematic diagram illustrating a state in which the projected image GP2 is displayed in the second period. The second period is a period after the first period. The projector 1 controls the projection device 16 to display the projected image GP2 in the area R1 of the wall surface W1. The projected image GP2 has corners CN2-1, CN2-2, CN2-3, and CN2-4. The projected image GP2 in the second period may be referred to as the "projected image GP2-1."

[0016] Furthermore, the projector 1 controls the projection device 16 to display a pointer image GC1-1 indicating the position of corner CN2-1, a pointer image GC1-2 indicating the position of corner CN2-2, a pointer image GC1-3 indicating the position of corner CN2-3, and a pointer image GC1-4 indicating the position of corner CN2-4. The pointer images GC1-1 to GC1-4 are, for example, in the shape of a cross in which two straight lines intersect with each other. When there is no need to distinguish between the pointer images GC1-1 to GC1-4, the pointer images GC1-1 to GC1-4 may be referred to as "pointer image GC1."

[0017] A user changes the position of a corner of the projection image GP2 indicated by the pointer image GC1 by executing an operation to move the pointer image GC1. For example, a user changes the position of a corner CN2-1 indicated by the pointer image GC1-1 by executing an operation to move the pointer image GC1-1. The projector 1 corrects the shape of the projection image GP2 by changing the position of the corner of the projection image GP2 based on an operation from the user who moves the pointer image GC1. The user corrects the shape of the projection image GP2 to a desired shape by changing the position of the corner of the projection image GP2. For example, the shape of the projection image GP2 is corrected by the user so that it becomes a shape approximately similar to the screen frame. That is, if the screen frame is rectangular, the shape of the projection image GP2 is shaped based on an operation from the user so that the shape of the projection image GP2 is also rectangular. Note that correcting the shape of the projection image based on an operation from the user is sometimes referred to as "first correction".

[0018] 3 is a schematic diagram illustrating a state in which the projected image GP2 is displayed in the third period. The third period is a period after the second period. The projected image GP2 in the third period may be referred to as the "projected image GP2-2." The projected image GP2-2 is the projected image GP2 that has been subjected to the first correction. The projected image GP2-2 is displayed in the region R1.

[0019] In the third period, the projector 1 controls the imaging device 14 to capture an image of the range including the region R1 where the projection image GP2-2 is displayed. The projector 1 also performs image processing on the captured image that represents the result of capturing an image of the range including the region R1 where the projection image GP2-2 is displayed, thereby detecting a plurality of points corresponding to the four corners of the image showing the region R1 included in the captured image. In order to detect a plurality of points, it is preferable that the image of the range including the region R1 is captured in a state where the projection light hits not only the inside of the region R1 but also a thick straight line (e.g., a screen frame) showing the region R1. This improves the brightness of the thick straight line, e.g., the screen frame, making it easier to detect a plurality of points related to the screen frame.

[0020] Furthermore, the projector 1 controls the projection device 16 to display on the wall surface W1 a corner image GC2-1 indicating the position of corner CN1-1, a corner image GC2-2 indicating the position of corner CN1-2, a corner image GC2-3 indicating the position of corner CN1-3, and a corner image GC2-4 indicating the position of corner CN1-4. When there is no need to distinguish between the corner images GC2-1 to GC2-4, the corner images GC2-1 to GC2-4 may be referred to as "corner images GC2."

[0021] Moreover, the projector 1 controls the projection device 16 to display on the wall surface W1 a plurality of instruction images GD corresponding one-to-one to a plurality of points detected from a captured image representing a result of capturing an area including the region R1 in which the projection image GP2-2 is displayed. The plurality of instruction images GD are images for showing the user the relative positional relationship between the plurality of points detected from the captured image and an image showing the region R1 included in the captured image. The plurality of instruction images GD include one or more instruction images GD corresponding to the corner CN1-1, one or more instruction images GD corresponding to the corner CN1-2, one or more instruction images GD corresponding to the corner CN1-3, and one or more instruction images GD corresponding to the corner CN1-4. The one or more instruction images GD corresponding to the corner CN1-1 include an instruction image GD-2. The one or more instruction images GD corresponding to the corner CN1-4 include an instruction image GD-1. The display mode of the instruction images GD is different from the display mode of the pointer image GC1.

[0022] The user executes an operation to select one corner image GC2 from among the corner images GC2-1 to GC2-4. In other words, the user executes an operation to select a corner of the region R1 indicated by one corner image GC2 from among the corner images GC2-1 to GC2-4. The projector 1 selects one corner image GC2 from among the corner images GC2-1 to GC2-4 based on the operation from the user. For example, the user executes an operation to select a corner image GC2-4 from among the corner images GC2-1 to GC2-4. In other words, the user executes an operation to select a corner CN1-4 indicated by the corner image GC2-4 from among the corner images GC2-1 to GC2-4. The projector 1 selects the corner image GC2-4 from among the corner images GC2-1 to GC2-4 based on the operation from the user. Note that the projector 1 may change the display mode of one corner image GC2 selected by the user. For example, when the corner image GC2-4 is selected by the user, the projector 1 may cause the display mode of the corner image GC2-4 to differ from the display modes of the other corner images GC2-1 to GC2-3 by controlling the projection device 16. This allows the user to easily confirm the currently selected corner image GC2.

[0023] When one corner image GC2 is selected from among the corner images GC2-1 to GC2-4, the user executes an operation to determine one instruction image GD that is a candidate for selection from among one or more instruction images GD corresponding to the corner of the region R1 indicated by the one corner image GC2. The projector 1 determines one instruction image GD that is a candidate for selection based on an operation from the user. In addition, the projector 1 controls the projection device 16 to display the one instruction image GD in a manner distinguishable from other instruction images GD. For example, when the corner image GC2-4 is selected from among the corner images GC2-1 to GC2-4, and the instruction image GD-1 is determined as one instruction image GD that is a candidate for selection from among one or more instruction images GD corresponding to the corner CN1-4 indicated by the corner image GC2-4, the projector 1 controls the projection device 16 to display the instruction image GD-1 in a manner distinguishable from other instruction images GD. Specifically, the projector 1 controls the projection device 16 to change the display mode of the instruction image GD-1 over time. More specifically, the projector 1 displays the instruction image GD-1 so that it can be distinguished from the other instruction images GD by repeatedly blinking the instruction image GD-1.

[0024] When one of the multiple instruction images GD is displayed in a manner distinguishable from the other instruction images GD, the user executes an operation to select the one instruction image GD. The projector 1 selects a point corresponding to the one instruction image GD and detected from a captured image representing the result of capturing an area including the area R1 in which the projection image GP2-2 is displayed, based on an operation from the user. Furthermore, the projector 1 controls the projection device 16 to change the display mode of the one instruction image GD. For example, when an instruction image GD-2 of the multiple instruction images GD is displayed in a manner distinguishable from the other instruction images GD, the user executes an operation to select the instruction image GD-2. The projector 1 selects a point corresponding to the instruction image GD-2 and detected from a captured image representing the result of capturing an area including the area R1 in which the projection image GP2-2 is displayed. Furthermore, the projector 1 controls the projection device 16 to change the display mode of the instruction image GD-2. Specifically, the projector 1 controls the projection device 16 to change the color of the instruction image GD-2. This allows the user to easily confirm that a point corresponding to the instruction image GD whose display mode has been changed has been selected.

[0025] In addition, when a point corresponding to one instruction image GD is selected based on an operation from a user who selects one instruction image GD displayed in a manner distinguishable from other instruction images GD, the user may execute an operation to determine another instruction image GD different from the one instruction image GD as a selection candidate. The projector 1 determines the other instruction image GD as a selection candidate based on an operation from the user. That is, the projector 1 controls the projection device 16 to display the other instruction image GD in a manner distinguishable from other instruction images GD. Also, the user may execute an operation to select the other instruction image GD displayed in a manner distinguishable from other instruction images GD. The projector 1 selects a point corresponding to the other instruction image GD and detected from a captured image representing a result of capturing an area including the area R1 in which the projection image GP2-2 is displayed. Also, the projector 1 controls the projection device 16 to change the display mode of the other instruction image GD. That is, the user selects one or more instruction images GD from one or more instruction images GD corresponding to a corner of the area R1 indicated by one corner image GC2 selected from the corner images GC2-1 to GC2-4. The projector 1 selects one or more points that correspond one-to-one with one or more instruction images GD selected by the user and that are detected from an captured image that represents the result of capturing an area including the area R1 in which the projected image GP2-2 is displayed.

[0026] When the selection of one or more points is completed from among the one or more points corresponding one-to-one to the one or more instruction images GD corresponding to the corners of the region R1 indicated by one of the corner images GC2 selected from the corner images GC2-1 to GC2-4, the projector 1 controls the projection device 16 to change the display mode of the one corner image GC2. For example, when the corner image GC2-1 is selected from the corner images GC2-1 to GC2-4, and the selection of one or more points is completed from among the one or more points corresponding one-to-one to the one or more instruction images GD corresponding to the corner CN1-1 indicated by the corner image GC2-1, the projector 1 controls the projection device 16 to change the display mode of the corner image GC2-1. Specifically, the projector 1 changes the color of the corner image GC2-1 by controlling the projection device 16. This allows the user to easily confirm that one or more points have been selected from among the one or more points corresponding one-to-one to the one or more instruction images GD corresponding to the corners of the region R1 indicated by the corner image GC2 whose display mode has been changed.

[0027] The user selects one or more instruction images GD at each of the four corners of the region R1. That is, the user selects one or more instruction images GD from one or more instruction images GD corresponding to the corner CN1-1 indicated by the corner image GC2-1. The user also selects one or more instruction images GD from one or more instruction images GD corresponding to the corner CN1-2 indicated by the corner image GC2-2. The user also selects one or more instruction images GD from one or more instruction images GD corresponding to the corner CN1-3 indicated by the corner image GC2-3. The user also selects one or more instruction images GD from one or more instruction images GD corresponding to the corner CN1-4 indicated by the corner image GC2-4. Therefore, the projector 1 selects four or more points that correspond one-to-one to the four or more instruction images GD selected by the user and are detected from the captured image that represents the result of capturing the range including the region R1 in which the projected image GP2-2 is displayed.

[0028] 4 is a schematic diagram illustrating how the projection image GP1 is displayed in the fourth period. The fourth period is a period after the third period. Specifically, the fourth period is a period after the user selects one or more instruction images GD at each of the four corners of the region R1.

[0029] In the fourth period, the position, orientation, angle, etc. of the projector 1 are assumed to be different from the position, orientation, angle, etc. of the projector 1 in the first period, the second period, and the third period. Such a change in the position, orientation, angle, etc. of the projector 1 may occur due to, for example, an unintentional contact of the user with the projector 1. That is, the projector 1 displays the projected image GP1 by projecting the projection light to an area R3 different from the area R2 of the wall surface W1 onto which the projection light is projected in the first period, the second period, and the third period. The area R3 includes the area R1. The projected image GP1 in the fourth period may be referred to as a "projected image GP1-2."

[0030] In the fourth period, the projector 1 controls the imaging device 14 to capture an image of a range including the region R3 where the projection image GP1-2 is displayed. That is, the projector 1 captures an image of a range including the region R1. The projector 1 also generates information indicating a correspondence relationship between a plurality of pixels included in the optical modulator 160 and a plurality of pixels included in the imaging element 140, based on a captured image showing a result of capturing an image of a range including the region R3 where the projection image GP1-2 is displayed and an image projected from the projector 1 when the projection image GP1-2 is displayed.

[0031] FIG. 5 is a schematic diagram illustrating a state in which the projected image GP2 is displayed in the fifth period. The fifth period is a period after the fourth period. The projector 1 controls the projection device 16 to display the projected image GP2 on the wall surface W1. Due to changes in the position, direction, angle, etc. of the projector 1, the position where the projected image GP2 is displayed in the fifth period differs from the position where the projected image GP2 was displayed in the third period. In addition, the shape of the projected image GP2 in the fifth period differs from the shape of the projected image GP2 in the third period. Note that the projected image GP2 in the fifth period may be referred to as the "projected image GP2-3". That is, the position where the projected image GP2-3 is displayed differs from the position where the projected image GP2-2 was displayed. In addition, the shape of the projected image GP2-3 differs from the shape of the projected image GP2-2. In addition, the projected image GP2-3 is displayed in the region R1. That is, the position and shape of the projected image GP2-3 are shifted from the projected image GP2-2 desired by the user.

[0032] In the fifth period, the projector 1 captures an image of the range including the region R1 where the projection image GP2-3 is displayed by controlling the imaging device 14. The projector 1 also detects a plurality of points corresponding to the four corners of the image showing the region R1 included in the captured image by executing image processing on the captured image representing the result of capturing the range including the region R1 where the projection image GP2-3 is displayed.

[0033] The projector 1 generates a position where the projection image GP2 is displayed and a projection transformation matrix for correcting the shape of the projection image GP2 based on information indicating the correspondence between the pixels of the optical modulator 160 and the pixels of the imaging element 140, generated based on the captured image acquired in the first period, coordinates of four or more points selected based on the user's operation among the multiple points detected from the captured image acquired in the third period, information indicating the correspondence between the pixels of the optical modulator 160 and the pixels of the imaging element 140, generated based on the captured image acquired in the fourth period, and coordinates of four or more points that correspond one-to-one to the four or more points selected based on the user's operation among the multiple points detected from the captured image acquired in the fifth period. Here, the four or more points selected by the user in the third period are searched for among the multiple points detected from the captured image in the fifth period. The projector 1 also corrects the projection image GP2 based on the projection transformation matrix. Correcting the projected image using a projective transformation matrix generated based on the coordinates of a plurality of points detected from a captured image is sometimes referred to as "second correction."

[0034] FIG. 6 is a schematic diagram illustrating a state in which the projected image GP2 is displayed in the sixth period. The sixth period is a period after the fifth period. Specifically, the sixth period is a period after the second correction is performed. The projected image GP2 in the sixth period may be referred to as a "projected image GP2-4." The projected image GP2-4 is the projected image GP2 on which the second correction has been performed. Moreover, the projected image GP2-4 is displayed in the region R1.

[0035] The position where the projected image GP2-4 is displayed is the same as the position where the projected image GP2-2 was displayed. Also, the shape of the projected image GP2-4 is the same as the shape of the projected image GP2-2. That is, even if the position, orientation, angle, etc. of the projector 1 change, the projector 1 can make the position where the projected image GP2-3 is displayed and the shape of the projected image GP2-3 similar to those of the projected image GP2-2, that is, can return to the state of the projected image GP2-2, by performing the second correction on the projected image GP2-3. In other words, the projector 1 can maintain the positional relationship between each of the four corners of the region R1 and each of the four corners of the projected image GP2 in the same manner as in the third period by performing the second correction on the projected image GP2. Specifically, by performing the second correction on the projected image GP2 at a certain period, the position of the angle CN2-1 relative to the angle CN1-1, the position of the angle CN2-2 relative to the angle CN1-2, the position of the angle CN2-3 relative to the angle CN1-3, and the position of the angle CN2-4 relative to the angle CN1-4 can be maintained almost constant. Therefore, by performing the second correction on the current (fifth period) projected image GP2, the state of the current projected image GP2 can be almost restored to the state of the previous (third period) projected image GP2. In addition, the projector 1 performs the first correction based on the operation from the user, thereby adjusting the shape of the projected image GP2-2, which is the reference of the projected image GP2 displayed when the second correction is performed, to a shape desired by the user. In other words, by performing the second correction on the projected image GP2 on which the first correction has been performed at a certain period, the user can easily adjust the shape of the projected image GP2-2 without accurately adjusting the position, direction, angle, etc. of the projector 1.

[0036] 1.2.Projector configuration and functions The configuration and functions of the projector 1 according to the first embodiment will be described below with reference to FIGS.

[0037] FIG. 7 is a block diagram showing the configuration of the projector 1 according to the first embodiment. The projector 1 includes a storage device 10 that stores various information, a processing device 12 that controls the operation of the projector 1, an imaging device 14 that captures an image of a range including a predetermined area of ​​the projection surface, a projection device 16 that projects projection light onto the projection surface, and an operation device 18 that accepts an input operation from a user. The processing device 12 has functions as a projection control unit 120, an imaging control unit 121, a detection unit 122, a correction unit 123, a coordinate management unit 124, and a matrix generation unit 125. As described above, the imaging device 14 includes an imaging element 140 and an imaging lens 142. Also, the projection device 16 includes a light source (not shown), a light modulator 160, and a projection lens 162, as described above.

[0038] The storage device 10 includes, for example, a volatile memory such as a RAM and a non-volatile memory such as a ROM. Here, RAM is an abbreviation for Random Access Memory, and ROM is an abbreviation for Read Only Memory.

[0039] 8 is a block diagram showing the configuration of the storage device 10 according to the first embodiment. The non-volatile memory of the storage device 10 stores a program 100 that defines the operation of the projector 1, projection image information 101 that represents an image projected onto the projection surface, captured image information 104 that represents the result of capturing an image of an area including the area on the projection surface on which the projection image is displayed, coordinate information 109 that represents the coordinates of points included in various images, and a projection transformation matrix 115 that is used for two-dimensional coordinate transformation, image correction, and the like.

[0040] The projection image information 101 includes first projection image information 102 representing an image projected when the projection image GP1 is displayed, and second projection image information 103 representing an image projected when the projection image GP2 is displayed.

[0041] The captured image information 104 includes first captured image information 105 representing captured images acquired in a first period, second captured image information 106 representing captured images acquired in a third period, third captured image information 107 representing captured images acquired in a fourth period, and fourth captured image information 108 representing captured images acquired in a fifth period.

[0042] The coordinate information 109 includes first coordinate information 110 representing the coordinates of multiple points included in the image represented by the first projection image information 102, second coordinate information 111 representing the coordinates of multiple points detected from the image represented by the first captured image information 105, third coordinate information 112 representing the coordinates of multiple points detected from the image represented by the second captured image information 106, fourth coordinate information 113 representing the coordinates of multiple points detected from the image represented by the third captured image information 107, and fifth coordinate information 114 representing the coordinates of multiple points detected from the image represented by the fourth captured image information 108.

[0043] The projective transformation matrix 115 includes a first projective transformation matrix 116 representing the correspondence between the multiple pixels of the optical modulator 160 in the first period and the multiple pixels of the image sensor 140 in the first period, a second projective transformation matrix 117 representing the correspondence between the multiple pixels of the optical modulator 160 in the fourth period and the multiple pixels of the image sensor 140 in the fourth period, and a third projective transformation matrix 118 used for the second correction.

[0044] The volatile memory of the storage device 10 is used by the processing device 12 as a work area when the program 100 is executed.

[0045] A part or all of the storage device 10 may be provided in an external storage device or an external server, etc. A part or all of the various information stored in the storage device 10 may be stored in advance in the storage device 10, or may be obtained from an external storage device or an external server, etc.

[0046] 7, the processing device 12 includes one or more CPUs. However, the processing device 12 may include a programmable logic device such as an FPGA instead of or in addition to a CPU. Here, CPU is an abbreviation for Central Processing Unit, and FPGA is an abbreviation for Field-Programmable Gate Array.

[0047] The processing device 12 functions as a projection control unit 120, an imaging control unit 121, a detection unit 122, a correction unit 123, a coordinate management unit 124, and a matrix generation unit 125 shown in FIG.

[0048] The projection control unit 120 controls the projection device to project projection light for displaying an image onto the projection surface. Specifically, the projection control unit 120 causes the projection device to project projection light based on the projection image information 101, thereby displaying a projection image on the projection surface. In other words, the projection control unit 120 causes the projection device to project an image indicated by the projection image information 101, thereby displaying the projection image on the projection surface. Furthermore, the projection control unit 120 controls the projection device to display an image for assisting the user's operation on the projection surface.

[0049] In this embodiment, the projection control unit 120 controls the projection device 16 to project light for displaying an image onto the wall surface W1.

[0050] Specifically, the projection control unit 120 causes the projection device 16 to project projection light based on the projection image information 101, thereby displaying a projection image on the wall surface W1. More specifically, the projection control unit 120 causes the projection device 16 to project projection light based on the first projection image information 102, thereby displaying a projection image GP1 on the wall surface W1. In addition, the projection control unit 120 causes the projection device 16 to project projection light based on the second projection image information 103, thereby displaying a projection image GP2 on the wall surface W1.

[0051] Furthermore, the projection control unit 120 controls the projection device 16 to display a pointer image GC1 located at a corner of the projection image GP2 on the wall surface W1.

[0052] Furthermore, the projection control unit 120 controls the projection device 16 to display a corner image GC2 located at a corner of the area R1 on the wall surface W1 on the wall surface W1. Furthermore, the projection control unit 120 changes the display mode of the corner image GC2 based on an operation by the user.

[0053] Moreover, the projection control unit 120 controls the projection device 16 to display on the wall surface W1 a plurality of instruction images GD each corresponding to a plurality of points detected from the image represented by the second captured image information 106. Furthermore, the projection control unit 120 changes the display mode of the instruction image GD based on an operation by the user.

[0054] 9 is a schematic diagram illustrating an example of an image indicated by the first projection image information 102. In this embodiment, the first projection image information 102 indicates an image GF1. The projection control unit 120 causes the projection device 16 to project projection light based on the first projection image information 102, thereby causing the projection image GP1 to be displayed on the wall surface W1. In other words, the projection control unit 120 causes the projection device 16 to project the image GF1 indicated by the first projection image information 102, thereby causing the projection image GP1 to be displayed on the wall surface W1.

[0055] The image GF1 includes a plurality of points that correspond one-to-one to the plurality of points included in the projected image GP1.

[0056] 7, the imaging control unit 121 controls the imaging device to capture an image of a range including an area on the projection surface on which the projection image is displayed. The imaging control unit 121 also acquires a captured image representing the result of the imaging from the imaging device. The imaging control unit 121 also stores captured image information 104 representing the acquired captured image in the storage device 10.

[0057] In this embodiment, the imaging control unit 121 controls the imaging device 14 to capture an image of a range including the area R2 on the wall surface W1 on which the projection image GP1-1 is displayed. The imaging control unit 121 also acquires a captured image representing the result of the imaging from the imaging device 14. The imaging control unit 121 also stores in the storage device 10 first captured image information 105 representing the acquired captured image.

[0058] The imaging control unit 121 also controls the imaging device 14 to capture an image of a range including the area R3 on the wall surface W1 where the projection image GP1-2 is displayed. The imaging control unit 121 also acquires a captured image representing the result of the imaging from the imaging device 14. The imaging control unit 121 also stores third captured image information 107 representing the acquired captured image in the storage device 10.

[0059] The imaging control unit 121 also controls the imaging device 14 to capture an image of a range including the area R1 on the wall surface W1 where the projection image GP2-2 is displayed. The imaging control unit 121 also acquires a captured image representing the result of the imaging from the imaging device 14. The imaging control unit 121 also stores, in the storage device 10, second captured image information 106 representing the acquired captured image.

[0060] The imaging control unit 121 also controls the imaging device 14 to capture an image of a range including the area R1 on the wall surface W1 on which the projection image GP2-3 is displayed. The imaging control unit 121 also acquires a captured image representing the result of the imaging from the imaging device 14. The imaging control unit 121 also stores, in the storage device 10, fourth captured image information 108 representing the acquired captured image.

[0061] The detection unit 122 detects points included in an image by executing image processing on the image indicated by various image information. That is, the detection unit 122 acquires coordinate information 109 indicating the coordinates of the detected points. The detection unit 122 also stores the acquired coordinate information 109 in the storage device 10.

[0062] In this embodiment, the detection unit 122 detects a plurality of points included in the image represented by the captured image information 104 by executing image processing on the image represented by the captured image information 104. That is, the detection unit 122 acquires coordinate information 109 indicating the coordinates of a plurality of points included in the image represented by the captured image information 104. Furthermore, the detection unit 122 stores the acquired coordinate information 109 in the storage device 10.

[0063] Specifically, the detection unit 122 detects a plurality of points included in the image represented by the first captured image information 105 by executing image processing on the image represented by the first captured image information 105. That is, the detection unit 122 acquires second coordinate information 111 representing the coordinates of a plurality of points included in the image represented by the first captured image information 105. Furthermore, the detection unit 122 stores the acquired second coordinate information 111 in the storage device 10.

[0064] Moreover, the detection unit 122 detects a plurality of points included in the image represented by the second captured image information 106 by executing image processing on the image represented by the second captured image information 106. That is, the detection unit 122 acquires third coordinate information 112 representing the coordinates of a plurality of points included in the image represented by the second captured image information 106. Moreover, the detection unit 122 stores the acquired third coordinate information 112 in the storage device 10.

[0065] Furthermore, the detection unit 122 detects a plurality of points included in the image represented by the third captured image information 107 by executing image processing on the image represented by the third captured image information 107. That is, the detection unit 122 acquires fourth coordinate information 113 representing the coordinates of a plurality of points included in the image represented by the third captured image information 107. Furthermore, the detection unit 122 stores the acquired fourth coordinate information 113 in the storage device 10.

[0066] Moreover, the detection unit 122 detects a plurality of points included in the image represented by the fourth captured image information 108 by executing image processing on the image represented by the fourth captured image information 108. That is, the detection unit 122 acquires fifth coordinate information 114 representing the coordinates of a plurality of points included in the image represented by the fourth captured image information 108. Moreover, the detection unit 122 stores the acquired fifth coordinate information 114 in the storage device 10.

[0067] The detection unit 122 may detect a plurality of points included in the image GF1 by executing image processing on the image GF1 represented by the first projection image information 102. When the detection unit 122 detects a plurality of points included in the image GF1, the detection unit 122 acquires first coordinate information 110 representing the coordinates of the plurality of points included in the image GF1 represented by the first projection image information 102. Furthermore, the detection unit 122 stores the acquired first coordinate information 110 in the storage device 10. Note that in this embodiment, it is assumed that the first coordinate information 110 is stored in advance in the storage device 10.

[0068] In the function related to the point detection, a known image processing technique may be used. Examples of known image processing techniques related to the point detection include template matching, center of gravity detection, and an algorithm called "AKAZE". In this specification, a detailed technical description of the point detection is omitted.

[0069] The coordinate management unit 124 manages the coordinates of points detected from the captured image based on an operation from the user, and also executes various processes related to the management of the coordinates of points detected from the captured image.

[0070] In this embodiment, the coordinate management unit 124 selects one corner image GC2 from the corner images GC2-1 to GC2-4 based on an operation from the user. The coordinate management unit 124 also determines whether or not an operation related to management of the coordinates of a point detected from a captured image has been received from the user.

[0071] Furthermore, based on a user's operation to select one instruction image GD that is displayed on the wall surface W1 so as to be distinguishable from other instruction images GD, the coordinate management unit 124 selects a point that corresponds to the one instruction image GD and that is detected from the image indicated by the second captured image information 106. Furthermore, the coordinate management unit 124 updates the third coordinate information 112 based on the user's operation.

[0072] The matrix generation unit 125 generates the projective transformation matrix 115 based on the coordinates of a plurality of corresponding points. That is, the matrix generation unit 125 generates the projective transformation matrix 115 based on the coordinate information 109.

[0073] In this embodiment, the matrix generation unit 125 generates the first projective transformation matrix 116 based on the first coordinate information 110 and the second coordinate information 111. In addition, the matrix generation unit 125 stores the generated first projective transformation matrix 116 in the storage device 10.

[0074] The first projective transformation matrix 116 is information indicating a correspondence relationship between a plurality of pixels included in the light modulator 160 in the first period and a plurality of pixels included in the image sensor 140 in the first period. Specifically, the first projective transformation matrix 116 is a projective transformation matrix for converting the coordinates of the pixels included in the image sensor 140 in the first period into the coordinates of the pixels included in the light modulator 160 in the first period. In other words, the first projective transformation matrix 116 is a projective transformation matrix for converting the coordinates of the pixels included in the image sensor 140 before the position, orientation, angle, etc. of the projector 1 change into the coordinates of the pixels included in the light modulator 160 before the position, orientation, angle, etc. of the projector 1 change.

[0075] Furthermore, the matrix generation unit 125 generates the second projective transformation matrix 117 based on the first coordinate information 110 and the fourth coordinate information 113. Furthermore, the matrix generation unit 125 causes the storage device 10 to store the generated second projective transformation matrix 117.

[0076] The second projective transformation matrix 117 is information indicating a correspondence relationship between a plurality of pixels included in the light modulator 160 in the fourth period and a plurality of pixels included in the image sensor 140 in the fourth period. Specifically, the second projective transformation matrix 117 is a projective transformation matrix for converting the coordinates of the pixels included in the image sensor 140 in the fourth period into the coordinates of the pixels included in the light modulator 160 in the fourth period. In other words, the second projective transformation matrix 117 is a projective transformation matrix for converting the coordinates of the pixels included in the image sensor 140 after the position, orientation, angle, etc. of the projector 1 have changed into the coordinates of the pixels included in the light modulator 160 after the position, orientation, angle, etc. of the projector 1 have changed.

[0077] Moreover, the matrix generation unit 125 generates a third projective transformation matrix 118 based on the third coordinate information 112, the first projective transformation matrix 116, the fifth coordinate information 114, and the second projective transformation matrix 117. Moreover, the matrix generation unit 125 causes the storage device 10 to store the generated third projective transformation matrix 118.

[0078] The third projective transformation matrix 118 is information indicating a correspondence relationship between a plurality of pixels included in the light modulator 160 in the third period and a plurality of pixels included in the light modulator 160 in the fifth period. Specifically, the third projective transformation matrix 118 is a projective transformation matrix for converting the coordinates of the pixels included in the light modulator 160 in the third period into the coordinates of the pixels included in the light modulator 160 in the fifth period. In other words, the third projective transformation matrix 118 is a projective transformation matrix for converting the coordinates of the pixels included in the light modulator 160 before the position, orientation, angle, etc. of the projector 1 change into the coordinates of the pixels included in the light modulator 160 after the position, orientation, angle, etc. of the projector 1 change.

[0079] The correction unit 123 corrects the position and shape of the projection image displayed on the projection surface. Specifically, the correction unit 123 executes a first correction based on an operation from a user. The correction unit 123 also executes a second correction based on the projective transformation matrix 115 generated by the matrix generation unit 125.

[0080] In this embodiment, the correction unit 123 executes a first correction for correcting the projection image GP2 by changing the position of a corner of the projection image GP2 based on an operation from a user who moves the pointer image GC1. In other words, the correction unit 123 updates the second projection image information 103 representing an image to be projected when the projection image GP2 is displayed based on an operation from a user who moves the pointer image GC1.

[0081] Moreover, the correction unit 123 executes a second correction for correcting the projection image GP2 based on the third projective transformation matrix 118. In other words, the correction unit 123 updates the second projection image information 103 representing the image projected when the projection image GP2 is displayed based on the third projective transformation matrix 118.

[0082] 10 is a schematic diagram showing an example of an image represented by the first captured image information 105. In this embodiment, the first captured image information 105 represents a captured image GS1.

[0083] The captured image GS1 includes an image GS11. The image GS11 is an image representing the wall surface W1. The image GS11 includes an image GV11 and an image GV12.

[0084] Image GV11 is an image representing region R1. Image GV11 has angle CN3-1, angle CN3-2, angle CN3-3, and angle CN3-4. Angle CN3-1 corresponds to angle CN1-1. Angle CN3-2 corresponds to angle CN1-2. Angle CN3-3 corresponds to angle CN1-3. Angle CN3-4 corresponds to angle CN1-4.

[0085] The image GV12 is an image that represents the projection image GP1-1. The image GV12 includes a plurality of points. The second coordinate information 111 represents the coordinates of the plurality of points included in the image GV12. The plurality of points included in the image GV12 correspond one-to-one to the plurality of points included in the projection image GP1-1. In addition, the plurality of points included in the image GV12 correspond one-to-one to the plurality of points included in the image GF1.

[0086] 11 is a schematic diagram showing an example of an image represented by the second captured image information 106. In this embodiment, the second captured image information 106 represents a captured image GS2.

[0087] The captured image GS2 includes an image GS21. The image GS21 is an image representing the wall surface W1. The image GS21 includes an image GV21 and an image GV22.

[0088] Image GV21 is an image representing region R1. Image GV21 has corners CN4-1, CN4-2, CN4-3, and CN4-4. Angle CN4-1 corresponds to angle CN1-1. That is, angle CN4-1 corresponds to corner image GC2-1. Angle CN4-2 corresponds to angle CN1-2. That is, angle CN4-2 corresponds to corner image GC2-2. Angle CN4-3 corresponds to angle CN1-3. That is, angle CN4-3 corresponds to corner image GC2-3. Angle CN4-4 corresponds to angle CN1-4. That is, angle CN4-4 corresponds to corner image GC2-4. Image GV22 is an image representing the projected image GP2-2.

[0089] The captured image GS2 includes a plurality of points D1. The plurality of points D1 correspond one-to-one to the plurality of instruction images GD. The plurality of points D1 also includes one or more points D1 corresponding to the corner CN4-1, one or more points D1 corresponding to the corner CN4-2, one or more points D1 corresponding to the corner CN4-3, and one or more points D1 corresponding to the corner CN4-4. The plurality of points D1 also includes a point D1-1. The point D1-1 corresponds to the instruction image GD-1. For example, when the user executes an operation to select the instruction image GD-1, the coordinate management unit 124 selects the point D1-1 corresponding to the instruction image GD-1. The coordinate management unit 124 also updates the third coordinate information 112 based on the operation from the user. The third coordinate information 112 updated based on the operation from the user represents the coordinates of the point D1 selected based on the operation from the user.

[0090] 12 is a schematic diagram showing an example of an image indicated by the third captured image information 107. In this embodiment, the third captured image information 107 indicates a captured image GS3.

[0091] The captured image GS3 includes an image GS31. The image GS31 is an image representing the wall surface W1. The image GS31 includes an image GV31 and an image GV32.

[0092] Image GV31 is an image representing region R1. Image GV31 has angle CN5-1, angle CN5-2, angle CN5-3, and angle CN5-4. Angle CN5-1 corresponds to angle CN1-1. Angle CN5-2 corresponds to angle CN1-2. Angle CN5-3 corresponds to angle CN1-3. Angle CN5-4 corresponds to angle CN1-4.

[0093] The image GV32 is an image representing the projection image GP1-2. The image GV32 includes a plurality of points. The fourth coordinate information 113 represents the coordinates of the plurality of points included in the image GV32. The plurality of points included in the image GV32 correspond one-to-one to the plurality of points included in the projection image GP1-2. In addition, the plurality of points included in the image GV32 correspond one-to-one to the plurality of points included in the image GF1.

[0094] 13 is a schematic diagram showing an example of an image represented by the fourth captured image information 108. In the present embodiment, the fourth captured image information 108 represents a captured image GS4.

[0095] The captured image GS4 includes an image GS41. The image GS41 is an image representing the wall surface W1. The image GS41 includes an image GV41 and an image GV42.

[0096] Image GV41 is an image representing region R1. Image GV41 has angle CN6-1, angle CN6-2, angle CN6-3, and angle CN6-4. Angle CN6-1 corresponds to angle CN1-1. Angle CN6-2 corresponds to angle CN1-2. Angle CN6-3 corresponds to angle CN1-3. Angle CN6-4 corresponds to angle CN1-4. Image GV42 is an image representing projected image GP2-3.

[0097] The captured image GS4 includes a plurality of points D2. The plurality of points D2 further includes one or more points D2 corresponding to the corner CN6-1, one or more points D2 corresponding to the corner CN6-2, one or more points D2 corresponding to the corner CN6-3, and one or more points D2 corresponding to the corner CN6-4. The fifth coordinate information 114 represents the coordinates of the plurality of points D2.

[0098] FIG. 14 is a schematic diagram for explaining the second correction. FIG. 14 shows images displayed on the light modulator 160 before and after the second correction. The image GF2-2 is an image displayed on the light modulator 160 in the third period and the fifth period. That is, the image GF2-2 is an image indicated by the second projection image information 103 updated by the first correction. The image GF2-2 is an image before the second correction is performed. The image GF2-4 is an image displayed on the light modulator 160 in the sixth period. That is, the image GF2-4 is an image indicated by the second projection image information 103 updated by the first correction and the second correction. The correction unit 123 corrects the image GF2-2 to the image GF2-4 based on the third projective transformation matrix 118.

[0099] The point D3-1 is a point obtained by transforming the coordinates of a point located at the corner CN4-1 of the image GV21 included in the captured image GS2 by the first projective transformation matrix 116. Similarly, the point D3-2 is a point obtained by transforming the coordinates of a point located at the corner CN4-2 of the image GV21 included in the captured image GS2 by the first projective transformation matrix 116. Moreover, the point D3-3 is a point obtained by transforming the coordinates of a point located at the corner CN4-3 of the image GV21 included in the captured image GS2 by the first projective transformation matrix 116. Moreover, the point D3-4 is a point obtained by transforming the coordinates of a point located at the corner CN4-4 of the image GV21 included in the captured image GS2 by the first projective transformation matrix 116. The region R4 is a region surrounded by a quadrangle having the points D3-1, D3-2, D3-3, and D3-4 as vertices. That is, the region R4 is a region on the light modulator 160 that corresponds to the region R1 in the third period represented by the image GV21. The image GF2-2 is displayed inside the region R4.

[0100] The point D4-1 is a point obtained by transforming the coordinates of a point located at the corner CN6-1 of the image GV41 included in the captured image GS4 by the second projective transformation matrix 117. Similarly, the point D4-2 is a point obtained by transforming the coordinates of a point located at the corner CN6-2 of the image GV41 included in the captured image GS4 by the second projective transformation matrix 117. Moreover, the point D4-3 is a point obtained by transforming the coordinates of a point located at the corner CN6-3 of the image GV41 included in the captured image GS4 by the second projective transformation matrix 117. Moreover, the point D4-4 is a point obtained by transforming the coordinates of a point located at the corner CN6-4 of the image GV41 included in the captured image GS4 by the second projective transformation matrix 117. The region R5 is a region surrounded by a quadrangle having the points D4-1, D4-2, D4-3, and D4-4 as vertices. That is, the region R5 is a region on the light modulator 160 that corresponds to the region R1 in the fifth period shown by the image GV41.

[0101] The third projective transformation matrix 118 used in the second correction is a matrix that converts the coordinates of point D3-1 to those of point D4-1, the coordinates of point D3-2 to those of point D4-2, the coordinates of point D3-3 to those of point D4-3, and the coordinates of point D3-4 to those of point D4-4. Therefore, when image GF2-2 is corrected to image GF2-4 based on the third projective transformation matrix 118, the positional relationship between the four corners of image GF2-2 and the four corners of region R4 is carried over to the positional relationship between the four corners of image GF2-4 and the four corners of region R5. As a result, in the sixth period, when the projected image GP2-4 is displayed on the wall surface W1 by displaying the image GF2-4 on the optical modulator 160, the positional relationship between the angles CN1-1 to CN1-4 of the region R1 and the angles CN2-1 to CN2-4 of the projected image GP2 is maintained in the same manner as in the third period.

[0102] 7, the imaging element 140 is an image sensor, for example, a CCD or a CMOS, etc. Here, CCD is an abbreviation for Charge Coupled Device, and CMOS is an abbreviation for Complementary Metal Oxide Semiconductor.

[0103] The imaging device 14 captures an image of an area including the area on the projection surface on which the projection image is displayed under the control of the imaging control unit 121. The imaging device 14 also outputs captured image information 104 representing the result of capturing an image of the area including the area on the projection surface on which the projection image is displayed to the processing device 12. In other words, the imaging device 14 outputs the captured image indicated by the captured image information 104 to the processing device 12.

[0104] The optical modulator 160 includes, for example, one or more liquid crystal panels. The optical modulator 160 may include a DMD instead of a liquid crystal panel. The optical modulator 160 modulates light emitted from a light source into projection light for displaying a projection image on a projection surface based on a signal input from the processing device 12. The light source includes, for example, a halogen lamp, a xenon lamp, an extra-high pressure mercury lamp, an LED, or a laser light source. Here, LED is an abbreviation for Light Emitting Diode, and DMD is an abbreviation for Digital Mirror Device.

[0105] The projection device 16 projects projection light for displaying a projection image on a projection surface under the control of the projection control unit 120. In other words, the projection device 16 projects an image input from the processing device 12 onto the projection surface.

[0106] The operation device 18 accepts an input operation on the projector 1 from a user of the projector 1. The operation device 18 includes, for example, a touch panel or operation buttons provided on the housing of the projector 1. When the operation device 18 includes a touch panel, the operation device 18 outputs data indicating the detected touch position to the processing device 12. When the operation device 18 includes operation buttons, the operation device 18 outputs data identifying the pressed button to the processing device 12. The operation device 18 may also include a receiving device for receiving an operation signal output from a remote controller based on a user's operation. When the operation device 18 includes a receiving device, the operation device 18 outputs data indicating the operation signal received from the remote controller to the processing device 12. As a result, the content of the input operation on the projector 1 is transmitted to the processing device 12.

[0107] 1.3.Projector Operation Hereinafter, the operation of the projector 1 according to the first embodiment will be described with reference to FIGS.

[0108] Fig. 15 is a flowchart for explaining the operation of the projector 1 in a time period including the first period to the third period. The series of operations shown in the flowchart of Fig. 15 are operations for preparation for the second correction. The series of operations shown in the flowchart of Fig. 15 are started, for example, when the projector 1 is turned on and the operation device 18 receives an input operation for starting the operation from the user of the projector 1.

[0109] In step S101, the projection control unit 120 causes the projection device 16 to project projection light based on the first projection image information 102, thereby displaying the projection image GP1-1 in the region R2 on the wall surface W1.

[0110] In step S102, the imaging control unit 121 controls the imaging device 14 to capture an image of a range including the area R2 on the wall surface W1 on which the projection image GP1-1 is displayed. The imaging control unit 121 also acquires a captured image GS1 representing the result of the imaging from the imaging device 14. The imaging control unit 121 also stores in the storage device 10 the first captured image information 105 representing the acquired captured image GS1.

[0111] In step S103, the detection unit 122 detects a plurality of points included in the captured image GS1 by executing image processing on the captured image GS1 indicated by the first captured image information 105. That is, the detection unit 122 acquires second coordinate information 111 representing the coordinates of a plurality of points included in the captured image GS1. The detection unit 122 also stores the acquired second coordinate information 111 in the storage device 10.

[0112] In step S104, the matrix generation unit 125 generates the first projective transformation matrix 116 based on the first coordinate information 110 and the second coordinate information 111. In addition, the matrix generation unit 125 causes the storage device 10 to store the generated first projective transformation matrix 116.

[0113] In step S105, the projection control unit 120 causes the projection device 16 to project projection light based on the second projection image information 103, thereby displaying the projection image GP2-1 in the region R1 of the wall surface W1. The projection control unit 120 also controls the projection device 16 to display a pointer image GC1 located at a corner of the projection image GP2-1 on the wall surface W1. That is, the projection control unit 120 controls the projection device 16 to display a pointer image GC1-1 indicating the position of the corner CN2-1, a pointer image GC1-2 indicating the position of the corner CN2-2, a pointer image GC1-3 indicating the position of the corner CN2-3, and a pointer image GC1-4 indicating the position of the corner CN2-4.

[0114] In step S106, the correction unit 123 executes a first correction to correct the shape of the projection image GP2 by changing the position of the corners of the projection image GP2-1 based on the operation from the user who moves the pointer image GC1. In other words, the correction unit 123 updates the second projection image information 103 based on the operation from the user who moves the pointer image GC1. When the first correction is executed, the projection image GP2-2 is displayed on the wall surface W1.

[0115] In step S107, the imaging control unit 121 controls the imaging device 14 to capture an image of a range including the area R1 on the wall surface W1 where the projection image GP2-2 is displayed. The imaging control unit 121 also acquires a captured image GS2 representing the result of the imaging from the imaging device 14. The imaging control unit 121 also stores, in the storage device 10, second captured image information 106 representing the acquired captured image GS2.

[0116] In step S108, the detection unit 122 detects a plurality of points D1 included in the captured image GS2 by executing image processing on the captured image GS2 indicated by the second captured image information 106. That is, the detection unit 122 acquires third coordinate information 112 representing the coordinates of the plurality of points D1 included in the captured image GS2. The detection unit 122 also stores the acquired third coordinate information 112 in the storage device 10.

[0117] In step S109, the projection control unit 120 controls the projection device 16 to display on the wall surface W1 a corner image GC2 located at a corner of the area R1 on the wall surface W1. That is, the projection control unit 120 controls the projection device 16 to display a corner image GC2-1 indicating the position of the corner CN1-1, a corner image GC2-2 indicating the position of the corner CN1-2, a corner image GC2-3 indicating the position of the corner CN1-3, and a corner image GC2-4 indicating the position of the corner CN1-4. In addition, the projection control unit 120 controls the projection device 16 to display on the wall surface W1 a plurality of instruction images GD corresponding one-to-one to a plurality of points D1.

[0118] In step S110, the coordinate management unit 124 selects one corner image GC2 from among the corner images GC2-1 to GC2-4 based on the user's operation. In other words, the coordinate management unit 124 selects a corner of the region R1 indicated by one corner image GC2 from among the corner images GC2-1 to GC2-4 based on the user's operation.

[0119] In step S111, the projection control unit 120 controls the projection device 16 to repeatedly flash one of the instruction images GD that is a candidate for selection and that is determined from one or more instruction images GD corresponding to the corner of the area R1 indicated by one corner image GC2 selected based on operation by the user, thereby displaying the instruction image GD in a manner that is distinguishable from the other instruction images GD.

[0120] In step S112, the coordinate management unit 124 selects a point D1 corresponding to one of the instruction images GD based on a user's operation to select one of the instruction images displayed in a manner that can be distinguished from the other instruction images GD. The coordinate management unit 124 also updates the third coordinate information 112 based on the user's operation.

[0121] In step S113, the projection control unit 120 controls the projection device 16 to change the color of the instruction image GD corresponding to the point D1 selected based on the operation by the user.

[0122] In step S114, the coordinate management unit 124 judges whether or not a user operation to end the selection of one or more pointing images GD at the corners of the region R1 indicated by one corner image GC2 selected based on the user operation has been received. In other words, the coordinate management unit 124 judges whether or not a user operation to end the selection of one or more points D1 at the corners of the image GV21 corresponding to one corner image GC2 selected based on the user operation has been received. If a user operation to end the selection of one or more points D1 at the corners of the image GV21 corresponding to one corner image GC2 selected based on the user operation has been received, that is, if the answer is YES in step S114, the coordinate management unit 124 advances the process to step S115. If a user operation to end the selection of one or more points D1 at the corners of the image GV21 corresponding to one corner image GC2 selected based on the user operation has not been received, that is, if the answer is NO in step S114, the coordinate management unit 124 advances the process to step S111.

[0123] If the result of the determination in step S114 is NO, the processing device 12 continues the selection of the point D1 based on the operation from the user by executing the operations in steps S111 to S113 again.

[0124] In step S115, the projection control unit 120 controls the projection device 16 to change the color of one corner image GC2 selected based on an operation by the user.

[0125] In step S116, the coordinate management unit 124 determines whether or not an operation from the user to end the selection of the instruction image GD has been received. That is, the coordinate management unit 124 determines whether or not an operation from the user to end the selection of the point D1 has been received. If an operation from the user to end the selection of the point D1 has been received, that is, if YES in step S116, the processing device 12 including the coordinate management unit 124 ends the series of operations shown in the flowchart of Fig. 15. If an operation from the user to end the selection of the point D1 has not been received, that is, if NO in step S116, the coordinate management unit 124 advances the process to step S110.

[0126] The processing device 12 selects four or more points D1 by repeatedly executing the operations in steps S110 to S115 until the result of the determination in step S116 becomes YES. That is, the processing device 12 selects one or more points D1 from one or more points D1 corresponding to the angle CN4-1 corresponding to the angle CN1-1 indicated by the corner image GC2-1, selects one or more points D1 from one or more points D1 corresponding to the angle CN4-2 corresponding to the angle CN1-2 indicated by the corner image GC2-2, selects one or more points D1 from one or more points D1 corresponding to the angle CN4-3 corresponding to the angle CN1-3 indicated by the corner image GC2-3, and selects one or more points D1 from one or more points D1 corresponding to the angle CN4-4 corresponding to the angle CN1-4 indicated by the corner image GC2-4, thereby selecting four or more points D1.

[0127] The processing device 12 may execute a process of thinning out the points D1, for example, after step S108 and before step S109. For example, the number of the points D1 corresponding to the corner CN1-1 is actually several tens, but if all the pointing images GD corresponding to these points are displayed, the user may find it difficult to understand which point D1 to select due to the large number of points. Therefore, the processing device 12 may execute a process of thinning out the points D1 by utilizing the fact that each of the points D1 has a value, i.e., a feature amount, according to the position in the captured image GS2. The feature amount is calculated as a result of executing image processing on the captured image GS2 in step S108 of FIG. 15. For example, the processing device 12 does not display the pointing images GD corresponding to the points D1 having a feature amount less than a predetermined threshold. The points D1 having a feature amount less than a predetermined threshold are components that have a small contribution to the control of the position and shape of the projected image GP, such as points that do not correspond to the corners of the region R1.

[0128] Fig. 16 is a flowchart for explaining the operation of the projector 1 in a time period including the fourth period to the sixth period. The series of operations shown in the flowchart of Fig. 16 are operations for executing the second correction. The series of operations shown in the flowchart of Fig. 16 may be started, for example, at regular time intervals, or may be started when an input operation for starting the operation is received from the user.

[0129] In step S201, the projection control unit 120 causes the projection device 16 to project projection light based on the first projection image information 102, thereby displaying the projection image GP1-2 in the region R3 on the wall surface W1.

[0130] In step S202, the imaging control unit 121 controls the imaging device 14 to capture an image of a range including the area R3 on the wall surface W1 where the projection image GP1-2 is displayed. The imaging control unit 121 also acquires a captured image GS3 representing the result of the imaging from the imaging device 14. The imaging control unit 121 also stores the third captured image information 107 representing the acquired captured image GS3 in the storage device 10.

[0131] In step S203, the detection unit 122 detects a plurality of points included in the captured image GS3 by executing image processing on the captured image GS3 indicated by the third captured image information 107. That is, the detection unit 122 acquires fourth coordinate information 113 representing the coordinates of a plurality of points included in the captured image GS3. The detection unit 122 also stores the acquired fourth coordinate information 113 in the storage device 10.

[0132] In step S204, the matrix generation unit 125 generates the second projective transformation matrix 117 based on the first coordinate information 110 and the fourth coordinate information 113. In addition, the matrix generation unit 125 causes the storage device 10 to store the generated second projective transformation matrix 117.

[0133] In step S205, the projection control unit 120 causes the projection device 16 to project projection light based on the second projection image information 103, thereby displaying the projection image GP2-3 in the region R1 on the wall surface W1.

[0134] In step S206, the imaging control unit 121 controls the imaging device 14 to capture an image of a range including the area R1 on the wall surface W1 where the projection image GP2-3 is displayed. The imaging control unit 121 also acquires a captured image GS4 representing the result of the imaging from the imaging device 14. The imaging control unit 121 also stores in the storage device 10 the fourth captured image information 108 representing the acquired captured image GS4.

[0135] In step S207, the detection unit 122 detects a plurality of points D2 included in the captured image GS4 by executing image processing on the captured image GS4 indicated by the fourth captured image information 108. That is, the detection unit 122 acquires fifth coordinate information 114 indicating the coordinates of the plurality of points D2 included in the captured image GS4. The detection unit 122 also stores the acquired fifth coordinate information 114 in the storage device 10.

[0136] In step S208, the matrix generation unit 125 generates the third projective transformation matrix 118 based on the third coordinate information 112, the first projective transformation matrix 116, the fifth coordinate information 114, and the second projective transformation matrix 117. In addition, the matrix generation unit 125 causes the storage device 10 to store the generated third projective transformation matrix 118.

[0137] After step S207 and before step S208, the processing device 12 may execute a process of searching for points corresponding to the four or more points D1 selected by the user in the third period from among the multiple points D2 detected from the captured image GS4. Specifically, the processing device 12 searches for points having the same feature amount as the four or more points D1 selected by the user from among the multiple points D2. This is because, when the processing device 12 executes the second correction, it is necessary to accurately recognize where the four or more points D1 in the captured image GS2 have moved in the captured image GS4. The processing device 12 utilizes the fact that the point D1 has a feature amount to accurately associate the multiple points D2 in the fifth period with the four or more points D1 in the third period, that is, to perform matching. This suppresses matching errors between the multiple points D2 in the fifth period and the four or more points D1 in the third period, and further improves the generation accuracy of the third projective transformation matrix 118.

[0138] In step S209, the correction unit 123 executes a second correction to correct the projection image GP2-3 based on the third projective transformation matrix 118. In other words, the correction unit 123 updates the second projection image information 103 based on the third projective transformation matrix 118. When the second correction is executed, the projection image GP2-4 is displayed on the wall surface W1.

[0139] As described above, according to the first embodiment, the projector 1 corrects the position and shape of the projection image displayed in the display area by executing the second correction. That is, even if the position, orientation, angle, etc. of the projector 1 change, the projector 1 can maintain the position where the projection image is displayed and the shape of the projection image in the same manner as before the position, orientation, angle, etc. of the projector 1 change. In other words, the projector 1 can maintain the positional relationship between the four corners of the display area and the four corners of the projection image.

[0140] Furthermore, according to the first embodiment, the projector 1 corrects the shape of the projection image displayed in the display area by executing a first correction based on an operation from the user. That is, the projector 1 can display the projection image in a shape desired by the user. Furthermore, the user can easily adjust the shape of the projection image without having to precisely adjust the position, orientation, angle, etc. of the projector 1.

[0141] As described above, the projection method according to the first embodiment includes controlling the projection device 16 to display a projection image GP2 having four corners inside an area R1 on a wall surface W1, the area having four corners; acquiring a captured image GS2 which is an image of a range including the area R1 in which the projection image GP2 is displayed; performing image processing on the captured image GS2 to detect a plurality of points D1 which correspond one-to-one to the four corners of an image GV21 which indicates the area R1 included in the captured image GS2; controlling the projection device 16 to display a pointer image GC1 which is located at the corners of the projection image GP2 and which is used to adjust the shape of the projection image GP2; performing a first correction to correct the shape of the projection image GP2 by changing the positions of the corners of the projection image GP2 based on an operation by a user who moves the pointer image GC1; and performing a second correction to maintain the positional relationship between the four corners of the area R1 and the four corners of the projection image GP2 determined by the first correction, based on the plurality of points D1.

[0142] In addition, the program 100 according to the first embodiment causes the processing device 12 to execute the following operations: by controlling the projection device 16, display a projection image GP2 having four corners inside an area R1 on a wall surface W1, the area R1 having four corners; by acquiring a captured image GS2 which is an image of a range including the area R1 in which the projection image GP2 is displayed; by performing image processing on the captured image GS2, detect a plurality of points D1 which correspond one-to-one to the four corners of an image GV21 which indicates the area R1 included in the captured image GS2; by controlling the projection device 16, display a pointer image GC1 which is located at the corners of the projection image GP2 and which is used to adjust the shape of the projection image GP2; by changing the position of the corners of the projection image GP2 based on an operation by a user who moves the pointer image GC1, execute a first correction for correcting the shape of the projection image GP2; and, based on the plurality of points D1, execute a second correction for maintaining the positional relationship between the four corners of the area R1 and the four corners of the projection image GP2 determined by the first correction.

[0143] That is, the projector 1 performs a first correction based on an operation from the user to correct the shape of the projected image GP2 to a shape desired by the user. Furthermore, the projector 1 performs a second correction to maintain the position where the projected image GP2 is displayed and the shape of the projected image GP2. Therefore, the projector 1 can maintain the shape of the projected image GP2 to a shape desired by the user. Furthermore, the first correction is performed based on an operation by the user. Therefore, the user can easily adjust the shape of the projected image GP2 without having to precisely adjust the position, orientation, angle, etc. of the projector 1.

[0144] In the first embodiment, the projection device 16 is an example of a "projection device", the wall surface W1 is an example of a "projection surface", the region R1 is an example of a "display region", the projected image GP2 is an example of a "projected image", the captured image GS2 is an example of a "first captured image", the image GV21 is an example of a "first image", the point D1 is an example of a "first feature point", the multiple points D1 are an example of a "multiple first feature points", the pointer image GC1 is an example of a "pointer image", the program 100 is an example of a "program", and the processing device 12 is an example of a "processing device". The four corners in the "display region having four corners" are corners CN1-1 to CN1-4 as an example. The four corners in the "projected image having four corners" are corners CN2-1 to CN2-4 as an example. The four corners of the "first image" are corners CN4-1 to CN4-4 as an example.

[0145] Moreover, in the projection method according to the first embodiment, the four corners of the image GV21 include corners CN4-1, CN4-2, CN4-3, and CN4-4, and the multiple points D1 include one or more points D1 corresponding to the angle CN4-1, one or more points D1 corresponding to the angle CN4-2, one or more points D1 corresponding to the angle CN4-3, and one or more points D1 corresponding to the angle CN4-4; by controlling the projection device 16, multiple instruction images GD corresponding one-to-one to the multiple points D1 are displayed on the wall surface W1; by controlling the projection device 16, one instruction image GD of the multiple instruction images GD is displayed on the wall surface W1 so as to be distinguishable from the other instruction images GD; and in response to a user operation of selecting one instruction image GD of the multiple instruction images GD displayed on the wall surface W1, one point D1 corresponding to the selected one instruction image GD is set as a point D1 for performing a second correction; Including,Setting a plurality of points D1 for performing the second correction further includes accepting a user operation to select four or more points D1, including selecting one or more points D1 from the one or more points D1 corresponding to corner CN4-1, selecting one or more points D1 from the one or more points D1 corresponding to corner CN4-2, selecting one or more points D1 from the one or more points D1 corresponding to corner CN4-3, and selecting one or more points D1 from the one or more points D1 corresponding to corner CN4-4; and performing the second correction includes obtaining a captured image GS4 representing the result of capturing an area including an area R1 in which a projection image GP2 on which the first correction has been performed is displayed, performing image processing on the captured image GS4 to detect a plurality of points D2, one or more of which correspond to each of four corners of an image GV41 showing the area R1 included in the captured image GS4, and performing the second correction based on the coordinates of the four or more points D1 and the coordinates of the plurality of points D2.

[0146] When the number of points D1 detected by image processing is large, the correspondence with the points D2 may not be good, and the generation accuracy of the third projective transformation matrix 118 used in the second correction may decrease. For this reason, it is preferable for the user to select the detected multiple points D1. The projector 1 displays multiple instruction images GD that correspond one-to-one with the detected multiple points D1. In addition, the projector 1 selects multiple points D1 based on the operation from the user who selects the instruction image GD, and executes the second correction based on the coordinates of the selected multiple points D1. Therefore, the projector 1 can generate the third projective transformation matrix 118 used in the second correction with high accuracy, and can more accurately maintain the shape of the projected image GP2 as desired by the user.

[0147] In the first embodiment, the angle CN4-1 is an example of a "first angle", the angle CN4-2 is an example of a "second angle", the angle CN4-3 is an example of a "third angle", the angle CN4-4 is an example of a "fourth angle", the multiple instruction images GD are an example of a "multiple instruction images", one instruction image GD is an example of a "single instruction image", the captured image GS4 is an example of a "second captured image", the image GV41 is an example of a "second image", the point D2 is an example of a "second feature point", and the multiple points D2 are an example of a "multiple second feature points". In addition, the "four corners of the second image" are the angles CN6-1 to CN6-4 as examples.

[0148] In addition, the projection method of the first embodiment further includes controlling the projection device 16 to display four corner images GC2 on the wall surface W1, which correspond one-to-one to the four corners of the region R1, and when selection of one or more points D1 at corner CN4-1 is completed, changing the display mode of the corner image GC2-1 among the four corner images GC2, which corresponds to corner CN4-1, so that the four corners of the image GV21 correspond one-to-one to the four corner images GC2.

[0149] This allows the user to easily distinguish between corners where the selection of point D1 has been completed and corners where the selection of point D1 has not been completed.

[0150] In the first embodiment, the four corner images GC2 are an example of the "four corner images", and the corner image GC2-1 is an example of the "one corner image".

[0151] Moreover, the projection method according to the first embodiment further includes changing the display mode of one instruction image GD in response to selecting one point D1.

[0152] This allows the user to easily distinguish between the instruction image GD corresponding to the selected point D1 and the instruction image GD corresponding to the unselected point D1.

[0153] Furthermore, in the projection method of the first embodiment, displaying one of a plurality of instruction images GD on the wall surface W1 in a manner distinguishable from the other instruction images GD includes displaying the one instruction image GD whose display mode changes over time.

[0154] For example, the projector 1 changes the display mode of one of the instruction images GD that is a candidate for the user's selection over time by repeatedly blinking the instruction image GD, allowing the user to easily distinguish between the one of the instruction images GD that is a candidate for the user's selection and the other instruction images GD.

[0155] Furthermore, in the projection method according to the first embodiment, displaying one of the multiple instruction images GD on the wall surface W1 so as to be distinguishable from the other instruction images GD includes displaying the one instruction image GD in a display mode different from that of the pointer image GC1.

[0156] This allows the user to easily distinguish between the instruction image GD and the pointer image GC1.

[0157] 2. Variations The above embodiment may be modified in various ways. Specific modified aspects are exemplified below. Two or more aspects arbitrarily selected from the following examples may be appropriately combined within a range that does not contradict each other. In the modified examples exemplified below, the elements whose actions and functions are equivalent to those of the above embodiment will be appropriately omitted by using the symbols used in the above explanation.

[0158] 2.1. Variation 1 In the above embodiment, a case where the color of the instruction image GD corresponding to one point D1 is changed in response to the selection of the point D1 has been exemplified, but the present invention is not limited to such an embodiment. For example, the shape of the instruction image GD corresponding to one point D1 may be changed in response to the selection of one point D1. Also, the size of the instruction image GD corresponding to one point D1 may be changed in response to the selection of one point D1.

[0159] 2.2. Variation 2 In the above-described embodiment and modified example, when the operation device 18 receives an operation from a user to end the selection of one or more points D1 at a corner of the image GV21 corresponding to one corner image GC2 selected based on a user operation, the color of the one corner image GC2 is changed. However, the present invention is not limited to such an embodiment. For example, when the operation device 18 receives an operation from a user to end the selection of one or more points D1 at a corner of the image GV21 corresponding to one corner image GC2 selected based on a user operation, the shape of the one corner image GC2 may be changed. Also, when the operation device 18 receives an operation from a user to end the selection of one or more points D1 at a corner of the image GV21 corresponding to one corner image GC2 selected based on a user operation, the size of the one corner image GC2 may be changed.

[0160] 2.3. Variation 3 In the above-described embodiment and modified example, the projector 1 realizes the projection method according to the present invention, but the present invention is not limited to such an embodiment. For example, the projection method according to the present invention may be realized by a projection system including an information processing device having the same functions as the processing device 12, and a projector connected to the information processing device so as to be able to communicate with the information processing device.

[0161] 2.4. Variation 4 In the above embodiment and modified example, the projected image GP1 is an image having a so-called polka dot pattern, but the present invention is not limited to such an embodiment. The projected image GP1 may be an image having a so-called lattice pattern, in which a plurality of straight lines parallel to the longitudinal direction of the image and a plurality of straight lines parallel to the lateral direction of the image are drawn. The projected image GP1 may also be an image having a so-called checkered pattern.

[0162] 2.5. Variation 5 In the above-described embodiment and modified example, the second captured image information 106 represents a captured image acquired in the third period, i.e., a captured image acquired after the first correction is performed, but the present invention is not limited to such an embodiment. The second captured image information 106 may represent a captured image acquired before the first correction is performed.

[0163] 2.6. Variation 6 A part or all of the control executed by the projector 1 may be executed by the processing device 12. Also, a part or all of the control executed by the processing device 12 may be executed by the projector 1.

[0164] 3. Notes The present invention will be summarized below as an appendix.

[0165] Appendix 1 By controlling a projection device, a projection image having four corners is displayed inside a display area having four corners as an area on a projection surface; acquiring a first captured image which is a captured image of a range including the display area in which the projection image is displayed; detecting a plurality of first feature points corresponding one-to-one to the four corners of a first image showing the display area included in the first captured image by executing image processing on the first captured image; displaying a pointer image located at a corner of the projection image and for adjusting a shape of the projection image by controlling the projection device; User Actionsperforming a first correction to correct a shape of the projected image by changing positions of corners of the projected image based on a plurality of first feature points; and performing a second correction to maintain a positional relationship between the four corners of the display area and the four corners of the projected image determined by the first correction, based on the plurality of first feature points.

[0166] That is, the projector or projection system that realizes the projection method described in Supplementary Note 1 performs a first correction based on an operation from the user, thereby correcting the shape of the projection image to a shape desired by the user. Also, the projector or projection system that realizes the projection method described in Supplementary Note 1 maintains the position where the projection image is displayed and the shape of the projection image by performing a second correction. Therefore, the projector or projection system that realizes the projection method described in Supplementary Note 1 can maintain the shape of the projection image to a shape desired by the user. Also, the first correction is performed based on the operation of the user. Therefore, the user can easily adjust the shape of the projection image without having to precisely adjust the position, orientation, angle, etc. of the projector.

[0167] Appendix 2 the four corners of the first image include a first corner, a second corner, a third corner, and a fourth corner, the plurality of first feature points include one or more first feature points corresponding to the first corners, one or more first feature points corresponding to the second corners, one or more first feature points corresponding to the third corners, and one or more first feature points corresponding to the fourth corner; controlling the projection device to display a plurality of instruction images corresponding one-to-one to the plurality of first feature points on the projection surface; controlling the projection device to display one instruction image of the plurality of instruction images on the projection surface so as to be distinguishable from other instruction images; setting one first feature point corresponding to the selected one instruction image as the first feature point for performing the second correction in response to a user operation of selecting one instruction image of the plurality of instruction images displayed on the projection surface; and receiving a user operation to select four or more first feature points, the user operation including: selecting one or more first feature points from the one or more first feature points corresponding to the first corner, selecting one or more first feature points from the one or more first feature points corresponding to the second corner, selecting one or more first feature points from the one or more first feature points corresponding to the third corner, and selecting one or more first feature points from the one or more first feature points corresponding to the fourth corner; and performing the second correction includes obtaining a second captured image representing a result of capturing an image of an area including the display area in which the projection image on which the first correction has been performed is displayed, and performing image processing on the second captured image to detect a plurality of second feature points, one or more of which correspond to each of four corners of a second image indicating the display area included in the second captured image; Including, The second correction is performed based on the coordinates of the four or more first feature points and the coordinates of the plurality of second feature points. Execute, Appendix 1 The projection method described.

[0168] When a large number of first feature points are detected by image processing, the correspondence with the second feature points may not be good, and the accuracy of generating the projective transformation matrix used in the second correction may decrease. For this reason, it is preferable for the user to select among the detected first feature points. A projector or projection system that realizes the projection method described in Appendix 2 displays a plurality of indication images that correspond one-to-one to the detected first feature points. Furthermore, a projector or projection system that realizes the projection method described in Appendix 2 selects a plurality of first feature points based on an operation by the user to select an indication image, and performs the second correction based on the coordinates of the selected first feature points. Therefore, Appendix 2 A projector or projection system that realizes the projection method described above is capable of generating the projection transformation matrix used for the second correction with high accuracy, and can more accurately maintain the shape of the projected image in the shape desired by the user.

[0169] Appendix 3 The projection method described in Appendix 2, further comprising: controlling the projection device to display four corner images on the projection surface, the four corner images corresponding one-to-one to the four corners of the display area; and, when selection of one or more first feature points has been completed at the first corner, changing a display mode of one of the four corner images corresponding to the first corner, wherein the four corners of the first image correspond one-to-one to the four corner images.

[0170] This allows the user to easily distinguish between a corner where selection of a first feature point has been completed and a corner where selection of a first feature point has not been completed.

[0171] Appendix 4 The projection method according to claim 2 or 3, further comprising changing a display mode of the one instruction image in response to selecting the one first feature point.

[0172] This allows the user to easily distinguish between a designation image corresponding to a selected first feature point and a designation image corresponding to an unselected first feature point.

[0173] Appendix 5 The projection method according to any one of Appendix 2 to Appendix 4, wherein displaying one of the plurality of instruction images on the projection surface in a manner distinguishable from other instruction images includes displaying the one instruction image whose display mode changes over time.

[0174] For example, a projector or projection system that realizes the projection method described in Supplementary Note 5 changes the display mode of one of the indication images that is a candidate for user selection over time by repeatedly blinking the indication image, etc. This allows the user to easily distinguish between the indication image that is a candidate for selection and other indication images.

[0175] Appendix 6 The projection method according to any one of Appendix 2 to Appendix 4, wherein displaying one of the plurality of instruction images on the projection surface in a manner distinguishable from other instruction images includes displaying the one instruction image in a display manner different from that of the pointer image.

[0176] This allows the user to easily distinguish between the indication image and the pointer image.

[0177] Appendix 7 The processing device controls a projection device to display a projection image having four corners inside a display area on a projection surface, the projection image having four corners; obtains a first captured image representing a result of capturing an image of an area including the display area in which the projection image is displayed; detects a plurality of first feature points corresponding one-to-one to the four corners of a first image showing the display area included in the first captured image by executing image processing on the first captured image; controls the projection device to display a pointer image located at a corner of the projection image and for adjusting a shape of the projection image; and controls a pointer image for moving the pointer image. The user operationand performing a second correction to maintain a positional relationship between the four corners of the display area and the four corners of the projected image determined by the first correction, based on the plurality of first feature points.

[0178] That is, the projector or projection system that operates according to the program described in Supplementary Note 7 performs a first correction based on an operation from the user, thereby correcting the shape of the projected image to a shape desired by the user. Furthermore, the projector or projection system that operates according to the program described in Supplementary Note 7 performs a second correction to maintain the position where the projected image is displayed and the shape of the projected image. Therefore, the projector or projection system that operates according to the program described in Supplementary Note 7 can maintain the shape of the projected image to a shape desired by the user. Furthermore, the first correction is performed based on the operation of the user. Therefore, the user can easily adjust the shape of the projected image without having to precisely adjust the position, orientation, angle, etc. of the projector. [Explanation of symbols]

[0179] 1...projector, 10...storage device, 12...processing device, 14...imaging device, 16...projection device, 18...operation device, 100...program, 101...projected image information, 102...first projection image information, 103...second projection image information, 104...captured image information, 105...first captured image information, 106...second captured image information, 107...third captured image information, 108...fourth captured image information, 109...coordinate information, 110...first coordinate information, 111...second coordinate information, 112...third coordinate information, 113...fourth coordinate information, 114...fifth coordinate information information, 115...projection transformation matrix, 116...first projective transformation matrix, 117...second projective transformation matrix, 118...third projective transformation matrix, 120...projection control unit, 121...imaging control unit, 122...detection unit, 123...correction unit, 124...coordinate management unit, 125...matrix generation unit, 140...imaging element, 142...imaging lens, 160...light modulator, 162...projection lens, GP1...projected image, GC1...pointer image, GC2...corner image, GD...pointing image, GF1...image, GS1...captured image, CN1...corner, D1...point, R1...area, W1...wall surface.

Claims

1. Displaying a projection image having four corners inside a display area having four corners as an area on a projection surface by controlling a projection device; acquiring a first captured image which is a captured image of a range including the display area in which the projection image is displayed; detecting a plurality of first feature points corresponding to four corners of a first image showing the display area included in the first captured image by executing image processing on the first captured image, the plurality of first feature points including at least one first feature point corresponding to each of the four corners of the display area; displaying a pointer image located at a corner of the projected image and for adjusting a shape of the projected image by controlling the projection device; performing a first correction for correcting a shape of the projection image by changing a position of a corner of the projection image based on a user operation for moving the pointer image; when a positional relationship between the four corners of the display area and the four corners of the projection image determined by the first correction has changed, performing a second correction based on the plurality of first feature points to return the positional relationship after the change to the positional relationship before the change; Including, the four corners of the first image include a first corner, a second corner, a third corner, and a fourth corner; the plurality of first feature points include one or more first feature points corresponding to the first corners, one or more first feature points corresponding to the second corners, one or more first feature points corresponding to the third corners, and one or more first feature points corresponding to the fourth corners; displaying, on the projection surface, a plurality of instruction images corresponding one-to-one to the plurality of first feature points by controlling the projection device; controlling the projection device to display one of the plurality of instruction images on the projection surface in a manner distinguishable from other instruction images; in response to a user operation of selecting one of the plurality of instruction images displayed on the projection surface, setting one first feature point corresponding to the selected one instruction image as the first feature point for performing the second correction; Setting the plurality of first feature points for performing the second correction includes: selecting one or more first feature points from the one or more first feature points corresponding to the first corner; selecting one or more first feature points from the one or more first feature points corresponding to the second corner; selecting one or more first feature points from the one or more first feature points corresponding to the third corner; selecting one or more first feature points from the one or more first feature points corresponding to the fourth corner; receiving a user operation to select four or more first feature points including: Further comprising: Performing the second correction includes: acquiring a second captured image representing a result of capturing an image of an area including the display area in which the projection image on which the first correction has been performed is displayed; executing image processing on the second captured image to detect a plurality of second feature points, at least one of which corresponds to each of four corners of a second image indicating the display area included in the second captured image; Including, A projection method for performing the second correction based on coordinates of the four or more first feature points and coordinates of the plurality of second feature points.

2. displaying, on the projection surface, four corner images that correspond one-to-one to the four corners of the display area by controlling the projection device; When the selection of one or more first feature points is completed for the first corner, changing a display mode of one corner image corresponding to the first corner among the four corner images; Further comprising: The four corners of the first image correspond one-to-one to the four corner images. The projection method according to claim 1 .

3. and changing a display mode of the one indication image in response to selecting the one first feature point. The projection method according to claim 1 or 2.

4. Displaying one of the plurality of instruction images on the projection surface in a manner distinguishable from other instruction images includes displaying the one instruction image whose display mode changes over time. The projection method according to claim 1 .

5. Displaying one of the plurality of instruction images on the projection surface in a manner distinguishable from the other instruction images includes displaying the one instruction image in a display mode different from that of the pointer image. The projection method according to claim 1 .

6. The processing device includes: Controlling the projection device to display a projection image having four corners inside a display area on a projection surface, the display area having four corners; acquiring a first captured image representing a result of capturing an image of an area including the display area in which the projection image is displayed; detecting a plurality of first feature points corresponding to four corners of a first image showing the display area included in the first captured image by executing image processing on the first captured image, the plurality of first feature points including at least one first feature point corresponding to each of the four corners of the display area; displaying a pointer image located at a corner of the projected image and for adjusting a shape of the projected image by controlling the projection device; performing a first correction for correcting a shape of the projection image by changing a position of a corner of the projection image based on a user operation for moving the pointer image; when a positional relationship between the four corners of the display area and the four corners of the projection image determined by the first correction has changed, executing a second correction for returning the positional relationship after the change to the positional relationship before the change, based on the plurality of first feature points; A program, the four corners of the first image include a first corner, a second corner, a third corner, and a fourth corner; the plurality of first feature points include one or more first feature points corresponding to the first corners, one or more first feature points corresponding to the second corners, one or more first feature points corresponding to the third corners, and one or more first feature points corresponding to the fourth corners; displaying, on the projection surface, a plurality of instruction images corresponding one-to-one to the plurality of first feature points by controlling the projection device; controlling the projection device to display one of the plurality of instruction images on the projection surface in a manner distinguishable from other instruction images; in response to a user operation of selecting one of the plurality of instruction images displayed on the projection surface, setting one first feature point corresponding to the selected one instruction image as the first feature point for performing the second correction; Setting the plurality of first feature points for performing the second correction includes: selecting one or more first feature points from the one or more first feature points corresponding to the first corner; selecting one or more first feature points from the one or more first feature points corresponding to the second corner; selecting one or more first feature points from the one or more first feature points corresponding to the third corner; selecting one or more first feature points from the one or more first feature points corresponding to the fourth corner; receiving a user operation to select four or more first feature points including: Further comprising: Performing the second correction includes: acquiring a second captured image representing a result of capturing an image of an area including the display area in which the projection image on which the first correction has been performed is displayed; executing image processing on the second captured image to detect a plurality of second feature points, at least one of which corresponds to each of four corners of a second image indicating the display area included in the second captured image; Including, executing the second correction based on coordinates of the four or more first feature points and coordinates of the plurality of second feature points; program.

Images