Image correction method, information processing apparatus, and program

Abstract

To improve convenience in operating to correct a shape of an image projected on a projection surface.SOLUTION: An image correction method includes receiving an input of selecting one of a first method for correcting a shape of an image projected from a projector onto a projection surface and a second method different from the first method, outputting a signal for causing the projector to project a first pattern image when the first method is selected, generating first correction data for correcting the shape of the image on the basis of a first captured image including the projection surface and a first projection image appearing on the projection surface by projecting the first pattern image, outputting a signal for causing the projector to project a second pattern image different from the first pattern image when the second method is selected, and generating second correction data for correcting the shape of the image on the basis of a second captured image including the projection surface and a second projection image appearing on the projection surface by projecting the second pattern image.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present disclosure relates to an image correction method, an information processing device, and a program. [Background technology]

[0002] 2. Description of the Related Art Techniques for correcting an image projected onto a projection surface are known. For example, Patent Document 1 discloses a system including an imaging device that captures images of multiple patterns projected onto a projection surface, and an image data output device that corrects image data corresponding to an image projected by a projector based on the captured patterns. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-178393 A Summary of the Invention [Problem to be solved by the invention]

[0004] There are multiple types of pattern images, as shown by the multiple examples of dot patterns described in Patent Document 1. In addition, there are multiple methods for correcting the shape of an image projected onto a projection surface. However, no consideration has been given to the pattern images used for each method, and therefore there is room for improvement in terms of user convenience when performing correction operations. [Means for solving the problem]

[0005] One aspect of the image correction method disclosed herein includes receiving an input for selecting either a first method for correcting a shape of an image projected from a projector onto a projection surface, or a second method for correcting the shape of the image, the second method being different from the first method; when the first method is selected, outputting a signal for causing the projector to project a first pattern image; and generating first correction data for correcting the shape of the image using the first method, based on a first captured image including a first projection image that appears on the projection surface by projecting the first pattern image, and the projection surface, obtained by capturing an image of the projection surface; when the second method is selected, outputting a signal for causing the projector to project a second pattern image that is different from the first pattern image; and generating second correction data for correcting the shape of the image using the second method, based on a second captured image including a second projection image that appears on the projection surface by projecting the second pattern image, and the projection surface, obtained by capturing an image of the projection surface.

[0006] One aspect of an information processing device of the present disclosure includes a processing device, which executes the following: accepting an input for selecting either a first method for correcting a shape of an image projected from a projector onto a projection surface, or a second method for correcting the shape of the image, the second method being different from the first method; when the first method is selected, outputting a signal for causing the projector to project a first pattern image; and generating first correction data for correcting the shape of the image using the first method, based on a first captured image including a first projection image that appears on the projection surface by projecting the first pattern image, and the projection surface, which is obtained by capturing an image of the projection surface; and when the second method is selected, outputting a signal for causing the projector to project a second pattern image that is different from the first pattern image; and generating second correction data for correcting the shape of the image using the second method, based on a second captured image including a second projection image that appears on the projection surface by projecting the second pattern image, and the projection surface, which is obtained by capturing an image of the projection surface.

[0007] One aspect of the program of the present disclosure causes a computer to execute the following: accepting an input for selecting either a first method for correcting a shape of an image projected from a projector onto a projection surface, or a second method for correcting the shape of the image, the second method being different from the first method; when the first method is selected, outputting a signal for causing the projector to project a first pattern image; and generating first correction data for correcting the shape of the image using the first method, based on a first captured image including a first projection image that appears on the projection surface by projecting the first pattern image, and the projection surface, which is obtained by capturing an image of the projection surface; and when the second method is selected, outputting a signal for causing the projector to project a second pattern image that is different from the first pattern image; and generating second correction data for correcting the shape of the image using the second method, based on a second captured image including a second projection image that appears on the projection surface by projecting the second pattern image, and the projection surface, which is obtained by capturing an image of the projection surface. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a projection system according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram illustrating an example of an electrical configuration of a projector. [Diagram 3] FIG. 2 is a diagram illustrating an example of an electrical configuration of an information processing device. [Figure 4] 3A to 3C are diagrams showing examples of a first pattern image, a second pattern image, and a third pattern image. [Diagram 5] 13 is a diagram showing the relationship between the contour of the projection range and the pattern portion when the second pattern image is projected onto the projection surface. FIG. [Figure 6] 10 is a flowchart illustrating an example of an image correction process performed by an information processing device. [Figure 7] 10 is a flowchart illustrating an example of an image correction process performed by an information processing device. [Figure 8]10A to 10C are diagrams illustrating an example of a transition of a user interface image displayed on a display as an image correction process progresses. [Figure 9] 10A to 10C are diagrams illustrating an example of a transition of a user interface image displayed on a display as an image correction process progresses. [Figure 10] 1A and 1B are explanatory diagrams of a first orientation, a second orientation, and an orientation of a user interface image. [Figure 11] 11 is a flowchart illustrating an example of a display process performed by the information processing device. [Figure 12] FIG. 13 is a diagram illustrating an example of a modified form of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, a preferred embodiment of the present disclosure will be described with reference to the drawings. In the drawings, the dimensions and scale of each part may differ from the actual ones, and some parts may be shown diagrammatically to facilitate understanding. In the following description, unless otherwise specified to limit the present disclosure, the scope of the present disclosure is not limited to the embodiments described in the following description. In addition, the scope of the present disclosure includes the equivalent scope of the embodiments.

[0010] 1. Embodiment [Configuration of Projection System 1] FIG. 1 is a diagram illustrating an example of the configuration of a projection system 1 according to this embodiment. As shown in FIG. 1, the projection system 1 includes a projector 10 and an information processing device 20. The projector 10 is a device that projects image light LG of an image based on image data DG onto a projection surface SP. The projection surface SP is the surface of an object onto which the image light LG is projected. In the following description, the object onto which the image light LG is projected is referred to as a "projected object OP." The image projected onto the projection surface SP is referred to as a "target image GO," and the image that appears on the projection surface SP by projecting the image light LG is referred to as a "projected image GP."

[0011] In this embodiment, the target image GO is rectangular with a long side ALGO and a short side ASGO. The target image GO is projected from the projector 10 so that the short side ASGO is aligned with the vertical line CV. The short side ASOG of the target image GO does not need to be completely parallel to the vertical line CV, but may be closer to the vertical line CV than the horizontal line CH.

[0012] The information processing device 20 is an example of a computer. The information processing device 20 is a device capable of transmitting signals between the projector 10, and supplies a signal of image data DG to the projector 10. The transmission path of the signal between the information processing device 20 and the projector 10 may be either wired or wireless. Furthermore, there are no particular limitations on the standard adopted for the transmission path. Note that FIG. 1 shows a wireless form of the transmission path.

[0013] The information processing device 20 of this embodiment is a portable device including a housing 200, a display 210, and an imaging device 220. That is, the housing 200 is large enough for a user U to carry, and the display 210 and the imaging device 220 are housed in the housing 200, and the housing 200 is held by the hand of the user U. Representative examples of specific forms of the information processing device 20 are portable terminals such as smartphones, mobile phones, tablet personal computers, and laptop personal computers. The imaging device 220 may be provided separately from the information processing device 20. In FIG. 1, the information processing device 20 is shown in the form of a smartphone incorporating the imaging device 220.

[0014] Next, the electrical configurations of the projector 10 and the information processing device 20 will be described in more detail.

[0015] [Projector 10 Configuration] FIG. 2 is a diagram showing an example of the electrical configuration of the projector 10. As shown in FIG. As shown in FIG. 2, the projector 10 includes a first processing device 100, a first storage device 120, an optical device 130, a first communication device 140, and an operation input device 150, each of which is connected to a bus. The first processing device 100 includes one or more processors, such as a CPU (Central Processing Unit). A part of the function of the first processing device 100 may be configured by a circuit such as an FPGA (Field Programmable Gate Array). The first storage device 120 is a recording medium readable by the first processing device 100. The first storage device 120 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically Erasable Programmable Read Only Memory). The volatile memory is, for example, a RAM (Random Access Memory). The first processing device 100 of this embodiment functions as a first control unit 101 that controls each unit of the projector 10. The first processing device 100 may further function as another functional unit.

[0016] The optical device 130 is a device that emits image light LG based on the target image GO based on the control by the first control unit 101. Specifically, the optical device 130 includes a light source, a light modulation device, and an optical system. The light source is a discharge lamp, a semiconductor light source, etc. Representative examples of the discharge lamp are a halogen lamp, a xenon lamp, and an extra-high pressure mercury lamp, etc. Representative examples of the semiconductor light source are an LED (light-emitting diode), a laser diode, etc. The light modulation device includes a light modulation element that optically modulates the light of the light source based on the control of the first control unit 101. The light modulation element is, for example, a liquid crystal panel, a digital mirror device (Digital Micromirror Device), etc. The light modulation element may be a single liquid crystal panel or digital mirror device, or may include multiple liquid crystal panels or digital mirror devices. The optical system includes one or more optical elements that adjust the magnification ratio and imaging position of the light output from the light modulation device. The output light of the optical system is emitted from the optical device 130 as image light LG, and is projected onto the projection surface SP.

[0017] The first communication device 140 is a device that communicates with the information processing device 20. The communication method may be either a wireless method or a wired method. When the communication method is a wireless method, the first communication device 140 includes a wireless communication module including, for example, an antenna, an RF (Radio Frequency) circuit, and a baseband circuit. When the communication method is a wired method, the first communication device 140 includes a wired communication module including a connector to which a communication cable is connected, and an interface circuit that processes signals transmitted and received via the connector. Through communication by the first communication device 140, various signals are provided from the information processing device 20 to the projector 10. The signals include, for example, a signal of image data DG of the target image GO, and signals of first correction data and second correction data, which will be described later.

[0018] The operation input device 150 is a device that accepts input through operations by the user U. The operation input device 150 is, for example, a button provided on the main body of the projector 10, a remote controller that remotely controls the projector 10, or the like.

[0019] [Configuration of information processing device 20] FIG. 3 is a diagram showing an example of the electrical configuration of the information processing device 20. As shown in FIG. As shown in FIG. 3, the information processing device 20 includes a second processing device 230, a second storage device 240, a second communication device 250, a display 210, and an imaging device 220, each of which is connected to a bus. The second processing unit 230 includes a processor, such as the CPU described above. The second memory device 240 includes, for example, the above-mentioned non-volatile memory and the above-mentioned volatile memory. The second communication device 250 is a device that communicates with the first communication device 140 of the projector 10. That is, the second communication device 250 includes the above-mentioned wireless communication module or wired communication module according to the communication method and communication standard compatible with the first communication device 140.

[0020] The display 210 is a device that displays images including various types of information. The display 210 of this embodiment is also used as an input device. Specifically, the display 210 includes a display panel 212 that displays images under the control of the second processing device 230, and a touch sensor panel 214 that outputs inputs operated by a user U. The external shape of the display 210 of this embodiment is rectangular with long sides ALD and short sides ASD, as shown in FIG. 1.

[0021] In the following description, a holding mode in which the user U holds the housing 200 of the information processing device 20 so that the long side ALD of the display 210 is aligned with the vertical CV is referred to as "portrait holding". Also, a holding mode in which the user U holds the housing 200 of the information processing device 20 so that the short side ASD of the display 210 is aligned with the vertical CV is referred to as "horizontal holding". Note that the portrait holding is not limited to a position in which the long side ALD of the display 210 is completely parallel to the vertical CV, but also includes a position in which the long side ALD is closer to the vertical CV than the horizontal CH. Also, the landscape holding is not limited to a position in which the short side ASD of the display 210 is completely parallel to the vertical CV, but also includes a position in which the short side ASD is closer to the vertical CV than the horizontal CH.

[0022] Returning to FIG. 3, the imaging device 220 is a device that performs imaging of an external subject based on the control of the second processing device 230, and outputs a captured image GC obtained by imaging. Specifically, the imaging device 220 includes an imaging optical system 222 and an imaging element 224. The imaging optical system 222 includes an optical element that forms an image of the subject on the imaging element 224. The imaging element 224 outputs a signal of a captured image GC of the subject image. In the imaging device 220, the shape of the imaging range B in a plan view is a rectangle having a long side ALB and a short side ASB, as shown in FIG. 1. The shape of this imaging range B is determined, for example, by the outer shape of the imaging surface of the imaging element 224 and the masking shape of the mask included in the imaging optical system 222. In the information processing device 20 of this embodiment, the orientation of the long side ALB and the short side ASB of the imaging range B coincides with the direction of the long side ALD and the short side ASD of the display 210. That is, the long side ALB of the imaging range B is along the long side ALD of the display 210, and the short side ASB of the imaging range B is along the short side ASD of the display 210. Therefore, when the user U captures an object that is long in the horizontal CH such as the projected image GP, the user U can easily fit the entire object in the imaging range B by holding the housing 200 of the information processing device 20 sideways. Note that the long side ALB and the short side ASB of the imaging range B do not need to be completely parallel to the direction of the long side ALD and the short side ASD of the display 210. There may be a misalignment between the two.

[0023] Returning to Fig. 3, the above-mentioned second storage device 240 stores an application 242 and resource data 244. The application 242 is an example of a program. The resource data 244 is resource data used for executing the application 242. The resource data 244 includes, for example, data of a user interface image GAU corresponding to a screen of the application 242, and data such as image data DG of a pattern image described later. The application 242 of this embodiment includes a program that causes the second processing device 230 to realize a function of executing image correction processing for correcting the shape of the target image GO. By executing this application 242, the second processing device 230 functions as a projection control unit 231, an imaging control unit 232, an application screen control unit 233, an input acquisition unit 234, and a correction unit 235.

[0024] The projection control unit 231 controls the projection of the target image GO by the projector 10. Specifically, the projection control unit 231 provides a signal of image data DG of the target image GO to the projector 10 via the second communication device 250, thereby causing the projector 10 to project the image light LG of the target image GO. The imaging control unit 232 controls the imaging device 220 to perform imaging by the imaging device 220 and obtains a captured image GC.

[0025] The application screen control unit 233 executes control for switching the user interface image GAU displayed on the display 210 in accordance with the progress of the image correction process. The input acquisition unit 234 acquires an input by a user U's operation on the user interface image GAU based on an output signal from the touch sensor panel 214, thereby accepting the input.

[0026] The correction unit 235 generates correction data for correcting the shape of the target image GO. The correction unit 235 of this embodiment generates correction data for correcting the shape of the target image GO using a different method for each type of projection object OP. However, all of the methods are methods for generating correction data based on a captured image GC obtained by capturing a projected image GP when a pattern image, which has a different pattern for each method, is projected from the projector 10 as the target image GO together with the projection surface SP.

[0027] In this embodiment, the type of projection object OP is classified according to whether it is an object designed for projection. In the following description, a projection object OP that is not designed for projection is referred to as a "first type projection object OP1," and a projection object OP that is designed for projection is referred to as a "second type projection object OP2." A typical example of the first type projection object OP1 is the interior of a building or an outdoor wall. Representative examples of the second type projection object OP2 are screens and boards that are sold for projection purposes.

[0028] According to the classification of this embodiment, there are at least two differences between the first type projection object OP1 and the second type projection object OP2. The first difference is as follows: the projection range on the surface of the second type projection target OP2 used for the projection surface SP is defined by the outline of the surface of the second type projection target OP2, or by a mark affixed to the surface, etc. On the other hand, the range on the surface of the first type projection target OP1 used for the projection surface SP is not clearly defined.

[0029] The second difference is as follows. That is, most of the projection surfaces SP of the second type projection object OP2 are flat surfaces. On the other hand, most of the projection surfaces SP of the first type projection object OP1 are not flat surfaces. In the present disclosure, a flat surface is not limited to a completely flat surface. That is, a flat surface is a surface that does not include a shape that causes distortion in the projection image GP that is visible to a viewer at a predetermined position when the image light LG is projected from a position directly facing the projection surface SP. The shape that causes distortion in the projection image GP is, for example, a curved shape, a concave shape, a convex shape, etc. in the depth direction of the projection surface SP. The curved shape is, for example, a curved shape, a bent shape, etc. Also, the predetermined position is, for example, a position away from the projection surface SP by a viewing distance defined by the specifications of the projector 10, etc.

[0030] Based on the above two differences, the second type projection object OP2 can be said to be an object in which the projection range used as the projection surface SP is clearly defined and the surface of the projection range is flat. On the other hand, the first type projection object OP1 can be said to be an object in which the projection range used as the projection surface SP is not clearly defined and the surface of the projection surface SP may not be flat.

[0031] Therefore, when the projection object OP corresponds to the first type projection object OP1, the correction unit 235 of this embodiment corrects the shape of the target image GO using a method that can eliminate distortion caused by the shape of the surface of the projection surface SP, which is not a flat surface. In the following description, this method is referred to as the "first method." Furthermore, when the projection object OP corresponds to a second type projection object OP2, the correction unit 235 corrects the shape of the target image GO using a method that can fit the projection image GP within the projection range of the projection surface SP. In the following description, this method is referred to as the "second method." Furthermore, the line that borders the projection range of the projection surface SP in the second type projection object OP2 is referred to as the "contour J." The contour J will be described later with reference to FIG. 5.

[0032] As described above, in both the first method and the second method, different pattern images are projected from the projector 10 as the target image GO. In the following description, the pattern image corresponding to the first method is referred to as the "first pattern image GOP1," and the pattern image corresponding to the second method is referred to as the "second pattern image GOP2." Note that the second method of this embodiment, which will be described in detail later, includes projecting the second pattern image GOP2 from the projector 10, as well as projecting a third pattern image GOP3 different from the second pattern image GOP2, from the projector 10 as the target image GO.

[0033] In addition, in the following description, the projected image GP that appears on the projection surface SP by projecting the first pattern image GOP1 will be referred to as the "first projected image GP1," and the captured image GC including the first projected image GP1 and the projection surface SP will be referred to as the "first captured image GC1." The projected image GP that appears on the projection surface SP by projecting the second pattern image GOP2 is referred to as a "second projected image GP2," and the captured image GC including the second projected image GP2 and the projection surface SP is referred to as a "second captured image GC2." The projected image GP that appears on the projection surface SP by projecting the third pattern image GOP3 is referred to as a "third projected image GP3," and the captured image GC including the third projected image GP3 and the projection surface SP is referred to as a "third captured image GC3."

[0034] [Example of pattern image] FIG. 4 is a diagram showing an example of the first pattern image GOP1, the second pattern image GOP2, and the third pattern image GOP3. The first pattern image GOP1 is a rectangular image showing a pattern in which dots E are arranged at a predetermined interval in each of the vertical direction F1 and horizontal direction F2 of the image. Each dot E is a detection point on the projection surface SP, and corresponds to a point for which the first correction amount is calculated. In other words, the higher the density of the dots E, the more precisely the projection surface SP is detected, and therefore, more precise correction is realized.

[0035] The first correction amount calculated for each dot E, which is a detection point, is a value for correcting the amount of deviation of the projection position in the depth direction caused by the shape of the projection surface SP, such as the unevenness of the surface. That is, the correction unit 235 of this embodiment calculates the above-mentioned first correction amount for each dot E in the first pattern image GOP1 based on the position of each dot E in the first pattern image GOP1 and the position of each dot E in the above-mentioned first captured image GC1. Note that the position of each dot E in the first pattern image GOP1 is a position in the projector coordinate system, and the position of each dot E in the first captured image GC1 is a position in the camera image coordinate system. The projector coordinate system is a coordinate system that uniquely specifies a position in the target image GO, such as the first pattern image GOP1. The camera image coordinate system is a coordinate system that uniquely specifies a position in the captured image GC acquired by imaging the imaging device 220.

[0036] By correcting the shape of the target image GO based on each of the first correction amounts, a projected image GP is obtained in which distortion caused by the shape of unevenness, etc. is suppressed. Note that the correction unit 235 may obtain the first correction amount at the position between each of the dots E by interpolation based on the first correction amount of each of the dots E in the first pattern image GOP1.

[0037] In this embodiment, a method including determining a first correction amount by using the first pattern image GOP1 corresponds to the above-mentioned first method.

[0038] The second pattern image GOP2 is an image including a pattern portion PT in which rectangles painted in a first color and rectangles painted in a second color are alternately arranged vertically and horizontally. In this embodiment, the first color is white, and the second color is black. The pattern represented by the pattern portion PT is generally called a checkered pattern. In the following description, each vertex of the rectangle included in the pattern portion PT is referred to as a "checkered corner PTC."

[0039] Each checker corner PTC of the pattern portion PT is a detection point on the projection surface SP, and each detection point is used to obtain the correspondence between the projector coordinate system and the camera image coordinate system described above. Also, the pattern portion PT is used to detect the contour J that defines the projection range of the second type projection object OP2. Specifically, the correction unit 235 of this embodiment obtains a projection transformation matrix based on the positions of the checker corners PTC, which are detection points in the second pattern image GOP2, and the positions of the checker corners PTC, which are detection points in the above-mentioned second captured image GC2. The projection transformation matrix is ​​a matrix that converts the camera image coordinate system into the projector coordinate system. Furthermore, the correction unit 235 detects the contour J of the projection range in the second type projection object OP2 based on a line located at a position surrounding the entire pattern portion PT in the second captured image GC2, or a mark indicating the projection range. Then, the correction unit 235 obtains a second correction amount for correcting the shape of the target image GO to fit the projection image GP within the projection range, based on the contour J in the captured image GC and the above-mentioned projection transformation matrix. The correction unit 235 may obtain a correction amount for correcting the rotation of the projection image GP in the roll angle direction based on the position of each checker corner PTC in the second pattern image GOP2 and the position of each checker corner PTC in the captured image GC, and may include the correction amount in the second correction amount. The roll angle direction is a rotation direction centered on the normal direction of the projection surface SP.

[0040] In this embodiment, a method including determining the second correction amount by using the second pattern image GOP2 corresponds to the above-mentioned second method.

[0041] Here, in the second method, even if the density of the checker corners PTC of the second pattern image GOP2 is smaller than the density of the dots E of the first pattern image GOP1, it is possible to obtain the correspondence between the projector coordinate system and the camera image coordinate system, and the contour J. In addition, as described above, the second type projection object OP2 has a relatively flat surface in the projection range, and distortion due to shape such as unevenness within the surface is not noticeable, so there is no need to obtain the above-mentioned first correction amount. Therefore, in the second pattern image GOP2, the density of the checker corners PTC which are detection points of the pattern portion PT is smaller than the density of the dots E which are detection points of the first pattern image GOP1. As a result, the second pattern image GOP2 has a smaller number of checker corners PTC, which are detection points, than the first pattern image GOP1, and therefore the processing speed using the second pattern image GOP2 can be increased. Furthermore, even if the second projected image GP2 is distorted due to the shape of the surface of the projection surface SP, such as unevenness, the density of the checker corners PTC, which are detection points, is low, so the number of checker corners PTC whose position is shifted due to the distortion is reduced. As a result, even if the surface of the projection surface SP includes a target shape such as unevenness, the influence of the shape is reduced, and the second correction amount cannot be obtained normally, preventing a situation in which the generation of the second correction data fails. In other words, the calculation of the second correction amount and the generation of the second correction data are more resistant to detection errors such as positional shifts of the checker corners PTCs and noise.

[0042] In the second pattern image GOP2 of this embodiment, the size of the pattern portion PT is relatively small in the center of the second pattern image GOP2, and a margin area portion PTM with no pattern is provided around the pattern portion PT. The size of the pattern portion PT is, for example, 1 / 2 of the second pattern image GOP2. According to this second pattern image GOP2, even if the second projection image GP2 is larger than the projection range, it is possible to increase the certainty that the pattern portion PT in the second projection image GP2 falls within the projection range.

[0043] However, because the size of the pattern portion PT is small, the pattern portion PT in the second projected image GP2 tends to be small relative to the contour J that indicates the projection range of the projection surface SP, as shown in Fig. 5. On the other hand, the projective transformation matrix that indicates the correspondence between the projector coordinate system and the camera image coordinate system, and the contour J of the projection range can be calculated with higher accuracy as the pattern portion PT is projected larger within the projection range. That is, there is sufficient room for improvement in the accuracy of the second correction amount calculated using the second pattern image GOP2. Therefore, the second method of this embodiment includes determining the contour J and the projection transformation matrix based on the second captured image GC2, and then projecting a third pattern image GOP3 different from the second pattern image GOP2 from the projector 10, and determining the contour J and the projection transformation matrix again based on the third captured image GC3 that captures the third projected image GP3 that appears by the projection. According to the second method, the accuracy of the contour J and the projective transformation matrix is ​​improved, and as a result, the accuracy of the second correction amount is also improved.

[0044] The third pattern image GOP3 in the second method is an image in which the size of the pattern portion PT is made larger than that of the second pattern image GOP2, as shown in Fig. 4. More specifically, the pattern portion PT of the third pattern image GOP3 is enlarged based on the distance between the contour J obtained based on the second captured image GC2 and the pattern portion PT in the second captured image GC2, with the size of the pattern portion PT of the second pattern image GOP2 as a reference. The generation of the third pattern image GOP3 is performed by, for example, the correction unit 235.

[0045] [Operation of information processing device] A: Image correction processing Figures 6 and 7 are flowcharts showing an example of image correction processing by information processing device 20. Figures 8 and 9 are diagrams showing an example of transition of user interface image GAU displayed on display 210 as the image correction processing progresses. When the application 242 is started up in the information processing device 20, the application screen control unit 233 of the second processing device 230 displays a first user interface image GAU1 on the display 210 (step Sa1), as shown in Fig. 6. The first user interface image GAU1 is an image that notifies the user U that the application 242 is running, as shown in Fig. 8.

[0046] Next, as shown in Fig. 6, application screen control unit 233 displays second user interface image GAU2 on display 210 (step Sa2). As shown in Fig. 8, second user interface image GAU2 is an image used for inputting an operation to select either the first method or the second method. As described above, the first method corresponds to the case where the projection object OP is a first type projection object OP1. The second method corresponds to the case where the projection object OP is a second type projection object OP2. As described above, the projection surface SP of this second type projection object OP2 includes a contour J that defines the projection range. The second user interface image GAU2 includes a first selection candidate image H21 suggesting a first type of projection object OP1 and a second selection candidate image H22 suggesting a second type of projection object OP2 whose projection surface SP includes a contour J. On the display 210, the display areas of the first selection candidate image H21 and the second selection candidate image H22 function as touch buttons. Therefore, when the user U performs a touch operation on either the first selection candidate image H21 or the second selection candidate image H22, either the first method or the second method is selected.

[0047] 6, when the user U selects either the first selection candidate image H21 or the second selection candidate image H22 by a touch operation, the input acquiring unit 234 accepts an input of an operation indicating the selection (step Sa3). Note that in this step Sa3, selecting either the first method or the second method includes selecting the projection object OP as either the first type projection object OP1 or the second type projection object OP2, that is, accepting a selection of whether the projection surface SP includes a contour J that partitions the projection range. Then, the input acquisition unit 234 determines whether the first method or the second method has been selected by the user U, based on the touch-operated first selection candidate image H21 and second selection candidate image H22 (step Sa4).

[0048] If the first method is selected (step Sa4: first method), application screen control unit 233 displays third user interface image GAU3 on display 210 (step Sa5). The third user interface image GAU3, as shown in FIG. 8, is an image that notifies the user U of the following four items. The first item is confirmation that keystone correction has been completed. The second item is the shape of the projected image GP after keystone correction has been completed. The third item is performing keystone correction by manual operation if keystone correction is not yet completed. The fourth item is the need to input an operation indicating that keystone correction has been completed. In the third user interface image GAU3, the first item is notified by character string information H31, the second item is notified by illustration H32, the third item is notified by character string information H33, and the fourth item is notified by a button image H34. In the display 210, the display area of ​​the button image H34 functions as a touch button. When the user U touches the button image H34, an operation indicating that the keystone correction has been completed is input.

[0049] Keystone distortion is a phenomenon in which the projected image GP is deformed into a trapezoidal shape when the projector 10 is installed at an angle with respect to the projection surface SP. Keystone distortion correction is performed by adjusting the positions of one or more of the four corners of the rectangular target image GO. Keystone distortion correction by manual operation is performed by the user U operating the operation input device 150 of the projector 10 to adjust the positions of the corners of the target image GO.

[0050] As shown in FIG. 6, when input acquisition unit 234 receives an operation input indicating that trapezoidal distortion correction has been completed (step Sa6), application screen control unit 233 displays a fourth user interface image GAU4 on display 210 (step Sa7). The fourth user interface image GAU4, as shown in FIG. 8, is an image that notifies the user U of the following three items: The first item is to capture the first projection image GP1 by projecting the above-mentioned first pattern image GOP1. The second item is to explain how to capture the image, specifically, the orientation of the display 210 when capturing the image. The third item is to input an operation to instruct the projector 10 to project the first pattern image GOP1. In the fourth user interface image GAU4, the first item is notified by character string information H41, the second item is notified by character string information H42 and an illustration H43, and the third item is notified by a button image H44. In the display 210, the display area of ​​the button image H44 functions as a touch button. When the user U touches the button image H44, an operation to start projection of the first pattern image GOP1 is input.

[0051] As shown in FIG. 6, when the input acquisition unit 234 receives an input of an operation instructing to project the first pattern image GOP1 (step Sa8), the projection control unit 231 performs control to project the first pattern image GOP1 from the projector 10 (step Sa9). Specifically, the projection control unit 231 reads out the image data DG of the first pattern image GOP1 from the second storage device 240. Then, the projection control unit 231 provides the image data DG to the projector 10 via the second communication device 250, and causes the projector 10 to project the first pattern image GOP1. As a result, the first pattern image GOP1 is projected from the projector 10, and a first projection image GP1 corresponding to the first pattern image GOP1 appears on the projection surface SP.

[0052] Next, application screen control unit 233 displays fifth user interface image GAU5 on display 210 (step Sa10). The fifth user interface image GAU5 is an image used for acquiring, by manually capturing a first captured image GC1 including the first projected image GP1 and the projection surface SP, as shown in FIG. Specifically, the fifth user interface image GAU5 includes a viewfinder display area H51 that displays the viewfinder image of the imaging device 220, string information H52 that informs the user U how to capture the first projection image GP1, and a shutter button image H53.

[0053] In this embodiment, the viewfinder display area H51 is a rectangular area having a long side ALH and a short side ASH with the same aspect ratio as the imaging range B. In the fifth user interface image GAU5, the viewfinder display area H51 is arranged so that the short side ASD of the display 210 and the short side ASH of the viewfinder display area H51 are parallel to each other. This arrangement makes the display area of ​​the viewfinder display area H51 larger than when the viewfinder display area H51 is displayed so that the short side ASH of the viewfinder display area H51 and the long side ALD of the display 210 are parallel to each other. The character string information H52 is superimposed on the viewfinder display area H51, and visibility is ensured without being obstructed by the viewfinder display area H51. The shutter button image H53 is an image for inputting an operation to instruct capture, and the display area of ​​the shutter button image H53 functions as a touch button. When the user U touches the shutter button image H53, an operation to instruct capture is input.

[0054] 6, when the input acquisition unit 234 receives an input of an operation instructing to capture an image (step Sa11), the imaging control unit 232 executes imaging by controlling the imaging device 220 (step Sa12). By this imaging, a first captured image GC1 including a first projected image GP1 corresponding to the first pattern image GOP1 and the projection surface SP is acquired.

[0055] Next, the correction unit 235 generates first correction data for correcting the shape of the target image GO based on the first captured image GC1 (step Sa13). The first correction data is data for correcting the shape of the target image GO using the first method described above. In this embodiment, the first correction data is data including the first correction amount described above.

[0056] Then, the projection control unit 231 transmits the first correction data to the projector 10 via the second communication device 250 (step Sa14). In the projector 10, for example, the first control unit 101 of the first processing device 100 corrects the shape of the target image GO based on the first correction data, thereby suppressing distortion of the projected image GP.

[0057] After that, application screen control unit 233 displays sixth user interface image GAU6 on display 210 (step Sa15). A sixth user interface image GAU6 is an image that notifies the user U that the correction has been completed, as shown in FIG.

[0058] As shown in FIG. 6, if the second method is selected by the user U in the above-mentioned step Sa3 (step Sa4: second method), as shown in FIG. 7, the application screen control unit 233 displays the seventh user interface image GAU7 on the display 210 (step Sa16). The seventh user interface image GAU7, as shown in FIG. 9, is an image that notifies the user U of the following three items: The first item is to manually adjust the position of the projector 10 to a position where the projected image GP is larger than the projection range of the projection surface SP. The second item is an example of a state where the projected image GP is larger than the projection range of the projection surface SP. The third item is to input an operation that indicates that the position of the projector 10 has been adjusted. In the seventh user interface image GAU7, the first item is notified by character string information H71, the second item is notified by an illustration H72, and the third item is notified by a button image H73. In the display 210, the display area of ​​the button image H73 functions as a touch button. When the user U touches the button image H73, an operation is input indicating that the position of the projector 10 has been adjusted and that the projected image GP is larger than the projection range of the projection surface SP.

[0059] As shown in FIG. 7, when the input acquisition unit 234 receives an operation input indicating that the projection image GP is larger than the projection range of the projection surface SP (step Sa17), the application screen control unit 233 displays the eighth user interface image GAU8 on the display 210 (step Sa18). The eighth user interface image GAU8, as shown in FIG. 9, is an image that notifies the user U of the following three items. The first item is to capture an image for each projection of the second pattern image GOP2 and the third pattern image GOP3, which are different from each other. The second item is an explanation of how to capture an image, specifically, the orientation of the display 210 when capturing an image. The third item is an input of an operation that instructs the projector 10 to project the second pattern image GOP2. By notifying the user U of the first item, the user U is notified that the second method of the present embodiment includes capturing images two times.

[0060] In the eighth user interface image GAU8, the first item is notified by character string information H81, the second item is notified by character string information H82 and an illustration H83, and the third item is notified by a button image H84. On the display 210, the display area of ​​the button image H84 functions as a touch button. When the user U touches the button image H84, an operation to start projection of the second pattern image GOP2 is input.

[0061] 7, when the input acquisition unit 234 receives an input of an operation instructing to project the second pattern image GOP2 (step Sa19), the projection control unit 231 performs control to project the second pattern image GOP2 from the projector 10 (step Sa20). The specific process of the projection control unit 231 in step Sa20 is similar to that in step Sa9. As a result of the process of step Sa20, the second pattern image GOP2 is projected from the projector 10, and a second projection image GP2 corresponding to the second pattern image GOP2 appears on the projection surface SP.

[0062] Next, application screen control unit 233 displays ninth user interface image GAU9 on display 210 (step Sa21). The ninth user interface image GAU9 is an image used for acquiring, by manually capturing a second captured image GC2 including the second projected image GP2 and the projection surface SP, as shown in FIG. Specifically, the ninth user interface image GAU9 includes a viewfinder display area H91, character string information H92, and a shutter button image H93. The external shapes and orientations of the viewfinder display area H91, character string information H92, and shutter button image H93 are all the same as those of the viewfinder display area H51, character string information H52, and shutter button image H53 of the fifth user interface image GAU5 shown in Fig. 8. In addition, the imaging method indicated by the character string information H92 is based on the second method.

[0063] 7, when the input acquisition unit 234 receives an input of an operation instructing to capture an image (step Sa22), the imaging control unit 232 controls the imaging device 220 to capture an image of the projection surface SP (step Sa23). By this imaging, a second captured image GC2 including a second projected image GP2 corresponding to the second pattern image GOP2 and the projection surface SP is acquired.

[0064] Next, the correction unit 235 generates second correction data for correcting the shape of the target image GO based on the second captured image GC2 (step Sa24). The second correction data is data for correcting the shape of the target image GO using the second method described above. In this embodiment, the second correction data is data including the second correction amount described above.

[0065] Next, the correction unit 235 enlarges the pattern portion PT in the second pattern image GOP2 based on the contour J of the projection range indicated by the second correction data, and generates a third pattern image GOP3 including the pattern portion PT (step Sa25). Then, the projection control unit 231 performs control to project the third pattern image GOP3 from the projector 10 (step Sa26). Specifically, the projection control unit 231 provides the image data DG of the third pattern image GOP3 generated in step Sa25 to the projector 10 via the second communication device 250, and causes the projector 10 to project the third pattern image GOP3. As a result, the third pattern image GOP3 is projected from the projector 10, and a third projection image GP3 corresponding to the third pattern image GOP3 appears on the projection surface SP.

[0066] Next, application screen control unit 233 displays tenth user interface image GAU10 on display 210 (step Sa27). The tenth user interface image GAU10 is an image used to obtain, by manually capturing a third captured image GC3 including a third projected image GP3 corresponding to the third pattern image GOP3 and a projection surface SP, as shown in FIG. 9. The tenth user interface image GAU10, like the ninth user interface image GAU9, includes a viewfinder display area H101, character string information H102, and a shutter button image H103. Moreover, the external shapes of the viewfinder display area H101, the character string information H102, and the shutter button image H103 are all the same as the viewfinder display area H91, the character string information H92, and the shutter button image H93 of the ninth user interface image GAU9.

[0067] 7, when the input acquisition unit 234 receives an input of an operation instructing to capture an image (step Sa28), the imaging control unit 232 executes imaging by controlling the imaging device 220 (step Sa29). By this imaging, a third captured image GC3 including a third projected image GP3 corresponding to the third pattern image GOP3 and the projection surface SP is acquired.

[0068] Next, the correction unit 235 again generates second correction data for correcting the shape of the target image GO based on the third captured image GC3 (step Sa30). As a result, the second correction data with higher accuracy is obtained.

[0069] Then, the projection control unit 231 transmits the second correction data obtained in step Sa30 to the projector 10 via the second communication device 250 (step Sa31). In the projector 10, for example, the first control unit 101 of the first processing device 100 corrects the shape of the target image GO based on the second correction data, so that the projection image GP fits within the projection range of the projection surface SP.

[0070] Thereafter, application screen control unit 233 advances the process to step Sa15 described above, and displays sixth user interface image GAU6 on display 210.

[0071] In the above-described image correction process, instead of the projector 10 correcting the shape of the target image GO based on the first correction data or the second correction data, the information processing device 20 may correct the shape of the target image GO based on the first correction data or the second correction data. In this case, it is not necessary to transmit the first correction data in step Sa14 and the second correction data in step Sa31. Also, in this case, the information processing device 20 transmits the corrected target image GO to the projector 10.

[0072] B: Display processing of user interface images As shown in Figures 8 and 9, as the image correction process progresses, user interface images GAU from the first user interface image GAU1 to the tenth user interface image GAU10 are displayed one by one on the display 210 under the control of the application screen control unit 233 of the information processing device 20.

[0073] In addition, the application screen control unit 233 of this embodiment displays the user interface image GAU to be displayed in either a first orientation K1 or a second orientation K2 depending on whether the user interface image GAU corresponds to a second image used in acquiring the captured image GC including the projection surface SP onto which the target image GO is projected from the projector 10 and the projection image GP that appears on the projection surface SP by projecting the target image GO from the projector 10. Of the group of user interface images GAU in this embodiment, the fifth user interface image GAU5, the ninth user interface image GAU9, and the tenth user interface image GAU10 correspond to the second image. Note that, of the group of user interface images GAU, the images other than the second image are referred to as "first images." That is, in this embodiment, the first images are the first user interface image GAU1 to the fourth user interface image GAU4, and the sixth user interface image GAU6 to the eighth user interface image GAU8.

[0074] FIG. 10 is an explanatory diagram of the first orientation K1, the second orientation K2, and the orientation L of the user interface image GAU. 10, the first direction K1 is along the long side ALD of the display 210. The second direction K2 is along the short side ASD of the display 210.

[0075] Furthermore, the orientation L of a user interface image GAU is determined by the orientation of an image object GAUO, the orientation of which is uniquely determined, among the image objects GAUO contained in the user interface image GAU. For example, as shown in form A of FIG. 10, when an image object GAUO is an image that notifies a user U of information by a character string, the orientation L of the image object GAUO is the up-down orientation of the characters in the character string. For example, when the image object GAUO is an image showing a picture such as an icon, as shown in form B of Fig. 10, the up-down direction of the picture is the up-down direction of the picture. Note that the picture does not include only simple figures such as circles and squares whose up-down direction is not specified.

[0076] FIG. 11 is a diagram showing an example of a display process of the user interface image GAU in accordance with the progress of the image correction process. 11, the application screen control unit 233 first displays the orientation L of the first user interface image GAU1 to the second user interface image GAU2 in the first orientation K1 (step Sb1). By the processing of step Sb1, for example, in the second user interface image GAU2, the first selection candidate image H21 and the second selection candidate image H22 are displayed facing the first orientation K1, as shown in FIG.

[0077] Next, as shown in Fig. 11, when the first method is selected in second user interface image GAU2 (step Sb2: first method), application screen control unit 233 displays third user interface image GAU3 to fourth user interface image GAU4 facing in first orientation K1 (step Sb3). By the processing of step Sb3, for example, in fourth user interface image GAU4, character string information H41, H42, illustration H43, etc. are displayed facing in first orientation K1, as shown in Fig. 8.

[0078] Next, application screen control unit 233 displays fifth user interface image GAU5 corresponding to the second image in second orientation K2 (step Sb4). By the processing of step Sb4, as shown in FIG. 8, in fifth user interface image GAU5, character string information H52 is displayed facing second orientation K2. When the fifth user interface image GAU5 is displayed in the second orientation K2, many users U hold the information processing device 20 in a position in which the short side ASD of the display 210 is close to the vertical CV, that is, in the above-mentioned horizontal orientation, in order to make it easier to read the character string information H52 indicating the imaging method. Therefore, by displaying the fifth user interface image GAU5 facing in the second orientation K2, the user U is induced to hold the information processing device 20 horizontally, even when the fourth user interface image GAU4 immediately before the fifth user interface image GAU5 is displayed facing in the first orientation K1.

[0079] On the other hand, the first projection image GP1 corresponding to the projection of the first pattern image GOP1 is a rectangle long in the horizontal CH. In addition, in the information processing device 20 of this embodiment, as described above, the orientations of the long side ALB and the short side ASB of the imaging range B coincide with the orientations of the long side ALD and the short side ASD of the display 210. Therefore, by holding the information processing device 20 horizontally, the first captured image GC1 including the entire first projected image GP1 is more reliably acquired. As a result, it is possible to prevent a situation in which a part of the pattern is missing from the first captured image GC1, and therefore it is possible to prevent a situation in which normal correction data cannot be obtained due to the missing pattern.

[0080] Furthermore, character string information H52 included in fifth user interface image GAU5 includes content notifying user U to hold information processing device 20 horizontally when capturing an image. Since the fifth user interface image GAU5 includes the character string information H52, an instruction to hold the information processing device 20 horizontally is more reliably conveyed to the user U.

[0081] As shown in FIG. 11, thereafter, application screen control unit 233 displays sixth user interface image GAU6 facing first orientation K1 (step Sb5), and ends the process.

[0082] Furthermore, when the second method is selected in second user interface image GAU2 (step Sb2: second method), application screen control unit 233 displays seventh user interface image GAU7 to eighth user interface image GAU8 facing the first orientation K1 (step Sb6). By the processing of step Sb6, for example, in eighth user interface image GAU8, character string information H81, H82, illustration H83, etc. are displayed facing the first orientation K1, as shown in FIG.

[0083] Next, as shown in Fig. 11, application screen control unit 233 displays ninth user interface image GAU9 to tenth user interface image GAU10, which correspond to the second image, facing in the second orientation K2 (step Sb7). By the processing of step Sb7, as shown in Fig. 9, in ninth user interface image GAU9, character string information H92 is displayed facing in the second orientation K2, and in tenth user interface image GAU10, character string information H102 is displayed facing in the second orientation K2. As with the process of step Sb4, the process of step Sb7 induces the user U to hold the information processing device 20 horizontally even when the eighth user interface image GAU8 immediately before the ninth user interface image GAU9 is displayed facing the first orientation K1. By holding the information processing device 20 horizontally, the second captured image GC2 including the entire second projected image GP2 that is long in the horizontal direction CH, and the third captured image GC3 including the entire third projected image GP3 are more reliably acquired. This prevents a situation in which some patterns are missing from the second captured image GC2 and the third captured image GC3, which can prevent a situation in which normal correction data cannot be obtained due to the missing patterns.

[0084] Furthermore, both character string information H92 contained in the ninth user interface image GAU9 and character string information H102 contained in the tenth user interface image GAU10 include content that notifies user U to hold information processing device 20 horizontally when capturing an image. Therefore, similarly to the fifth user interface image GAU5, an instruction to hold information processing device 20 horizontally is more reliably conveyed to user U.

[0085] The second processing device 230 included in the information processing device 20 described above executes the following series of processes. One of the processes is to receive an input for selecting either a first method for correcting the shape of the target image GO projected from the projector 10 onto the projection surface SP, or a second method for correcting the shape of the target image GO, which is different from the first method (FIG. 6: step Sa3). Another of the processes is to output a signal for causing the projector 10 to project the first pattern image GOP1 when the first method is selected (FIG. 6: step Sa9). Another of the processes is to generate first correction data for correcting the shape of the target image GO using the first method, based on a first projected image GP1 that appears on the projection surface SP by projecting the first pattern image GOP1, which is acquired by capturing an image of the projection surface SP, and a first captured image GC1 including the projection surface SP (FIG. 6: step Sa13). Also, one of the processes is to output a signal to the projector 10 to project a second pattern image GOP2 different from the first pattern image GOP1 when the second method is selected (FIG. 7: step Sa20). Also, one of the processes is to generate second correction data for correcting the shape of the target image GO using the second method based on a second projected image GP2 that appears on the projection surface SP by projecting the second pattern image GOP2 and a second captured image GC2 that includes the projection surface SP, which is acquired by capturing an image of the projection surface SP (FIG. 7: step Sa24).

[0086] The second processing device 230 executes this series of processes, whereby appropriate first pattern images GOP1 and second pattern images GOP2 that are different from each other are used in the first method and the second method. Therefore, in both the first method and the second method, for example, a situation in which an image of the same pattern is used, resulting in low accuracy of correction data or failure to obtain correction data normally, can be prevented. As a result, a high-quality projected image GP can be obtained using more accurate correction data, and the occurrence of a situation in which the user U has to repeat correction operations multiple times is reduced, improving convenience for the user U.

[0087] In this embodiment, receiving the input (FIG. 6: step Sa3) includes receiving a selection as to whether or not the projection surface SP includes a contour J that defines the projection range. Therefore, the user U can select either the first method or the second method depending on whether the projection surface SP includes the contour J of the projection range, and perform the correction appropriately.

[0088] In this embodiment, the first method includes correcting the shape of the target image GO based on the surface shape of the projection surface SP. This first method makes it possible to appropriately perform correction using the first pattern image GOP1 when the projection surface SP is not a flat surface and distortion occurs in the projected image GP due to the shape of the surface.

[0089] In this embodiment, the second method includes a correction for fitting the projected image GP that appears on the projection surface SP by projecting the target image GO by the projector 10 within the projection range on the projection surface SP. This second method allows the user U to appropriately perform correction to fit the projected image GP within the contour J of the projection range, using the second pattern image GOP2.

[0090] In this embodiment, the series of processes executed by the second processing device 230 includes the following processes. That is, when the second method is selected, a signal is output to the projector 10 to project a third pattern image GOP3 different from the first pattern image GOP1 and the second pattern image GOP2 (FIG. 7: step Sa26). Also, generating the second correction data includes generating the second correction data for correcting the shape of the target image GO using the second method based on the third projected image GP3 that appears on the projection surface SP by projecting the third pattern image GOP3, which is acquired by capturing an image of the projection surface SP, the third captured image GC3 including the projection surface SP, and the second captured image GC2 (FIG. 7: steps Sa24 to Sa30). Through the above process, the second correction data is obtained more accurately.

[0091] In this embodiment, the second pattern image GOP2 has a smaller number of detection points than the first pattern image GOP1. This increases the speed of the process using the second pattern image GOP2. Also, the generation of the second correction data is more resistant to detection errors and noise.

[0092] Moreover, the above-mentioned information processing device 20 is a portable device having a display 210 having a long side ALD and a short side ASD. The second processing device 230 of the information processing device 20 executes the following series of processes. One of the processes is to display a first image on the display 210 (FIG. 11: steps Sb1 to Sb3, Sb5, and Sb6). In this embodiment, the first images are the first user interface image GAU1 to the fourth user interface image GAU4, the sixth user interface image GAU6 to the eighth user interface image GAU8, and the tenth user interface image GAU10, respectively. Also, one of the processes is to acquire, by capturing an image GC including the projection surface SP onto which the target image GO is projected from the projector 10 and the projected image GP that appears on the projection surface SP by projecting the target image GO from the projector 10, and to display a second image used in the capture in an orientation along the short side ASD of the display 210 (FIG. 11: steps Sb4, Sb7). In this embodiment, the second images are the fifth user interface image GAU5, the eighth user interface image GAU8, and the ninth user interface image GAU9, respectively. The second processing device 230 executes this series of processes, so that a captured image GC including the entire projected image GP is reliably acquired by capturing an image of the user U.

[0093] In this embodiment, the series of processes executed by the second processing device 230 includes the following processes. That is, one of the processes is to acquire the captured image GC by capturing an image of the projection surface SP when an input instructing imaging via a second image is received (FIG. 6: step Sa12, FIG. 7: steps Sa23 and Sa29). Another process is to generate correction data for correcting the shape of the target image GO based on the captured image GC (FIG. 6: step Sa10, FIG. 7: steps Sa24 and Sa30). By the second processing device 230 executing this series of processes, it is possible to prevent a situation in which normal correction data cannot be obtained because a part of the projected image GP is not included in the captured image GC.

[0094] In this embodiment, the second image includes information notifying the user to hold the portable information processing device 20 in a position in which the short side ASD of the display 210 is closer to the vertical CV than to the horizontal CH. In this embodiment, the information is each of the character string information H52, H92, and H102. Therefore, the instruction to hold the information processing device 20 horizontally is more reliably conveyed to the user U.

[0095] In this embodiment, the imaging range B in which the portable information processing device 20 images the projection surface SP has a long side ALB and a short side ASB, and the short side ASB of the imaging range B is aligned with the short side ASD of the display 210, and the long side ALB of the imaging range B is aligned with the long side SLD of the display 210. Therefore, by holding the information processing device 20 horizontally, a captured image GC including the entire projected image GP can be obtained more reliably.

[0096] 3. Variations The above-mentioned exemplary embodiments can be modified in various ways. Specific modified embodiments that can be applied to the above-mentioned embodiments are exemplified below. Two or more embodiments arbitrarily selected from the following examples can be appropriately combined with each other to the extent that no technical contradiction occurs.

[0097] (Transformation form 1) The first pattern image GOP1 used in the first method is not limited to the pattern shown in Fig. 4. That is, the first pattern image GOP1 may be a pattern in which each dot E has a non-uniform luminance distribution and each dot E is surrounded by an area painted with a single color. By using this pattern, the effect of noise during imaging is suppressed in the detection of the dots E. In addition, the pattern of the second pattern image GOP2 used in the second method is not limited to a checkered pattern. For example, instead of the checkered pattern, a cross line pattern in which cross lines are provided at each position corresponding to the checkered corners PTC, and a pattern in which ArUco markers are provided at each of the positions may be used.

[0098] (Transformation form 2) When a user interface image GAU corresponding to the second image is displayed on the display 210, the user U may be notified to hold the portable information processing device 20 in a position in which the short side ASD of the display 210 is closer to the vertical CV than to the horizontal CH, as follows: That is, the information processing device 20 executes at least one of the following first to fourth processes instead of or in conjunction with displaying the character string information H52, H92, and H102.

[0099] The first process is a process for notifying the user by changing the viewfinder image in the viewfinder display areas H51, H91, and H101 to another image. Specifically, as shown in FIG. 12, the information processing device 20 includes a sensor 260 that detects the attitude of the information processing device 20, and the second processing device 230 executes the following operation when the second image is displayed. That is, when the output of the sensor 260 indicates that the information processing device 20 is held vertically, the second processing device 230 switches the viewfinder image in the viewfinder display areas H51, H91, and H101 to another image, such as an image of a sandstorm. This switching of the image indicates to the user U that imaging is not possible in the vertically held state. The image may be either a moving image or a still image. The sensor 260 is, for example, an acceleration sensor or a gyro sensor.

[0100] The second process is a process for performing the above notification by rotating the viewfinder image in the viewfinder display areas H51, H91, and H101 by 90 degrees. Specifically, the information processing device 20 has the configuration shown in Fig. 12, and the second processing device 230 executes the following operation when the second image is displayed. That is, when the output of the sensor 260 indicates that the information processing device 20 is held vertically, the second processing device 230 displays the viewfinder image in the viewfinder display areas H51, H91, and H101 rotated by 90 degrees. The viewfinder image being displayed rotated by 90 degrees suggests to the user U that he or she needs to rotate the information processing device 20 by 90 degrees and hold it horizontally.

[0101] The third process is a process of making the above notification by vibration of a vibrator. Specifically, the information processing device 20 further includes a vibrator in addition to the configuration shown in Fig. 12. The second processing device 230 executes the following operation when the second image is displayed. That is, the information processing device 20 has the configuration shown in Fig. 12, and when the output of the sensor 260 indicates that the information processing device 20 is being held vertically, the second processing device 230 outputs a vibration that gives a negative feeling to the user U, such as "vvv, vvv", by the vibrator. After that, when the user U holds it horizontally, the vibration that gives a positive feeling to the user U, such as "vvv, vvv", is output and then stopped.

[0102] The fourth process is a process of performing the above notification by the projection image GP of the projector 10. Specifically, the information processing device 20 has a configuration shown in FIG. 12, and when the output of the sensor 260 indicates that the information processing device 20 is held vertically, the second processing device 230 outputs a signal to make the projector 10 project an image notifying the user that the information processing device 20 is held horizontally. After that, when the output of the sensor 260 indicates that the information processing device 20 is held horizontally, the second processing device 230 makes the projector 10 stop projecting the image notifying the user that the information processing device 20 is held horizontally. When displaying a second image on the information processing device 20, the second processing device 230 may output a signal to make the projector 10 project an image notifying the user that the information processing device 20 is held horizontally during a period in which the second image is displayed, or may output the signal before the second image is displayed, for example, when transitioning from the first image to the second image.

[0103] Through these first to fourth processes, an instruction to hold the information processing device 20 horizontally is more reliably conveyed to the user U.

[0104] (Transformation form 3) The information processing device 20 may display the user interface image GAU corresponding to the first image in a direction according to the attitude of the information processing device 20. Specifically, the information processing device 20 has the configuration shown in Fig. 12. Then, the second processing device 230 executes the following operations in the process of displaying the first image on the display 210 (Fig. 11: steps Sb1, Sb3, Sb5, and Sb7). That is, the second processing device 230 displays the first image on the display 210 in either a first orientation K1 in which the first image is oriented along the long side ALD of the display 210, or a second orientation K2 in which the first image is oriented along the short side ASD of the display 210, based on the output of the sensor 260. Through this process, the first image is displayed in an appropriate orientation depending on the attitude of the information processing device 20, that is, the way the user U holds it.

[0105] (Transformation form 4) In the third modified embodiment, the second processing device 230 of the information processing device 20 may further perform the following process. That is, in the process of displaying a first image on the display 210 (FIG. 11: steps Sb1, Sb3, Sb5, and Sb6), when the output of the sensor 260 indicates that the long side ALD of the display 210 is closer to the vertical CV than to the horizontal CH, the second processing device 230 displays the first image in a first orientation K1 in which the first image is oriented along the long side ALD of the display 210. In addition, in the process of displaying the second image in a second orientation K2 in which the second image is oriented along the short side ASD of the display 210 (Figure 11: steps Sb4 and Sb7), the second processing device 230 displays the first image in a first orientation K1 in which the first image is oriented along the long side ALD of the display 210, and then displays the second image in a second orientation K2 in which the second image is oriented along the short side ASD of the display 210. As a result of this processing, even if the first image immediately before the second image is displayed in the first orientation K1, the second image is displayed in the second orientation K2, thereby inducing the user U to hold the information processing device 20 horizontally.

[0106] (Transformation form 5) The information processing device 20 is not limited to being implemented by one computer, but may be implemented by multiple computers. Specifically, each functional unit of the second processing device 230 of the information processing device 20 may be implemented by each processor included in each of the multiple computers.

[0107] (Transformation form 6) The first processing device 100 of the projector 10 may include one or more of the functional units included in the second processing device 230 of the information processing device 20. In this case, at least a part of the application 242 and the resource data 244 may be stored in the first storage device 120 of the projector 10.

[0108] (Transformation form 7) In the second method, steps Sa25 to Sa30 may be omitted. That is, the third pattern image GOP3 may not be used, and the second correction data generated based on the second pattern image GOP2 may be transmitted to the projector 10 in step Sa31. In this case, the size of the pattern portion PT in the second pattern image GOP2 may be set according to the certainty that it fits within the projection range and the accuracy of the correction amount, and may be different from the above-mentioned embodiment.

[0109] 4. Summary of this disclosure The following is a summary of this disclosure.

[0110] (Appendix 1) receiving an input for selecting either a first method for correcting a shape of an image projected from a projector onto a projection surface, or a second method for correcting the shape of the image, the second method being different from the first method; outputting a signal for causing the projector to project a first pattern image when the first method is selected; generating first correction data for correcting a shape of the image using the first method based on a first captured image including a first projected image that appears on the projection surface by projecting the first pattern image and the projection surface, the first captured image being acquired by capturing an image of the projection surface; outputting, when the second method is selected, a signal for projecting a second pattern image different from the first pattern image to the projector; generating second correction data for correcting a shape of the image using the second method based on a second captured image including a second projected image that appears on the projection surface by projecting the second pattern image and the projection surface, the second captured image being acquired by capturing an image of the projection surface; 13. An image correction method comprising: According to Appendix 1, user convenience is improved.

[0111] (Appendix 2) Receiving the input includes: and receiving a selection as to whether the projection surface includes an outline that defines a projection range. 2. The image correction method according to claim 1. According to Supplementary Note 2, either the first method or the second method can be selected depending on whether the projection surface includes the contour of the projection range, and correction can be performed appropriately.

[0112] (Appendix 3) The first method comprises: correcting the shape of the image based on the surface shape of the projection screen; Includes 3. An image correction method according to claim 1 or 2. According to Supplementary Note 3, correction can be appropriately performed using the first pattern image when the projection surface is not a flat surface.

[0113] (Appendix 4) The second method comprises: a correction for fitting a projected image that appears on the projection surface by projecting the image by the projector within a projection range on the projection surface; 4. An image correction method according to any one of claims 1 to 3. According to Supplementary Note 4, the second pattern image can be used to appropriately perform correction for fitting the projected image within the contour of the projection range.

[0114] (Appendix 5) outputting a signal to the projector to project the first pattern image and a third pattern image different from the second pattern image when the second method is selected; Generating the second correction data includes: generating the second correction data for correcting a shape of the image using the second method based on a third captured image including a third projected image appearing on the projection surface by projecting the third pattern image and the projection surface, the third captured image being acquired by capturing an image of the projection surface, and the second captured image; 5. The image correction method according to claim 4. According to Supplementary Note 5, the second correction data is determined more accurately.

[0115] (Appendix 6) the second pattern image has a smaller number of detection points than the first pattern image; 6. An image correction method according to any one of claims 1 to 5. According to Supplementary Note 6, the speed of the processing using the second pattern image can be increased. Also, the generation of the second correction data can be made more resistant to detection errors and noise.

[0116] (Appendix 7) a processor; The processing device includes: receiving an input for selecting either a first method for correcting a shape of an image projected from a projector onto a projection surface, or a second method for correcting the shape of the image, the second method being different from the first method; outputting a signal for causing the projector to project a first pattern image when the first method is selected; generating first correction data for correcting a shape of the image using the first method based on a first captured image including a first projected image that appears on the projection surface by projecting the first pattern image and the projection surface, the first captured image being acquired by capturing an image of the projection surface; outputting, when the second method is selected, a signal for projecting a second pattern image different from the first pattern image to the projector; generating second correction data for correcting a shape of the image using the second method based on a second captured image including a second projected image that appears on the projection surface by projecting the second pattern image and the projection surface, the second captured image being acquired by capturing an image of the projection surface; An information processing device that executes the above. According to Appendix 7, the same effect as that of Appendix 1 is achieved.

[0117] (Appendix 8) receiving an input for selecting either a first method for correcting a shape of an image projected from a projector onto a projection surface, or a second method for correcting the shape of the image, the second method being different from the first method; outputting a signal for causing the projector to project a first pattern image when the first method is selected; generating first correction data for correcting a shape of the image using the first method based on a first captured image including a first projected image that appears on the projection surface by projecting the first pattern image and the projection surface, the first captured image being acquired by capturing an image of the projection surface; outputting, when the second method is selected, a signal for projecting a second pattern image different from the first pattern image to the projector; generating second correction data for correcting a shape of the image using the second method based on a second captured image including a second projected image that appears on the projection surface by projecting the second pattern image and the projection surface, the second captured image being acquired by capturing an image of the projection surface; A program that causes a computer to execute the following. According to Appendix 8, the same effect as that of Appendix 1 is achieved. [Explanation of symbols]

[0118] 1...projection system, 10...projector, 20...information processing device, 220...imaging device, 230...second processing device, 231...projection control unit, 232...imaging control unit, 235...correction unit, 242...application, GC...captured image, GO...target image, GOP1...first pattern image, GOP2...second pattern image, GOP3...third pattern image, GP...projected image, J...contour, OP...projected object, SP...projection surface.

Claims

1. receiving an input for selecting either a first method for correcting a shape of an image projected from a projector onto a projection surface, or a second method for correcting the shape of the image and different from the first method; outputting a signal for causing the projector to project a first pattern image when the first method is selected; generating first correction data for correcting a shape of the image using the first method based on a first captured image including a first projected image that appears on the projection surface by projecting the first pattern image and the projection surface, the first captured image being acquired by capturing an image of the projection surface; outputting, when the second method is selected, a signal for projecting a second pattern image different from the first pattern image to the projector; generating second correction data for correcting a shape of the image using the second method based on a second captured image including a second projected image that appears on the projection surface by projecting the second pattern image and the projection surface, the second captured image being acquired by capturing an image of the projection surface; 13. An image correction method comprising:

2. Receiving the input includes: and receiving a selection as to whether the projection surface includes an outline that defines a projection range. The image correction method according to claim 1 .

3. The first method comprises: correcting the shape of the image based on the surface shape of the projection screen; Includes The image correction method according to claim 1 .

4. The second method comprises: a correction for fitting a projected image that appears on the projection surface by projecting the image by the projector within a projection range on the projection surface; The image correction method according to claim 1 .

5. outputting a signal to the projector to project the first pattern image and a third pattern image different from the second pattern image when the second method is selected; Generating the second correction data includes: generating the second correction data for correcting a shape of the image using the second method based on a third captured image including a third projected image appearing on the projection surface by projecting the third pattern image and the projection surface, the third captured image being acquired by capturing an image of the projection surface, and the second captured image; The image correction method according to claim 4.

6. the second pattern image has a smaller number of detection points than the first pattern image; The image correction method according to any one of claims 1 to 5.

7. a processor; The processing device includes: receiving an input for selecting either a first method for correcting a shape of an image projected from a projector onto a projection surface, or a second method for correcting the shape of the image and different from the first method; outputting a signal for causing the projector to project a first pattern image when the first method is selected; generating first correction data for correcting a shape of the image using the first method based on a first captured image including a first projected image that appears on the projection surface by projecting the first pattern image and the projection surface, the first captured image being acquired by capturing an image of the projection surface; outputting, when the second method is selected, a signal for projecting a second pattern image different from the first pattern image to the projector; generating second correction data for correcting a shape of the image using the second method based on a second captured image including a second projected image that appears on the projection surface by projecting the second pattern image and the projection surface, the second captured image being acquired by capturing an image of the projection surface; An information processing device that executes the above.

8. receiving an input for selecting either a first method for correcting a shape of an image projected from a projector onto a projection surface, or a second method for correcting the shape of the image and different from the first method; outputting a signal for causing the projector to project a first pattern image when the first method is selected; generating first correction data for correcting a shape of the image using the first method based on a first captured image including a first projected image that appears on the projection surface by projecting the first pattern image and the projection surface, the first captured image being acquired by capturing an image of the projection surface; outputting, when the second method is selected, a signal for projecting a second pattern image different from the first pattern image to the projector; generating second correction data for correcting a shape of the image using the second method based on a second captured image including a second projected image that appears on the projection surface by projecting the second pattern image and the projection surface, the second captured image being acquired by capturing an image of the projection surface; A program that causes a computer to execute the following.

Images