Projection method, system, and program

Abstract

To enable various adjustments of a projector.SOLUTION: A projection method includes: generating first correspondence between a plurality of first coordinates in a first imaged pictured by a first camera and a plurality of second coordinates in a second imaged pictured by a second camera; calculating a first external parameter showing a position and / or posture of the second camera relative to the first camera on the basis of a first internal parameter of the first camera, a second internal parameter of the second camera, and the first correspondence; obtaining a plurality of third coordinates which are three-dimensional coordinates on a projection surface in a projection picture on the basis of the first internal parameter, the second internal parameter, the first correspondence, and the first external parameter; generating second correspondence between the plurality of first coordinates and the plurality of third coordinates; generating third correspondence between the plurality of first coordinates and a plurality of fourth coordinates which are coordinates of a plurality of third pixels on a display panel of a projector; and generating fourth correspondence between the plurality of third coordinates and the plurality of fourth coordinates on the basis of the second correspondence and the third correspondence.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to a projection method, system, and program.

Background Art

[0002] For example, Patent Document 1 discloses a system that identifies the correspondence between the position on the image projected onto the object to be projected and the position on the captured image obtained by capturing the object to be projected by an imaging device, based on the image projected from the projector onto the object to be projected and the captured image of the object to be projected captured by the imaging device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In order for the projector to perform various adjustments, it is necessary to identify what correspondence relationship exists between the two-dimensional coordinate system representing the position on the display panel of the projector and the three-dimensional coordinates of the projected image on the screen. However, Patent Document 1 does not disclose this point.

Means for Solving the Problems

[0005] A projection method according to one aspect of the present disclosure includes projecting a projection image from a projector onto a projection surface, obtaining a first captured image by capturing the projection image with a first camera having first internal parameters, obtaining a second captured image by capturing the projection image with a second camera having second internal parameters, generating a first correspondence relationship in which a plurality of first coordinates, which are coordinates of a plurality of first pixels in the first captured image, and a plurality of second coordinates, which are coordinates of a plurality of second pixels corresponding to the plurality of first pixels in the second captured image, are associated with each other, calculating a first external parameter indicating one or both of the position and orientation of the second camera with respect to the first camera based on the first internal parameters, the second internal parameters, and the first correspondence relationship, obtaining a plurality of third coordinates, which are three-dimensional coordinates on the projection surface of a first partial image included in both the first captured image and the second captured image in the projection image, based on the first internal parameters, the second internal parameters, the first correspondence relationship, and the first external parameter, generating a second correspondence relationship in which the plurality of first coordinates and the plurality of third coordinates are associated with each other, generating a third correspondence relationship in which a plurality of fourth coordinates, which are coordinates of a plurality of third pixels in the display panel of the projector, and the plurality of first coordinates are associated with each other, and generating a fourth correspondence relationship in which the plurality of third coordinates and the plurality of fourth coordinates are associated with each other based on the second correspondence relationship and the third correspondence relationship.

[0006] A system according to one aspect of the present disclosure includes an optical device of a projector, a first camera, a second camera, and a processing device that controls operations of the optical device, the first camera, and the second camera. The processing device projects a projection image from the projector onto a projection surface, acquires a first captured image by imaging the projection image with the first camera having first internal parameters, acquires a second captured image by imaging the projection image with the second camera having second internal parameters, generates a first correspondence relationship in which a plurality of first coordinates that are coordinates of a plurality of first pixels in the first captured image and a plurality of second coordinates that are coordinates of a plurality of second pixels corresponding to the plurality of first pixels in the second captured image are associated with each other, calculates a first external parameter indicating one or both of the position and orientation of the second camera with respect to the first camera based on the first internal parameters, the second internal parameters, and the first correspondence relationship, obtains a plurality of third coordinates that are three-dimensional coordinates of a first partial image included in both the first captured image and the second captured image on the projection surface in the projection image based on the first internal parameters, the second internal parameters, the first correspondence relationship, and the first external parameter, generates a second correspondence relationship in which the plurality of first coordinates and the plurality of third coordinates are associated with each other, generates a third correspondence relationship in which a plurality of fourth coordinates that are coordinates of a plurality of third pixels in a display panel of the projector and the plurality of first coordinates are associated with each other, and generates a fourth correspondence relationship in which the plurality of third coordinates and the plurality of fourth coordinates are associated with each other based on the second correspondence relationship and the third correspondence relationship.

[0007] A program according to an aspect of the present disclosure includes projecting a projection image from a projector onto a projection surface, obtaining a first captured image by capturing the projection image with a first camera having first internal parameters, obtaining a second captured image by capturing the projection image with a second camera having second internal parameters, generating a first correspondence relationship in which a plurality of first coordinates, which are coordinates of a plurality of first pixels in the first captured image, and a plurality of second coordinates, which are coordinates of a plurality of second pixels corresponding to the plurality of first pixels in the second captured image, are associated with each other, calculating a first external parameter indicating one or both of the position and orientation of the second camera with respect to the first camera based on the first internal parameters, the second internal parameters, and the first correspondence relationship, obtaining a plurality of third coordinates, which are three-dimensional coordinates on the projection surface of a first partial image included in both the first captured image and the second captured image in the projection image, based on the first internal parameters, the second internal parameters, the first correspondence relationship, and the first external parameter, generating a second correspondence relationship in which the plurality of first coordinates and the plurality of third coordinates are associated with each other, generating a third correspondence relationship in which a plurality of fourth coordinates, which are coordinates of a plurality of third pixels in the display panel of the projector, and the plurality of first coordinates are associated with each other, and generating a fourth correspondence relationship in which the plurality of third coordinates and the plurality of fourth coordinates are associated with each other based on the second correspondence relationship and the third correspondence relationship, and causing a computer to execute the above steps.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Embodiments for Carrying Out the Invention

[0009] Hereinafter, preferred embodiments according to the present disclosure will be described with reference to the accompanying drawings. Note that the dimensions and scales of each part in the drawings are appropriately different from the actual ones, and there are also some parts that are schematically shown for easy understanding. In addition, the scope of the present disclosure is not limited to these forms unless otherwise specified in the following description.

[0010] 1. First Embodiment 1-1. Outline of the System FIG. 1 is a diagram showing an outline of a system 100 used for a projection method according to the first embodiment. The system 100 is a projection system that projects a projection image G1 onto a projection surface SC.

[0011] The projection surface SC is the surface of an object such as a screen. In the example shown in FIG. 1, the projection surface SC is a concave curved surface such that the center of the projection surface SC is located deeper than the left and right sides. Note that the shape of the projection surface SC is not limited to the example shown in FIG. 1, and for example, the projection surface SC may be a convex curved surface such that the center of the projection surface SC is located in front of the left and right sides.

[0012] As shown in FIG. 1, the system 100 includes a projector 10-1, a first camera 20-1, a second camera 20-2, and a terminal device 30.

[0013] The projector 10-1 is a display device that projects the projection image G1 indicated by the video data IMG1 output from the terminal device 30 onto the projection surface SC. In the example shown in FIG. 1, the projection image G1 is projected onto a region shifted to the left side in FIG. 1 of the projection surface SC. Note that the projection position of the projection image G1 on the projection surface SC is not limited to the example shown in FIG. 1 and is arbitrary.

[0014] The projector 10-1 of the present embodiment has a function of controlling the operations of the first camera 20-1 and the second camera 20-2, and a function of adjusting the shape of the projection image G1 using the imaging results of the first camera 20-1 and the second camera 20-2.

[0015] Each of the first camera 20-1 and the second camera 20-2 is a digital camera having an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

[0016] The first camera 20-1 images a region RC1. The region RC1 is a region including a projected image G1 projected onto the projection surface SC. On the other hand, the second camera 20-2 images a region RC2. The region RC2 is a region including only a part of the projected image G1 projected onto the projection surface SC. Thus, the regions RC1 and RC2 have an overlapping region RC. Note that the first camera 20-1 may be a component of the projector 10-1. Also, the second camera 20-2 may be a component of a projector different from the projector 10-1.

[0017] The terminal device 30 is a computer having a function of supplying video data IMG1 to the projector 10-1. In the example shown in FIG. 1, the terminal device 30 is a notebook computer. Note that the terminal device 30 is not limited to a notebook computer, and may be, for example, a desktop computer, a smartphone, or a tablet terminal, or may be a video playback device, a DVD (Digital Versatile Disk) player, a Blu-ray disk player, a hard disk recorder, a television tuner device, a set-top box for CATV (Cable television), a video game machine, or the like.

[0018] 1-2. Projector FIG. 2 is a block diagram of the projector 10-1 used in the system 100 according to the first embodiment. In FIG. 2, in addition to the projector 10-1, the connection states of the first camera 20-1 and the second camera 20-2 to the projector 10-1 are shown.

[0019] As shown in FIG. 2, the projector 10-1 includes a storage device 11, a processing device 12, a communication device 13, an image processing circuit 14, an optical device 15, and an operation device 16. These are connected to be communicable with each other.

[0020] The storage device 11 is a storage device that stores programs executed by the processing device 12 and data processed by the processing device 12. The storage device 11 includes, for example, a hard disk drive or a semiconductor memory. Note that part or all of the storage device 11 may be provided in an external storage device or a server of the projector 10-1.

[0021] The storage device 11 stores a program PR1, first imaging data D1, second imaging data D2, first internal parameter information PC1, second internal parameter information PC2, third internal parameter information PC3, first external parameter information PC4, second external parameter information PC5, first coordinate information DP1, second coordinate information DP2, third coordinate information DP3, fourth coordinate information DP4, fifth coordinate information DP5, fifth transformed coordinate information DP6, shape information DP7, first marker coordinate information DP8, second marker coordinate information DP9, first correspondence information DC1, second correspondence information DC2, third correspondence information DC3, and fourth correspondence information DC4.

[0022] The program PR1 is a program for executing a projection method to be described in detail later.

[0023] The first imaging data D1 is information indicating a first imaging image GG1, which is described later and is obtained by the first camera 20-1 imaging the projection image G1.

[0024] The second imaging data D2 is information indicating a second imaging image GG2, which is described later and is obtained by the second camera 20-2 imaging the projection image G1.

[0025] The first internal parameter information PC1 is information indicating the first internal parameters of the first camera 20-1. The first internal parameters are known parameters representing optical characteristics such as the optical center, focal length, and aberration of the first camera 20-1, for example.

[0026] The second internal parameter information PC2 is information indicating the second internal parameters of the second camera 20-2. The second internal parameters are known parameters representing optical characteristics such as the optical center, focal length, and aberration of the second camera 20-2, for example.

[0027] The third internal parameter information PC3 is information indicating the third internal parameters which are the internal parameters of the projector 10-1. The third internal parameters are parameters representing optical characteristics such as the angle of view and optical center of the optical device 15 of the projector 10-1, for example. Note that the angle of view varies depending on the slow ratio or optical zoom. The optical center varies depending on the lens offset or lens shift.

[0028] The first external parameter information PC4 is information indicating the first external parameters indicating one or both of the position and orientation of the second camera 20-2 with respect to the first camera 20-1.

[0029] The second external parameter information PC5 is information indicating the second external parameters indicating one or both of the position and orientation of the projector 10-1 with respect to the first camera 20-1.

[0030] The first coordinate information DP1 is information indicating a plurality of first coordinates which are the coordinates of a plurality of first pixels in a first captured image GG1 described below indicated by the first captured data D1. The first pixel is a pixel of the first camera 20-1. The first coordinate is the coordinate of the first pixel in the coordinate system of the first camera 20-1.

[0031] The second coordinate information DP2 is information indicating a plurality of second coordinates that are the coordinates of a plurality of second pixels corresponding to a plurality of first pixels in a second captured image GG2 shown in the second captured data D2. The second pixel is a pixel of the second camera 20-2 and is a pixel corresponding to the first pixel. The second coordinate is the coordinate of the second pixel in the coordinate system of the second camera 20-2.

[0032] The third coordinate information DP3 is information indicating a plurality of third coordinates that are three-dimensional coordinates on a projection plane SC of a first partial image R1-1 (described later) included in both a first captured image GG1 (described later) shown in the first captured data D1 and a second captured image GG2 (described later) shown in the second captured data D2 in the projection image G1. In this specification, the three-dimensional coordinates on the projection plane SC are coordinates in a world coordinate system that is a three-dimensional coordinate system set in the real space where the projection plane SC is installed or a virtual three-dimensional space corresponding to the real space. Hereinafter, the three-dimensional coordinate system may be simply referred to as the "three-dimensional coordinate system". Also, the three-dimensional coordinates may be simply referred to as the "three-dimensional coordinates".

[0033] The fourth coordinate information DP4 is information indicating a plurality of fourth coordinates that are the coordinates of a plurality of third pixels in the display panel 15b of the projector 10-1. The third pixel is a pixel of the display panel 15b. The fourth coordinate is the coordinate of the third pixel in the coordinate system of the display panel 15b.

[0034] The fifth coordinate information DP5 is information indicating a plurality of fifth coordinates that are the coordinates of a plurality of reference points PR in the display panel 15b in the projection image G1.

[0035] The fifth transformed coordinate information DP6 is information indicating coordinates obtained by transforming the plurality of fifth coordinates indicated by the fifth coordinate information DP5 into three-dimensional coordinates.

[0036] The shape information DP7 is information indicating the three-dimensional shape on the projection plane SC of the second partial image R1-2 included in the first captured image GG1 indicated by the first captured data D1 but not included in the second captured image GG2 indicated by the second captured data D2 in the projected image G1, in the form of a coordinate group or the like. Note that the shape information DP7 may include information indicating the three-dimensional shape on the projection plane SC of the first partial image R1-1, which will be described later, and is included in both the first captured image GG1 indicated by the first captured data D1 and the second captured image GG2 indicated by the second captured data D2 in the projected image G1.

[0037] The first marker coordinate information DP8 is information indicating the three-dimensional coordinates of the marker MK, which will be described later, at the first timing.

[0038] The second marker coordinate information DP9 is information indicating the three-dimensional coordinates of the marker MK, which will be described later, at the second timing later than the first timing.

[0039] The first correspondence information DC1 is information indicating a first correspondence in which a plurality of first coordinates indicated by the first coordinate information DP1 and a plurality of second coordinates indicated by the second coordinate information DP2 are associated with each other. That is, the first correspondence indicated by the first correspondence information DC1 is the correspondence between the coordinate system of the first camera 20-1 and the coordinate system of the second camera 20-2.

[0040] The second correspondence information DC2 is information indicating a second correspondence in which a plurality of first coordinates indicated by the first coordinate information DP1 and a plurality of third coordinates indicated by the third coordinate information DP3 are associated with each other. That is, the second correspondence indicated by the second correspondence information DC2 is the correspondence between the coordinate system of the first camera 20-1 and the three-dimensional coordinate system.

[0041] The third correspondence information DC3 is information indicating a third correspondence in which a plurality of fourth coordinates indicated by the fourth coordinate information DP4 and a plurality of first coordinates indicated by the first coordinate information DP1 are associated with each other. That is, the third correspondence indicated by the third correspondence information DC3 is the correspondence between the coordinate system of the first camera 20-1 and the coordinate system of the display panel 15b of the projector 10-1.

[0042] The fourth correspondence relationship information DC4 is information indicating a fourth correspondence relationship in which a plurality of third coordinates indicated by the third coordinate information DP3 and a plurality of fourth coordinates indicated by the fourth coordinate information DP4 are associated with each other. That is, the fourth correspondence relationship indicated by the fourth correspondence relationship information DC4 is the correspondence relationship between the three-dimensional coordinate system and the coordinate system of the display panel 15b of the projector 10-1.

[0043] The processing device 12 is a processing device having a function of controlling each part of the projector 10-1 and a function of processing various data. The processing device 12 is configured to include a processor such as a CPU (Central Processing Unit), for example. Note that the processing device 12 may be configured by a single processor or may be configured by a plurality of processors. Also, part or all of the functions of the processing device 12 may be realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). Further, the processing device 12 may be integrated with at least a part of the image processing circuit 14.

[0044] The communication device 13 is a communication device capable of communicating with various devices, and acquires video data IMG1 from the terminal device 30 and communicates with each of the first camera 20-1 and the second camera 20-2. For example, the communication device 13 is a wired communication device such as a wired LAN (Local Area Network), a USB (Universal Serial Bus), or an HDMI (High Definition Multimedia Interface), or a wireless communication device such as an LPWA (Low Power Wide Area), a wireless LAN including Wi-Fi, or Bluetooth. Each of "HDMI", "Wi-Fi", and "Bluetooth" is a registered trademark.

[0045] The image processing circuit 14 is a circuit that performs necessary processing on the video data IMG1 from the communication device 13 and inputs it to the optical device 15. The image processing circuit 14 has, for example, a frame memory (not shown), expands the video data IMG1 in the frame memory, and appropriately executes various processes such as resolution conversion processing, resizing processing, and distortion correction processing, and inputs the processed data to the optical device 15. Here, one or both of the aforementioned fourth correspondence relation information DC4 and shape information DP7 are appropriately used for the various processes. Note that the image processing circuit 14 may, if necessary, execute processes such as OSD (On Screen Display) processing that generates image information for menu display or operation guidance and synthesizes it with the video data IMG1.

[0046] The optical device 15 is a device that projects image light onto the projection surface SC. The optical device 15 includes a light source 15a, a display panel 15b, and an optical system 15c.

[0047] The light source 15a includes, for example, a light source such as a halogen lamp, a xenon lamp, an ultra-high pressure mercury lamp, an LED (Light Emitting Diode), or a laser light source, and emits red, green, and blue light respectively. The display panel 15b is a light modulator including three light modulation elements provided corresponding to red, green, and blue. Each light modulation element includes, for example, a transmissive liquid crystal panel, a reflective liquid crystal panel, or a DMD (Digital Mirror Device), and generates image light of each color by modulating the light of the corresponding color. The image light of each color generated by the display panel 15b is synthesized by a color synthesis optical system to become full-color image light. The optical system 15c is a projection optical system including a projection lens or the like that forms and projects the full-color image light from the display panel 15b onto the projection surface SC.

[0048] The operation device 16 is a device that receives operations from the user. For example, the operation device 16 includes an operation panel (not shown) and an infrared receiver unit. The operation panel is provided on the exterior housing of the projector 10-1 and outputs a signal based on an operation from the user. The infrared receiver unit receives an infrared signal from a remote controller (not shown), decodes the infrared signal, and outputs a signal based on the operation of the remote controller. Note that the operation device 16 is provided as necessary and may be omitted.

[0049] In the above projector 10-1, the processing device 12 functions as a projection control unit 12a, an imaging control unit 12b, and a correction unit 12c by executing the program PR1 stored in the storage device 11. Therefore, the processing device 12 includes the projection control unit 12a, the imaging control unit 12b, and the correction unit 12c.

[0050] The projection control unit 12a controls the operations of the image processing circuit 14 and the optical device 15. More specifically, the projection control unit 12a causes the optical device 15 to project the projection image G1 onto the projection surface SC by controlling the operation of the optical device 15.

[0051] The imaging control unit 12b controls the operations of the first camera 20-1 and the second camera 20-2 respectively. More specifically, the imaging control unit 12b obtains the first imaging data D1 by causing the first camera 20-1 to image the projection image G1 projected onto the projection surface SC, or obtains the second imaging data D2 by causing the second camera 20-2 to image the projection image G1 projected onto the projection surface SC, and stores the obtained first imaging data D1 and second imaging data D2 in the storage device 11.

[0052] The correction unit 12c performs correction to adjust the shape of the projection image G1 using the first imaging data D1, the second imaging data D2, the first internal parameter information PC1, and the second internal parameter information PC2.

[0053] More specifically, the correction unit 12c generates first correspondence relationship information DC1 based on the first imaging data D1 and the second imaging data D2, and stores the generated first correspondence relationship information DC1 in the storage device 11.

[0054] Also, the correction unit 12c generates first external parameter information PC4 based on the first internal parameter information PC1, the second internal parameter information PC2, and the first correspondence relationship information DC1, and stores the generated first external parameter information PC4 in the storage device 11. This generation is performed for each first period. As a result, the first external parameter information PC4 is updated for each first period.

[0055] Furthermore, the correction unit 12c generates third coordinate information DP3 based on the first internal parameter information PC1, the second internal parameter information PC2, the first correspondence relationship information DC1, and the first external parameter information PC4, and stores the generated third coordinate information DP3 in the storage device 11.

[0056] Also, the correction unit 12c generates second correspondence relationship information DC2 based on the first coordinate information DP1 and the third coordinate information DP3, and stores the generated second correspondence relationship information DC2 in the storage device 11.

[0057] Furthermore, the correction unit 12c generates third correspondence relationship information DC3 based on the first coordinate information DP1 and the fourth coordinate information DP4, and stores the generated third correspondence relationship information DC3 in the storage device 11.

[0058] Also, the correction unit 12c generates fourth correspondence relationship information DC4 based on the second correspondence relationship information DC2 and the third correspondence relationship information DC3, and stores the generated fourth correspondence relationship information DC4 in the storage device 11.

[0059] Furthermore, the correction unit 12c generates second external parameter information PC5 and third internal parameter information PC3 based on the fourth correspondence relationship information DC4, and stores the generated second external parameter information PC5 and third internal parameter information PC3 in the storage device 11.

[0060] Further, the correction unit 12c generates shape information DP7 based on the second external parameter information PC5, the third internal parameter information PC3, and the third correspondence relationship information DC3, and stores the generated shape information DP7 in the storage device 11.

[0061] Furthermore, when the correction unit 12c adjusts the optical system 15c of the projector 10-1, it updates one or both of the second external parameter information PC5 and the third internal parameter information PC3 based on the results of photographing the projection image G1 projected after adjusting the optical system 15c with the first camera 20-1 and the second camera 20-2.

[0062] Also, the correction unit 12c generates fifth coordinate information DP5 based on a plurality of reference points PR (to be described later) in the projection image G1, and stores the generated fifth coordinate information DP5 in the storage device 11.

[0063] Furthermore, the correction unit 12c generates fifth transformed coordinate information DP6 by transforming the fifth coordinate information DP5 using the fourth correspondence relationship information DC4, and stores the generated fifth transformed coordinate information DP6 in the storage device 11.

[0064] Also, the correction unit 12c generates first marker coordinate information DP8 based on the fifth coordinate information DP5 and the second correspondence relationship information DC2, and stores the generated first marker coordinate information DP8 in the storage device 11.

[0065] Furthermore, the correction unit 12c generates second marker coordinate information DP9 based on the fifth transformed coordinate information DP6 and the second correspondence relationship information DC2, and stores the generated second marker coordinate information DP9 in the storage device 11.

[0066] Further, the correction unit 12c adjusts the projected image G1 based on the first marker coordinate information DP8, the second marker coordinate information DP9, the fifth transformation coordinate information DP6, the second internal parameter information PC2, and the second external parameter information PC5. Here, the correction unit 12c generates the fifth coordinate information DP5 based on the result of receiving from the user an operation of moving a plurality of reference points PR, which will be described later, in order to adjust the shape of the projected image G1.

[0067] 1-3. Control Method FIG. 3 is a flowchart showing the flow of the projection method according to the first embodiment. The projection method is performed by the processing device 12 executing the program PR1 using the aforementioned system 100. As shown in FIG. 3, the projection method includes steps S100 to S500.

[0068] Specifically, first, in step S100, the shape of the projection surface SC is measured. Through this measurement, after acquiring the first imaging data D1 and the second imaging data D2, the third internal parameter information PC3, the first external parameter information PC4, the second external parameter information PC5, the first coordinate information DP1, the second coordinate information DP2, the third coordinate information DP3, the fourth coordinate information DP4, the shape information DP7, the first correspondence information DC1, the second correspondence information DC2, the third correspondence information DC3, and the fourth correspondence information DC4 are acquired using the first internal parameter information PC1 and the second internal parameter information PC2. Details of step S100 will be described later with reference to FIGS. 4 to 7.

[0069] After step S100, in step S200, the projected image G1 is adjusted. Through this adjustment, the fifth coordinate information DP5, the fifth transformation coordinate information DP6, and the first marker coordinate information DP8 are acquired. Details of this acquisition will be described later with reference to FIGS. 8 to 10.

[0070] And in step S300, it is determined whether readjustment of the projected image G1 is necessary. For example, when the amount of change in the positional relationship of the projector 10-1 with respect to the projection surface SC is equal to or greater than a predetermined value, it is determined that readjustment of the projected image G1 is necessary. Note that it may be determined that readjustment of the projected image G1 is necessary based on an instruction from the user, or it may be determined that readjustment of the projected image G1 is necessary based on the output of a sensor or the like that detects the amount of change in the positional relationship of the projector 10-1 with respect to the projection surface SC.

[0071] Step S300 is repeated until it is determined that readjustment of the projected image G1 is necessary (step S300: NO). When it is determined that readjustment of the projected image G1 is necessary (step S300: YES), in step S400, by performing readjustment of the projected image G1, the projection state of the projected image G1 with respect to the projection surface SC is restored from the state at the second timing to the state at the first timing. Details of this restoration will be described later with reference to FIGS. 11 to 14.

[0072] After step S400, in step S500, it is determined whether there is an instruction to end. This determination is made, for example, based on whether there is an instruction to end based on an operation by the user.

[0073] When there is no instruction to end (step S500: NO), the aforementioned step S300 is executed. As a result, until there is an instruction to end, the aforementioned steps S300 and S400 are repeated. On the other hand, when there is an instruction to end (step S500: YES), the process ends.

[0074] FIG. 4 is a flowchart showing the flow of measurement of the shape of the projection surface SC. Step S100 shown in FIG. 3 includes steps S101 to S112 as shown in FIG. 4.

[0075] Specifically, in step S101, a projection image G1 is projected from the projector 10-1 onto the projection surface SC. This projection is performed by the projection control unit 12a controlling the operation of the projector 10-1.

[0076] In step S102, the first camera 20-1 captures the projection image G1 to obtain a first captured image GG1, which will be described later. This acquisition is performed by the imaging control unit 12b controlling the operation of the first camera 20-1. The acquired first captured image GG1 is stored in the storage device 11 as first imaging data D1.

[0077] In step S103, the second camera 20-2 captures the projection image G1 to obtain a second captured image GG2, which will be described later. This acquisition is performed by the imaging control unit 12b controlling the operation of the second camera 20-2. The acquired second captured image GG2 is stored in the storage device 11 as second imaging data D2. Note that step S103 may be executed before step S102 or during a period overlapping with step S102.

[0078] In step S104, a first correspondence relationship is generated. This generation is performed by the correction unit 12c based on the first coordinate information DP1 and the second coordinate information DP2. The obtained first correspondence relationship is stored in the storage device 11 as first correspondence relationship information DC1.

[0079] In step S105, a first external parameter is calculated. This calculation is performed by the correction unit 12c based on the first internal parameter indicated by the first internal parameter information PC1, the second internal parameter indicated by the second internal parameter information PC2, and the first correspondence relationship indicated by the first correspondence relationship information DC1. The calculated first external parameter is stored in the storage device 11 as first external parameter information PC4.

[0080] Step S106 obtains a plurality of third coordinates. The plurality of third coordinates are obtained by the correction unit 12c based on the first internal parameter indicated by the first internal parameter information PC1, the second internal parameter indicated by the second internal parameter information PC2, the first correspondence relationship indicated by the first correspondence relationship information DC1, and the first external parameter indicated by the first external parameter information PC4. The obtained plurality of third coordinates are stored in the storage device 11 as the third coordinate information DP3.

[0081] Step S107 generates a second correspondence relationship. This generation is performed by the correction unit 12c based on the first coordinate information DP1 and the second coordinate information DP2. The generated second correspondence relationship is stored in the storage device 11 as the second correspondence relationship information DC2.

[0082] Step S108 generates a third correspondence relationship. This generation is performed by the correction unit 12c based on the fourth coordinate information DP4 and the first coordinate information DP1. The generated third correspondence relationship is stored in the storage device 11 as the third correspondence relationship information DC3. Note that step S108 may be executed before step S107 as long as it is after the generation of the fourth coordinate information DP4 and the first coordinate information DP1.

[0083] Step S109 generates a fourth correspondence relationship. This generation is performed by the correction unit 12c based on the second correspondence relationship indicated by the second correspondence relationship information DC2 and the third correspondence relationship indicated by the third correspondence relationship information DC3.

[0084] Step S110 calculates a second external parameter. This calculation is performed by the correction unit 12c based on the fourth correspondence relationship indicated by the fourth correspondence relationship information DC4. The calculated second external parameter is stored in the storage device 11 as the second external parameter information PC5.

[0085] Step S111 calculates the third internal parameter. This calculation is performed by the correction unit 12c based on the fourth correspondence relationship indicated by the fourth correspondence information DC4. The calculated third internal parameter is stored in the storage device 11 as the third internal parameter information PC3. Note that step S111 may be executed before step S110 or may be executed during a period overlapping with step S110.

[0086] Step S112 obtains the shape information DP7. The shape information DP7 is obtained by the correction unit 12c based on the second external parameter indicated by the second external parameter information PC5, the third internal parameter indicated by the third internal parameter information PC3, and the third correspondence relationship indicated by the third correspondence information DC3. The obtained shape information DP7 is stored in the storage device 11.

[0087] The above general steps S101 to S112 will be described in detail below with reference to FIGS. 5 to 7.

[0088] FIG. 5 is a diagram for explaining the relationship between the coordinates on the display panel 15b of the projected image G1 and the coordinates in the first captured image GG1 and the second captured image GG2. In FIG. 5, the third pixel P0 which is a pixel of the display panel 15b that displays the projected image G1, the first pixel P1 which is a pixel of the first captured image GG1 indicated by the first captured data D1, and the second pixel P2 which is a pixel of the second captured image GG2 indicated by the second captured data D2 are shown in correspondence.

[0089] In step S101, the projected image G1 is projected onto the projection surface SC. At this time, as shown in the upper part of FIG. 5, the projected image G1 is displayed on the display panel 15b. In the example shown in FIG. 5, the projected image G1 is an image in which a plurality of alphabets are arranged. Note that the projected image G1 is an image for detecting the correspondence between the first pixel P1 and the second pixel P2 with respect to the third pixel P0, and is not limited to the example shown in FIG. 5, and may be a phase shift pattern, a gray code pattern, a pattern image, or the like.

[0090] In step S102, the first camera 20-1 captures the projected image G1 projected onto the projection surface SC, and as shown on the lower left side in FIG. 5, the first captured image GG1 indicated by the first captured data D1 is obtained.

[0091] In step S103, the second camera 20-2 captures the projected image G1 projected onto the projection surface SC, and as shown on the lower right side in FIG. 5, the second captured image GG2 indicated by the second captured data D2 is obtained.

[0092] In step S104, first, first coordinate information DP1 is generated based on a plurality of first coordinates that are the coordinates of a plurality of first pixels P1 in the coordinate system of the first captured image GG1, and second coordinate information DP2 is generated based on a plurality of second coordinates that are the coordinates of a plurality of second pixels P2 in the coordinate system of the second captured image GG2. Then, in step S104, based on the first coordinate information DP1 and the second coordinate information DP2, first correspondence relationship information DC1 indicating a first correspondence relationship in which a plurality of first coordinates and a plurality of second coordinates are associated with each other is generated.

[0093] Here, the plurality of first pixels P1 and the plurality of second pixels P2 correspond to each other. Therefore, the corresponding first pixel P1 and second pixel P2 correspond to a common third pixel P0. The correspondence between such a plurality of first pixels P1 and a plurality of second pixels P2 is detected by a known measurement method. Also, in step S104, along with the detection of the correspondence between the plurality of first pixels P1 and the plurality of second pixels P2, fourth coordinate information DP4 indicating fourth coordinates that are the coordinates of a plurality of third pixels P0 in the coordinate system of the display panel 15b is obtained. In FIG. 5, for convenience of explanation, one first pixel P1, one second pixel P2, and one third pixel P0 are each illustrated.

[0094] In step S105, with the coordinate system of the first camera 20-1 as the reference coordinate, based on the first internal parameter information PC1, the second internal parameter information PC2, and the first correspondence information DC1, one or both of the position and orientation of the second camera 20-2 with respect to the first camera 20-1 are estimated. Thereby, the first external parameter is obtained. For this estimation, known methods such as the five-point algorithm in OpenCV are used. Here, the position is represented by, for example, a three-dimensional vector (tx, ty, tz). Also, the orientation is represented by, for example, a three-dimensional vector (rx, ry, rz) or a 3×3 rotation matrix.

[0095] Here, in the first camera 20-1 or the second camera 20-2, when the coordinates on the image sensor are (u, v) and the normalized coordinates obtained by normalizing these coordinates with the focal length f of the camera are (x, y), these coordinates have the following relationship.

Equation

[0096] In the above equations, (c u , c v ) are the optical center coordinates of the camera. The internal parameter of the camera is represented by the internal parameter matrix A which is the product of the optical center coordinates (c u , c v ) and the focal length f. Note that the internal parameter of the camera may be the product of the internal parameter matrix A and other parameters such as the lens distortion coefficient.

[0097] FIG. 6 is a diagram for explaining the first partial image R1-1 and the second partial image R1-2 of the projected image G1. In FIG. 6, in the first captured image GG1, the part of the contour of the second captured image GG2 that is included in the first captured image GG1 is shown by a dashed white line, and the part of the contour of the first captured image GG1 that is included in the second captured image GG2 is shown by a thick dashed line.

[0098] As shown in FIG. 6, the projected image G1 shown in the first captured image GG1 is divided into a first partial image R1-1 and a second partial image R1-2 by a white dashed line indicating a part of the contour of the second captured image GG2. The first partial image R1-1 is an area belonging to the aforementioned area RC. That is, the first partial image R1-1 is an area included in both the first captured image GG1 and the second captured image GG2 in the projected image G1 projected onto the projection plane SC. On the other hand, the second partial image R1-2 is an area included in the first captured image GG1 but not included in the second captured image GG2 in the projected image G1 projected onto the projection plane SC.

[0099] Thus, the first partial image R1-1 is shown in both the first camera 20-1 and the second camera 20-2. Also, after step S105, as described above, the first internal parameter information PC1, the second internal parameter information PC2, and the second external parameter information PC5 have been acquired. Therefore, a plurality of three-dimensional coordinates of the first partial image R1-1 on the projection plane SC can be calculated by triangulation.

[0100] In step S106, a plurality of third coordinates calculated by such triangulation are acquired as the third coordinate information DP3.

[0101] After acquiring the third coordinate information DP3 in step S106 in this way, in step S107, second correspondence relationship information DC2 indicating a second correspondence relationship in which a plurality of first coordinates and a plurality of third coordinates are associated with each other is generated based on the first coordinate information DP1 and the third coordinate information DP3.

[0102] In step S108, third correspondence relationship information DC3 indicating a third correspondence relationship in which a plurality of fourth coordinates and a plurality of first coordinates are associated with each other is generated based on the fourth coordinate information DP4 and the first coordinate information DP1.

[0103] After obtaining the third correspondence information DC3 in step S108 as described above, in step S109, fourth correspondence information DC4 indicating a fourth correspondence in which a plurality of third coordinates and a plurality of fourth coordinates are associated with each other is generated based on the second correspondence information DC2 and the third correspondence information DC3.

[0104] After obtaining the fourth correspondence information DC4 in step S109 as described above, in step S110, second external parameter information PC5 indicating one or both of the position and orientation of the projector 10-1 with respect to the first camera 20-1 is calculated based on the fourth correspondence information DC4. For this calculation, for example, the method described in the document (Zhang, Zhengyou. "A flexible new technique for camera calibration." IEEE Transactions on pattern analysis and machine intelligence 22.11 (2000): 1330-1334.) is used.

[0105] Also, after obtaining the fourth correspondence information DC4 in step S109, in step S111, third internal parameter information PC3 indicating a third internal parameter that is an internal parameter of the projector 10-1 is calculated based on the fourth correspondence information DC4. For this calculation, similar to step S110, for example, the method described in the document (Zhang, Zhengyou. "A flexible new technique for camera calibration." IEEE Transactions on pattern analysis and machine intelligence 22.11 (2000): 1330-1334.) is used.

[0106] FIG. 7 is a diagram for explaining the shape measurement of the second partial image R1-2 of the projected image G1. In FIG. 7, in the first captured image GG1, the contour of the region R1-3 where the imaging region of the first camera 20-1 and the display region of the display panel 15b overlap is indicated by a dashed line.

[0107] The regions R1-3 are included in the first captured image GG1 and include the first partial image R1-1 and the second partial image R1-2. Also, after steps S110 and S111, as described above, the third internal parameter information PC3, the second external parameter information PC5, and the third correspondence information DC3 have been acquired. Therefore, similar to the triangulation between the projection plane SC, the first camera 20-1, and the second camera 20-2 in step S106 described above, by the triangulation between the projection plane SC, the first camera 20-1, and the projector 10-1, a plurality of three-dimensional coordinates on the projection plane SC can be calculated.

[0108] In step S112, based on the result calculated by such triangulation, shape information DP7 indicating the three-dimensional shape on the projection plane SC of the second partial image R1-2 that is included in the first captured image GG1 but not included in the second captured image GG2 in the projection image G1 is acquired.

[0109] FIG. 8 is a flowchart showing the flow of adjustment of the projection image G1. Step S200 shown in FIG. 3 includes steps S201 to S204 as shown in FIG. 8.

[0110] Specifically, in step S200, first, in step S201, it is determined whether there is an operation on a plurality of reference points PR (to be described later) in the projection image G1. This determination is made by the correction unit 12c based on whether it has received an operation by the user to move the plurality of reference points PR. When the operation is received, it is determined that there is an operation on the plurality of reference points PR in the projection image G1. Thus, step S201 receives an operation from the user to move the plurality of reference points PR to adjust the shape of the projection image G1.

[0111] Step S201 is repeated until there is an operation on a plurality of reference points PR in the projected image G1 (Step S201: NO). When there is an operation on a plurality of reference points PR in the projected image G1 (Step S201: YES), in Step S202, a plurality of fifth coordinates, which are the coordinates of the plurality of reference points PR on the display panel 15b, are specified. The plurality of fifth coordinates are stored in the storage device 11 as fifth coordinate information DP5. Thus, Step S202 specifies a plurality of fifth coordinates after an operation of moving a plurality of reference points PR to adjust the shape of the projected image G1.

[0112] After Step S202, in Step S203, the plurality of fifth coordinates indicated by the fifth coordinate information DP5 are converted into three-dimensional coordinates. This conversion is performed by the correction unit 12c using the fourth correspondence relationship indicated by the fourth correspondence information DC4. By this conversion, fifth converted coordinate information DP6 is generated, and the generated fifth converted coordinate information DP6 is stored in the storage device 11.

[0113] After Step S203, in Step S204, the three-dimensional coordinates of a marker MK, which will be described later, set on the projection plane SC are acquired. This acquisition is performed by converting the coordinates of the marker MK in the first captured image GG1 into three-dimensional coordinates using the second correspondence relationship indicated by the second correspondence information DC2. The acquired three-dimensional coordinates are stored in the storage device 11 as first marker coordinate information DP8.

[0114] The above general steps S201 to S204 will be described in detail below with reference to FIGS. 11 and 12.

[0115] FIG. 9 is a diagram for explaining a plurality of reference points PR on the display panel 15b of the projected image G1. In FIG. 9, a mode in which a plurality of reference points PR are arranged in a grid pattern on the display panel 15b is illustrated. In the example shown in FIG. 9, the plurality of reference points PR include a plurality of reference points PR arranged along a portion corresponding to the outer edge of the projection plane SC on the display panel 15b. Note that the arrangement and number of the reference points PR on the display panel 15b are not limited to the example shown in FIG. 9 and are arbitrary.

[0116] Each reference point PR can be moved by an operation by the user as necessary. When two or more reference points PR selected by the user among the plurality of reference points PR are moved, along with the change in the interval between the plurality of reference points PR, the portion of the projected image G1 corresponding to the interval is deformed. Thereby, the shape of the projected image G1 can be adjusted to match the shape of the projection surface SC.

[0117] In step S201, the presence or absence of such an operation is determined. After such step S201, in step S202, fifth coordinate information DP5 indicating a plurality of fifth coordinates, which are the coordinates of the plurality of reference points PR on the display panel 15b, is acquired.

[0118] After such step S202, in step S203, fifth transformed coordinate information DP6 is generated by transforming the fifth coordinate information DP5 using the fourth correspondence relationship indicated by the fourth correspondence relationship information DC4.

[0119] FIG. 10 is a diagram for explaining the three-dimensional coordinates of the marker MK at the first timing. In FIG. 10, in the three-dimensional coordinate system set in the real space where the projection surface SC is installed or the virtual three-dimensional space corresponding to the real space, the plurality of reference points PR and the plurality of markers MK detected based on the first captured image GG1 and the second captured image GG2 are shown.

[0120] The plurality of markers MK are marks set at arbitrary positions on the projection surface SC. Note that the shape, position, and number of the markers MK on the projection surface SC are not limited to the example shown in FIG. 10 and are arbitrary. However, when the shape of the marker MK is not a shape capable of discriminating the posture of the projection surface SC, the marker MK is set at three or more arbitrary positions on the projection surface SC.

[0121] In step S204, after the coordinates of the marker MK in the first captured image GG1 are detected using a known technique such as the phase shift method or image recognition technology, the coordinates are converted using the second correspondence information DC2, thereby generating first marker coordinate information DP8 indicating the three-dimensional coordinates of the marker MK at the first timing.

[0122] FIG. 11 is a flowchart showing the restoration process. Step S400 shown in FIG. 3 includes steps S401 to S406 as shown in FIG. 11.

[0123] Specifically, first, in step S401, it is determined whether or not the first period has elapsed. This determination is made by the correction unit 12c based on whether or not a predetermined timing after the previous generation timing of the first external parameter information PC4 has been reached. If the predetermined timing after the previous generation timing of the first external parameter information PC4 has been reached, it is determined that the first period has elapsed. The predetermined timing is, for example, a timing arbitrarily set by the user, a timing when the output of a sensor such as a temperature sensor or a vibration sensor reaches a predetermined state, or a timing when a predetermined period has elapsed after the previous generation timing of the first external parameter information PC4.

[0124] If it is determined that the first period has elapsed (step S401: YES), in step S402, the first external parameter is updated. As a result, the first external parameter is updated every first period. This update is performed by executing steps S101, S102, S103, S104, and S105 again. The updated first external parameter is stored in the storage device 11 as the first external parameter information PC4.

[0125] After step S402, or when it is determined that the first period has not elapsed (step S401: NO), in step S403, it is determined whether the optical system 15c has been adjusted. This determination is made by the correction unit 12c based on whether adjustments such as lens shift of the optical system 15c have been made. When adjustments such as lens shift of the optical system 15c have been made, it is determined that the optical system 15c has been adjusted.

[0126] When the optical system 15c has been adjusted (step S403: YES), in step S404, the parameters related to the optical system 15c are updated. The parameters are one or both of the second external parameter and the third internal parameter. This update is performed by executing one or both of the aforementioned steps S111 and S112 again. Thereby, when the optical system 15c of the projector 10-1 is adjusted, based on the results of photographing the projection image G1 projected after adjusting the optical system 15c with the first camera 20-1 and the second camera 20-2, one or both of the second external parameter indicated by the second external parameter information PC5 and the third internal parameter indicated by the third internal parameter information PC3 are updated. The updated second external parameter is stored in the storage device 11 as the second external parameter information PC5. Also, the updated third internal parameter is stored in the storage device 11 as the third internal parameter information PC3.

[0127] After step S404, or when the optical system 15c has not been adjusted (step S403: NO), in step S405, the three-dimensional coordinates of the marker MK at the second timing after the first timing are acquired. This acquisition is performed by the correction unit 12c by imaging the projection surface SC again with the first cameras 20-1 and 20-2 and then performing the same processing as in step S204 based on the captured image. The acquired three-dimensional coordinates are stored in the storage device 11 as the second marker coordinate information DP9.

[0128] After step S405, in step S406, the projected image G1 is adjusted. This adjustment is performed by the correction unit 12c based on the three-dimensional coordinates of the marker MK at the first timing, the three-dimensional coordinates of the marker MK at the second timing, the coordinates obtained by converting a plurality of fifth coordinates indicated by the fifth coordinate information DP5 into three-dimensional coordinates, the second internal parameter indicated by the second internal parameter information PC2, and the second external parameter indicated by the second external parameter information PC5.

[0129] Among the above steps S401 to S406, step S406 will be described in detail below with reference to FIGS. 12 to 14.

[0130] FIG. 12 is a diagram for explaining the three-dimensional coordinates of the marker MK at the second timing. In FIG. 12, a plurality of markers MK detected based on the imaging result in step S405 are shown in a three-dimensional coordinate system set in the real space where the projection plane SC is installed or a virtual three-dimensional space corresponding to the real space.

[0131] FIG. 13 is a diagram for explaining the three-dimensional coordinates of the marker MK at the first timing and the second timing. In FIG. 13, in a three-dimensional coordinate system set in the real space where the projection plane SC is installed or a virtual three-dimensional space corresponding to the real space, a plurality of markers MK detected based on the first captured image GG1 and the second captured image GG2 are shown as marker MK-1, and a plurality of markers MK detected based on the imaging result in step S405 are shown as marker MK-2.

[0132] In step S406, first, based on the coordinates of such a plurality of markers MK-1 and the coordinates of the plurality of markers MK-2, a function F representing the change in the position and orientation of the projection plane SC from the first timing to the second timing is obtained. The function F is a function representing a three-dimensional affine transformation composed of three-dimensional translation, three-dimensional rotation, and scale transformation.

[0133] FIG. 14 is a diagram for explaining the adjustment of the projected image G1. In FIG. 14, in a three-dimensional coordinate system set in the real space where the projection surface SC is installed or a virtual three-dimensional space corresponding to the real space, a plurality of reference points PR at the first timing are shown as a plurality of reference points PR-1, and a plurality of reference points PR at the second timing are shown as a plurality of reference points PR-2.

[0134] In step S406, as described above, after obtaining the function F representing the change in the position and orientation of the projection surface SC from the first timing to the second timing, by using this function F, the conversion from the plurality of reference points PR-1 to the plurality of reference points PR-2 is performed. Thereby, the adjustment of the projected image G1 is performed.

[0135] As described above, the above projection method includes step S101, step S102, step S103, step S104, step S105, step S106, step S107, step S108, step S109, and step S109. Thereby, a fourth correspondence relationship that can be used for various adjustments can be obtained.

[0136] As described above, the projection method of the present embodiment includes step S110 and step S111. By executing step S110, even if one or both of the position and orientation of the projector 10-1 with respect to the first camera 20-1 change, various adjustments can be performed with high accuracy based on the second external parameter indicated by the second external parameter information PC5. Further, by executing step S111, even if the optical characteristics of the projector 10-1 change, various adjustments can be performed with high accuracy based on the third internal parameter indicated by the third internal parameter information PC3. Note that one of step S110 and step S111 may be omitted.

[0137] Further, as described above, the projection method of the present embodiment includes step S112. Thereby, in addition to the three-dimensional information of the projection surface SC of the first partial image R1-1, the three-dimensional information of the projection surface SC of the second partial image R1-2 can be obtained.

[0138] Furthermore, as described above, the projection method of the present embodiment includes step S402. Thereby, it is possible to cope with the positional fluctuations of the first camera 20-1 and the second camera 20-2.

[0139] Also, as described above, the projection method of the present embodiment includes step S404. Thereby, it is possible to cope with the case where the optical settings of the projector 10-1 are changed.

[0140] Furthermore, as described above, the projection method of the present embodiment includes steps S202, S203, S204, S405, and S406. Thereby, even if the projection image G1 at the second timing is displaced with respect to the projection image G1 at the first timing on the projection surface SC, it is possible to return to the state of the projection image G1 at the first timing.

[0141] The projection method of the present embodiment includes step S201 as described above. And step S202 specifies a plurality of fifth coordinates after an operation of moving a plurality of reference points PR to adjust the shape of the projection image G1. Thereby, it is possible to maintain or restore the projection image G1 whose shape has been adjusted by the user.

[0142] 2. Second Embodiment Hereinafter, a second embodiment of the present disclosure will be described. For elements whose actions and functions are the same as those in the first embodiment in the forms exemplified below, the reference numerals used in the description of the first embodiment are reused and the detailed description of each is appropriately omitted.

[0143] FIG. 15 is a diagram showing an outline of a system 100A used for the projection method according to the second embodiment. The system 100A is a multi-projection system that projects an image group GG onto a projection surface SC.

[0144] The system 100A is configured in the same manner as the system 100 of the first embodiment except that a projector 10-2 is added. However, the projector 10-1 is the main device and controls the operation of the projector 10-2 which is the sub-device.

[0145] The projector 10-2 is a display device that projects the projection image G2 shown in the video data IMG2 output from the terminal device 30 onto the projection surface SC. The projector 10-2 is configured in the same manner as the projector 10-1, except that it is a sub-device. Note that the projector 10-2 may be configured to be controlled in operation by the projector 10-1, and may have a configuration different from that of the projector 10-1.

[0146] The projection image G1 and the projection image G2 are arranged side by side in the left-right direction in FIG. 15, and are projected onto the projection surface SC as an image group GG in a state where they are joined together. In the example shown in FIG. 15, the projection image G1 is projected onto the left region in FIG. 1 of the projection surface SC, while the projection image G2 is projected onto the right region in FIG. 15 of the projection surface SC. Then, the right end of the projection image G1 in FIG. 15 and the left end of the projection image G2 in FIG. 1 overlap each other in the overlapping region RG and are joined together.

[0147] In this way, a part of the projection image G1 and the projection image G2 overlap each other in the overlapping region RG. The overlapping region RG is a region where a blending process is performed to make the joint between the projection image G1 and the projection image G2 less noticeable.

[0148] In the present embodiment, the second camera 20-2 images the region RC2 including the projection image G2.

[0149] In the present embodiment, the terminal device 30 divides the video data indicating one image into the video data IMG1 and the video data IMG2, and then supplies the video data IMG1 to the projector 10-1 and the video data IMG2 to the projector 10-2.

[0150] FIG. 16 is a block diagram of the projector 10-1 used in the system 100A according to the second embodiment. The projector 10-1 is configured in the same manner as the projector 10-1 of the first embodiment, except that the program PR2 is used instead of the program PR1 of the first embodiment. In FIG. 16, the configuration of the projector 10-1 is typically shown. The configuration of the projector 10-2 is the same as that of the projector 10-1, except that it is a sub machine. Therefore, for the configuration of the projector 10-2, in the following description of the components, the video data IMG1 may be replaced with the video data IMG2. In the following, for the components of the projector 10-1, the suffix “-1” is added to the reference numerals of the components of the projector 10-1, or the suffix “-2” is added to the reference numerals of the components of the projector 10-2, so as to distinguish the components of the projector 10-1 from the components of the projector 10-2 in some cases.

[0151] The program PR2 is a program for executing the projection method of the present embodiment and is stored in the storage device 11.

[0152] In the projector 10-1 of the present embodiment, the processing device 12 functions as a projection control unit 12a, an imaging control unit 12b, and a correction unit 12d by executing the program PR2.

[0153] The correction unit 12d is the same as the correction unit 12c of the first embodiment, except that it additionally performs correction for adjusting the shape of the projection image G2. That is, the correction unit 12d performs correction for adjusting the shapes of the projection images G1 and G2 using the first imaging data D1, the second imaging data D2, the first internal parameter information PC1, and the second internal parameter information PC2.

[0154] More specifically, the correction unit 12d generates the shape information DP10 by the same method as the generation of the shape information DP7, and stores the generated shape information DP10 in the storage device 11.

[0155] The shape information DP10 is information indicating the three-dimensional shape on the projection plane SC of a third partial image R2-2, which is not included in the first captured image GG1 indicated by the first captured data D1 but is included in the second captured image GG2 indicated by the second captured data D2, in the projected image G2.

[0156] In this embodiment, when acquiring the second captured data D2 by capturing the projected image G1 with the second camera 20-2, not only the projected image G1 but also the projected image G2 is projected onto the projection plane SC. For this reason, in the second captured image GG2 indicated by the second captured data D2, in addition to the projected image G1, the projected image G2 is also displayed. When acquiring the first captured data D1 by capturing the projected image G1 with the first camera 20-1, not only the projected image G1 but also the projected image G2 may be projected onto the projection plane SC. In this case, in the first captured image GG1 indicated by the first captured data D1, in addition to the projected image G1, the projected image G2 is also displayed.

[0157] Hereinafter, matters regarding the generation of the shape information DP10 and the adjustment of the projected image G2 will be described based on FIGS. 17 to 20.

[0158] FIG. 17 is a diagram for explaining the projection of the projected images G1 and G2 onto the projection plane SC. In step S101 of this embodiment, as shown in FIG. 17, the projected image G1 is projected from the projector 10-1 onto the projection plane SC, and at the same time, the projected image G2 is projected from the projector 10-2 onto the projection plane SC.

[0159] Similar to the projected image G1, the projected image G2 is an image for detecting the correspondence between the pixels of the first captured image GG1 and the pixels of the second captured image GG2 indicated by the second captured data D2 with respect to the pixels of the display panel 15b.

[0160] In the example shown in FIG. 17, the projected images G1 and G2 are images that can be displayed discriminably in the region RC. As a result, steps S102 and S103 can be executed in a state where the projected image G1 and the projected image G2 are simultaneously projected onto the projection surface SC. Note that the specific display content of the projected images G1 and G2 is not limited to the example shown in FIG. 17 and is arbitrary. Also, the projected images G1 and G2 do not have to be discriminable in the region RC. In this case, steps S102 and S103 are executed in each of a state where the projected image G1 is projected onto the projection surface SC without projecting the projected image G2 and a state where the projected image G2 is projected onto the projection surface SC without projecting the projected image G1.

[0161] FIG. 18 is a diagram for explaining the relationship between the coordinates in the display panels 15b-1 and 15b-2 of the projected images G1 and G2 and the coordinates in the first captured image GG1 and the second captured image GG2. In FIG. 18, a third pixel P0-1 which is a pixel of the display panel 15b-1 that displays the projected image G1, a fourth pixel P0-2 which is a pixel of the display panel 15b-2 that displays the projected image G2, first pixels P1-1 and P1-2 which are pixels of the first captured image GG1 indicated by the first captured data D1, and second pixels P2-1 and P2-2 which are images of the second captured image GG2 indicated by the second captured data D2 are shown in a corresponding relationship.

[0162] In step S101 of the present embodiment, the projected images G1 and G2 are projected onto the projection surface SC. At this time, as shown on the upper left side in FIG. 18, similar to the first embodiment, the projected image G1 is displayed on the display panel 15b-1. Also, as shown on the upper right side in FIG. 18, the projected image G2 is displayed on the display panel 15b-2.

[0163] In step S102 of the present embodiment, the first camera 20-1 captures the projected images G1 and G2 projected onto the projection surface SC, and thus, as shown on the lower left side in FIG. 18, the first captured image GG1 indicated by the first captured data D1 is obtained. Note that in step S102, the projected image G2 does not have to be projected onto the projection surface SC.

[0164] In step S103 of the present embodiment, the second camera 20-2 captures the projection images G1 and G2 projected onto the projection surface SC, so that, as shown on the lower right side of FIG. 18, the second captured image GG2 shown in the second captured data D2 is obtained.

[0165] In step S104 of the present embodiment, first, first coordinate information DP1 is generated based on a plurality of first coordinates that are the coordinates of a plurality of first pixels P1-1 and P1-2 in the coordinate system of the first captured image GG1, and second coordinate information DP2 is generated based on a plurality of second coordinates that are the coordinates of a plurality of second pixels P2-1 and P2-2 in the coordinate system of the second captured image GG2. Then, in step S104 of the present embodiment, based on the first coordinate information DP1 and the second coordinate information DP2, first correspondence relationship information DC1 indicating a first correspondence relationship in which a plurality of first coordinates and a plurality of second coordinates are associated with each other is generated.

[0166] Here, the plurality of first pixels P1-1 and the plurality of second pixels P2-1 correspond to each other in the same manner as the correspondence relationship between the plurality of first pixels P1 and the plurality of second pixels P2 in the first embodiment. Therefore, the mutually corresponding first pixel P1-1 and second pixel P2-1 correspond to a common third pixel P0-1. Also, the plurality of first pixels P1-2 and the plurality of second pixels P2-2 correspond to each other. Therefore, the mutually corresponding first pixel P1-2 and second pixel P2-2 correspond to a common fourth pixel P0-2. In FIG. 18, for convenience of explanation, one each of the first pixels P1-1, P1-2, the second pixels P2-1, P2-2, and the third pixels P0-1, P0-2 are shown.

[0167] FIG. 19 is a diagram for explaining the shape measurement of the third partial image R2-2 of the projection image G2. In FIG. 19, in the second captured image GG2, the portion of the contour of the second captured image GG2 that is included in the first captured image GG1 is shown by a white dashed line, and the portion of the contour of the first captured image GG1 that is included in the second captured image GG2 is shown by a thick dashed line. Also, in FIG. 19, in the second captured image GG2, the contour of the region R2-3 where the imaging region of the second camera 20-2 and the display region of the display panel 15b-2 overlap is shown by a one-dot chain line.

[0168] As shown in FIG. 19, the projected image G2 shown in the second captured image GG2 is divided into a partial image R2-1 and a third partial image R2-2 by a thick dashed line indicating a part of the contour of the first captured image GG1. The partial image R2-1 is an area belonging to the aforementioned area RC. That is, the partial image R2-1 is an area included in both the first captured image GG1 and the second captured image GG2 in the projected image G2 projected onto the projection plane SC. On the other hand, the third partial image R2-2 is an area included in the second captured image GG2 but not included in the first captured image GG1 in the projected image G2 projected onto the projection plane SC.

[0169] The area R2-3 is included in the second captured image GG2 and includes the partial image R2-1 and the third partial image R2-2. In step S112 of the present embodiment, similar to calculating the three-dimensional shape of the second partial image R1-2 of the projected image G1 on the projection plane SC, the three-dimensional shape of the third partial image R2-2 of the projected image G2 on the projection plane SC is calculated. Thereby, the shape information DP10 is acquired.

[0170] Also according to the above-described second embodiment, various adjustments of the projectors 10-1 and 10-2 can be performed.

[0171] 3. Modification Each of the embodiments exemplified above can be variously modified. Specific modification modes applicable to each of the above-described embodiments are exemplified below. Two or more modes arbitrarily selected from the following examples can be appropriately combined within a range that does not conflict with each other.

[0172] 3-1. Modification 1 In the above-described embodiment, the mode in which the processing device 12 of the projector 10-1 executes the programs PR1 and PR2 is exemplified, but the present invention is not limited to this mode. For example, a processing device of a computer communicably connected to the projector 10-1 and the first camera 20-1 may execute the programs PR1 and PR2.

[0173] 3-2. Modification 2 In the foregoing embodiments, the mode in which the fourth correspondence relationship is used for adjusting the projection image G1 is exemplified, but the present invention is not limited to this mode. For example, the fourth correspondence relationship may be used to display a pattern such as a grid pattern having uniformity on the projection surface SC. Alternatively, after reflecting the model of the three-dimensional shape of the projection surface SC as viewed from the first camera 20-1 in three-dimensional image editing software or the like, it may be used to draw a picture on the model and display on a PC monitor or the like how the picture looks as viewed from the projector 10-1, or to project the picture onto the projector 10-1.

[0174] 4. Supplementary Note Hereinafter, a summary of the present disclosure will be provided as a supplementary note.

[0175] (Supplementary Note 1) The projection method according to the first aspect of the present disclosure includes projecting a projection image from a projector onto a projection surface, acquiring a first captured image by imaging the projection image with a first camera having first internal parameters, acquiring a second captured image by imaging the projection image with a second camera having second internal parameters, generating a first correspondence relationship in which a plurality of first coordinates, which are coordinates of a plurality of first pixels in the first captured image, and a plurality of second coordinates, which are coordinates of a plurality of second pixels corresponding to the plurality of first pixels in the second captured image, are associated with each other, calculating a first external parameter indicating one or both of the position and orientation of the second camera with respect to the first camera based on the first internal parameters, the second internal parameters, and the first correspondence relationship, obtaining a plurality of third coordinates, which are three-dimensional coordinates of a first partial image included in both the first captured image and the second captured image on the projection surface in the projection image, based on the first internal parameters, the second internal parameters, the first correspondence relationship, and the first external parameter, generating a second correspondence relationship in which the plurality of first coordinates and the plurality of third coordinates are associated with each other, generating a third correspondence relationship in which a plurality of fourth coordinates, which are coordinates of a plurality of third pixels in the display panel of the projector, and the plurality of first coordinates are associated with each other, and generating a fourth correspondence relationship in which the plurality of third coordinates and the plurality of fourth coordinates are associated with each other based on the second correspondence relationship and the third correspondence relationship.

[0176] In the above aspect, a fourth correspondence relationship that can be used for various adjustments can be obtained.

[0177] (Appendix 2) In the second aspect, which is a preferred example of the first aspect, calculating a second external parameter indicating one or both of the position and orientation of the projector with respect to the first camera based on the fourth correspondence relationship, and a third internal parameter that is the internal parameter of the projector. In the above aspect, even if one or both of the position and orientation of the projector with respect to the first camera vary, various adjustments can be performed with high precision based on the second external parameter. Also, even if the optical characteristics of the projector vary, various adjustments can be performed with high precision based on the third internal parameter.

[0178] (Appendix 3) In the third aspect, which is a preferred example of the first or second aspect, obtaining shape information indicating the three-dimensional shape on the projection plane of a second partial image that is included in the first captured image but not included in the second captured image in the projected image, based on the second external parameter, the third internal parameter, and the third correspondence relationship. In the above aspect, in addition to the first partial image, three-dimensional information on the projection plane of the second partial image can be obtained.

[0179] (Appendix 4) In the fourth aspect, which is a preferred example of any one of the first to third aspects, updating the first external parameter for each first period. In the above aspect, it is possible to cope with position fluctuations of the first camera and the second camera.

[0180] (Appendix 5) In the fifth aspect, which is a preferred example of the second aspect, when adjusting the optical system of the projector, updating one or both of the second external parameter and the third internal parameter based on the results of photographing the projected image projected after adjusting the optical system with the first camera and the second camera. In the above aspect, it is possible to cope with the case where the optical settings of the projector are changed.

[0181] (Appendix 6) In the sixth aspect, which is a preferred example of the second aspect, specifying a plurality of fifth coordinates in the display panel of a plurality of reference points in the projected image; converting the plurality of fifth coordinates into three-dimensional coordinates based on the fourth correspondence relationship; obtaining the three-dimensional coordinates of the marker at the first timing based on the second correspondence relationship; obtaining the three-dimensional coordinates of the marker at the second timing after the first timing based on the second correspondence relationship; and adjusting the projected image based on the three-dimensional coordinates of the marker at the first timing, the three-dimensional coordinates of the marker at the second timing, the coordinates obtained by converting the plurality of fifth coordinates into three-dimensional coordinates, the second internal parameter, and the second external parameter. In the above aspect, even if the projected image at the second timing is displaced relative to the projected image at the first timing on the projection surface, it is possible to return to the state of the projected image at the first timing.

[0182] (Appendix 7) In the seventh aspect, which is a preferred example of the sixth aspect, receiving an operation from the user to move the plurality of reference points to adjust the shape of the projected image, and specifying the plurality of fifth coordinates after the operation. In the above aspect, it is possible to maintain or restore the projected image whose shape has been adjusted by the user.

[0183] (Supplementary Note 8) The system according to the eighth aspect, which is a preferred example of the present disclosure, includes an optical device of a projector, a first camera, a second camera, and a processing device that controls the operations of the optical device, the first camera, and the second camera. The processing device projects a projection image from the projector onto a projection surface, acquires a first captured image by imaging the projection image with the first camera having first internal parameters, acquires a second captured image by imaging the projection image with the second camera having second internal parameters, generates a first correspondence relationship in which a plurality of first coordinates, which are the coordinates of a plurality of first pixels in the first captured image, and a plurality of second coordinates, which are the coordinates of a plurality of second pixels corresponding to the plurality of first pixels in the second captured image, are associated with each other, calculates a first external parameter indicating one or both of the position and orientation of the second camera with respect to the first camera based on the first internal parameters, the second internal parameters, and the first correspondence relationship, obtains a plurality of third coordinates, which are the three-dimensional coordinates of a first partial image included in both the first captured image and the second captured image on the projection surface in the projection image, based on the first internal parameters, the second internal parameters, the first correspondence relationship, and the first external parameter, generates a second correspondence relationship in which the plurality of first coordinates and the plurality of third coordinates are associated with each other, generates a third correspondence relationship in which a plurality of fourth coordinates, which are the coordinates of a plurality of third pixels in the display panel of the projector, and the plurality of first coordinates are associated with each other, and generates a fourth correspondence relationship in which the plurality of third coordinates and the plurality of fourth coordinates are associated with each other based on the second correspondence relationship and the third correspondence relationship.

[0184] In the above aspect, a fourth correspondence relationship that can be used for various adjustments can be obtained.

[0185] (Supplementary Note 9) The program according to the ninth aspect, which is a preferred example of the present disclosure, includes projecting a projection image from a projector onto a projection surface, capturing the projection image with a first camera having first internal parameters to obtain a first captured image, capturing the projection image with a second camera having second internal parameters to obtain a second captured image, generating a first correspondence relationship in which a plurality of first coordinates, which are the coordinates of a plurality of first pixels in the first captured image, and a plurality of second coordinates, which are the coordinates of a plurality of second pixels corresponding to the plurality of first pixels in the second captured image, are associated with each other, calculating a first external parameter indicating one or both of the position and orientation of the second camera with respect to the first camera based on the first internal parameters, the second internal parameters, and the first correspondence relationship, obtaining a plurality of third coordinates, which are three-dimensional coordinates of a first partial image included in both the first captured image and the second captured image on the projection surface in the projection image, based on the first internal parameters, the second internal parameters, the first correspondence relationship, and the first external parameter, generating a second correspondence relationship in which the plurality of first coordinates and the plurality of third coordinates are associated with each other, generating a third correspondence relationship in which a plurality of fourth coordinates, which are the coordinates of a plurality of third pixels in the display panel of the projector, and the plurality of first coordinates are associated with each other, and generating a fourth correspondence relationship in which the plurality of third coordinates and the plurality of fourth coordinates are associated with each other based on the second correspondence relationship and the third correspondence relationship, and causing a computer to execute these operations.

[0186] In the above aspect, a fourth correspondence relationship that can be used for various adjustments can be obtained.

Description of Reference Numerals

[0187] 10-1... Projector, 10-2... Projector, 11... Memory device, 12... Processing device, 12a... Projection control unit, 12b... Imaging control unit, 12c... Correction unit, 12d... Correction unit, 13... Communication device, 14... Image processing circuit, 15... Optical device, 15a... Light source, 15b... Display panel, 15b-1... Display panel, 15b-2... Display panel, 15c... Optical system, 16... Operating device, 20-1... First camera, 20-2... Second camera, 30... Terminal device, 100... System, 100A... System, D1... First imaging data, D2... Second imaging data, DC1... First correspondence relationship information, DC2... Second correspondence relationship information, DC3... Third correspondence relationship information, DC4... Fourth correspondence relationship information, DP1... First coordinate information, DP2... Second coordinate information, DP3... Third coordinate information, DP4... Fourth coordinate information, DP5... Fifth coordinate information, DP6... Fifth conversion coordinate information, DP7... Shape information, DP8... First marker coordinate information, DP9... Second marker coordinate information, DP10... Shape information, F... Function, G1... Projected image, G2... Projected image, GG... Image group, GG1... First captured image, GG2... Second captured image, IMG1... Video data, IMG2... Video data, MK... Marker, MK-1... Marker, MK-2... Marker, P0... Third pixel, P0-1... Third pixel, P0-2... Fourth pixel, P1... First pixel, P1-1... First pixel, P1-2... First pixel, P2... Second pixel, P2-1... Second pixel, P2-2... Second pixel, PC1... First internal parameter information, PC2... Second internal parameter information, PC3... Third internal parameter information, PC4... First external parameter information, PC5... Second external parameter information, PR... Reference point, PR-1... Reference point, PR-2... Reference point, PR1... Program, PR2... Program, R1-1... First partial image, R1-2... Second partial image, R1-3... Region, R2-1... Partial image, R2-2... Third partial image, R2-3... Region, RC... Region, RC1... Region, RC2... Region, RG... Overlap region, S100... Step, S101... Step, S102... Step, S103... Step, S104... Step, S105... Step, S106... Step, S107... Step, S108... Step, S109... Step, S110... Step, S111... Step, S112... Step, S200... Step, S201... Step, S202... Step, S203... Step, S204... Step, S300... Step,S400... step, S401... step, S402... step, S403... step, S404... step, S405... step, S406... step, S500... step, SC... projection plane, f... focal length.

Claims

1. Projecting a projected image from a projector onto a projection surface; Acquiring a first captured image by capturing the projected image with a first camera having first internal parameters; Acquiring a second captured image by capturing the projected image with a second camera having second internal parameters; Generating a first correspondence relationship in which a plurality of first coordinates that are coordinates of a plurality of first pixels in the first captured image and a plurality of second coordinates that are coordinates of a plurality of second pixels corresponding to the plurality of first pixels in the second captured image are associated with each other; Calculating a first external parameter indicating one or both of the position and orientation of the second camera with respect to the first camera based on the first internal parameters, the second internal parameters, and the first correspondence relationship; Determining a plurality of third coordinates that are three-dimensional coordinates of a first partial image included in both the first captured image and the second captured image in the projection surface in the projected image based on the first internal parameters, the second internal parameters, the first correspondence relationship, and the first external parameter; Generating a second correspondence relationship in which the plurality of first coordinates and the plurality of third coordinates are associated with each other; Generating a third correspondence relationship in which a plurality of fourth coordinates that are coordinates of a plurality of third pixels in the display panel of the projector and the plurality of first coordinates are associated with each other; Generating a fourth correspondence relationship in which the plurality of third coordinates and the plurality of fourth coordinates are associated with each other based on the second correspondence relationship and the third correspondence relationship; A projection method comprising the above steps.

2. Calculating a second external parameter indicating one or both of the position and orientation of the projector with respect to the first camera and a third internal parameter that is an internal parameter of the projector based on the fourth correspondence relationship, the projection method according to claim 1. The projection method according to claim 1.

3. Obtaining shape information indicating a three-dimensional shape of a second partial image included in the first captured image but not included in the second captured image in the projection surface in the projected image based on the second external parameter, the third internal parameter, and the third correspondence relationship, the projection method according to claim 2. The projection method according to claim 2.

4. Updating the first external parameter for each first period, the projection method according to claim 1. The projection method according to claim 1.

5. When adjusting the optical system of the projector, updating one or both of the second external parameter and the third internal parameter based on the results of photographing the projection image projected after adjusting the optical system with the first camera and the second camera. The projection method according to claim 2.

6. Identifying a plurality of fifth coordinates that are the coordinates in the display panel of a plurality of reference points in the projection image. Converting the plurality of fifth coordinates into three-dimensional coordinates based on the fourth correspondence. Obtaining the three-dimensional coordinates of the marker at a first timing based on the second correspondence. Obtaining the three-dimensional coordinates of the marker at a second timing after the first timing based on the second correspondence. Adjusting the projection image based on the three-dimensional coordinates of the marker at the first timing, the three-dimensional coordinates of the marker at the second timing, the coordinates obtained by converting the plurality of fifth coordinates into three-dimensional coordinates, the second internal parameter, and the second external parameter. The projection method according to claim 2.

7. Receiving from the user an operation of moving the plurality of reference points to adjust the shape of the projection image, and identifying the plurality of fifth coordinates after the operation. Identifying the plurality of fifth coordinates after the operation. The projection method according to claim 6.

8. An optical device of a projector, A first camera, A second camera, And a processing device that controls the operations of the optical device, the first camera, and the second camera. The processing device, Projecting a projection image from the projector onto a projection surface, Obtaining a first captured image by imaging the projection image with the first camera having a first internal parameter, Obtaining a second captured image by imaging the projection image with the second camera having a second internal parameter, Generating a first correspondence in which a plurality of first coordinates that are the coordinates of a plurality of first pixels in the first captured image and a plurality of second coordinates that are the coordinates of a plurality of second pixels corresponding to the plurality of first pixels in the second captured image are associated with each other, Calculating a first external parameter indicating one or both of the position and orientation of the second camera with respect to the first camera based on the first internal parameter, the second internal parameter, and the first correspondence. Based on the first internal parameter, the second internal parameter, the first correspondence relationship, and the first external parameter, obtaining a plurality of third coordinates that are three-dimensional coordinates on the projection plane of a first partial image included in both the first captured image and the second captured image in the projected image; Generating a second correspondence relationship in which the plurality of first coordinates and the plurality of third coordinates are associated; Generating a third correspondence relationship in which a plurality of fourth coordinates that are coordinates of a plurality of third pixels in the display panel of the projector and the plurality of first coordinates are associated; Based on the second correspondence relationship and the third correspondence relationship, generating a fourth correspondence relationship in which the plurality of third coordinates and the plurality of fourth coordinates are associated; A system for executing. [

9. ] Projecting a projected image from a projector onto a projection plane; Obtaining a first captured image by capturing the projected image with a first camera having a first internal parameter; Obtaining a second captured image by capturing the projected image with a second camera having a second internal parameter; Generating a first correspondence relationship in which a plurality of first coordinates that are coordinates of a plurality of first pixels in the first captured image and a plurality of second coordinates that are coordinates of a plurality of second pixels corresponding to the plurality of first pixels in the second captured image are associated; Calculating a first external parameter indicating one or both of the position and orientation of the second camera with respect to the first camera based on the first internal parameter, the second internal parameter, and the first correspondence relationship; Based on the first internal parameter, the second internal parameter, the first correspondence relationship, and the first external parameter, obtaining a plurality of third coordinates that are three-dimensional coordinates on the projection plane of a first partial image included in both the first captured image and the second captured image in the projected image; Generating a second correspondence relationship in which the plurality of first coordinates and the plurality of third coordinates are associated; Generating a third correspondence relationship in which a plurality of fourth coordinates that are coordinates of a plurality of third pixels in the display panel of the projector and the plurality of first coordinates are associated; Based on the second correspondence relationship and the third correspondence relationship, generating a fourth correspondence relationship in which the plurality of third coordinates and the plurality of fourth coordinates are associated; A program that causes a computer to execute.

Images