Projection method, projection system and program product

Abstract

Provided are a projection method, a projection system, and a program product that enable various adjustments of a projector. The projection method generates a first correspondence relationship between a plurality of first coordinates in a first captured image of a first camera and a plurality of second coordinates in a second captured image of a second camera. A first external parameter indicating a position and / or orientation of the second camera with respect to the first camera is calculated from a first internal parameter of the first camera, a second internal parameter of the second camera, and a first correspondence, and a plurality of third coordinates, which are three-dimensional coordinates of the projection image on the projection surface, are obtained from the first and second internal parameters, the first correspondence, and the first external parameter. A second correspondence relationship between the plurality of first coordinates and the plurality of third coordinates and a third correspondence relationship between the plurality of first coordinates and a plurality of fourth coordinates, which are coordinates of a plurality of third pixels in the display panel of the projector, are generated, and a fourth correspondence relationship between the plurality of third coordinates and the plurality of fourth coordinates is generated on the basis of the second and third correspondence relationships.

Description

Technical Field

[0001] The present disclosure relates to a projection method, a projection system, and a program product. Background Art

[0002] For example, Patent Document 1 discloses the following system: Based on an image projected from a projector onto a projection target and a captured image obtained by a capturing device capturing the projection target, a correspondence relationship between positions on the image projected onto the projection target and positions on the captured image is determined.

[0003] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2022-69176

[0004] In order to make various adjustments to a projector, it is necessary to determine how a two-dimensional coordinate system representing positions on a display panel of the projector corresponds to three-dimensional coordinates of a projection image on a screen. However, this is not disclosed in Patent Document 1. Summary of the Invention

[0005] A projection method according to one aspect of the present disclosure includes: projecting a projection image from a projector onto a projection surface; capturing the projection image with a first camera having first internal parameters to obtain a first captured image; capturing the projection image with a second camera having second internal parameters to obtain a second captured image; generating a first correspondence relationship in which a plurality of first coordinates and a plurality of second coordinates are corresponding, the plurality of first coordinates being coordinates of a plurality of first pixels in the first captured image, and the plurality of second coordinates being coordinates of a plurality of second pixels corresponding to the plurality of first pixels in the second captured image; calculating first external parameters representing one or both of the position and orientation of the second camera relative to the first camera based on the first internal parameters, the second internal parameters, and the first correspondence relationship; obtaining a plurality of third coordinates based on the first internal parameters, the second internal parameters, the first correspondence relationship, and the first external parameters, the plurality of third coordinates being three-dimensional coordinates of a first partial image included in both the first captured image and the second captured image in the projection surface; generating a second correspondence relationship in which the plurality of first coordinates and the plurality of third coordinates are corresponding; generating a third correspondence relationship in which a plurality of fourth coordinates and the plurality of first coordinates are corresponding, the plurality of fourth coordinates being coordinates of a plurality of third pixels in a display panel of the projector; and generating a fourth correspondence relationship in which the plurality of third coordinates and the plurality of fourth coordinates are corresponding based on the second correspondence relationship and the third correspondence relationship.

[0006] A projection system according to an aspect of the present disclosure includes: an optical device of a projector; a first camera; a second camera; and a processing device that controls operations of the optical device, the first camera, and the second camera. The processing device performs the following processes: projecting a projection image from the projector onto a projection surface; capturing the projection image with the first camera having first internal parameters to obtain a first captured image; capturing the projection image with the second camera having second internal parameters to obtain a second captured image; generating a first correspondence in which a plurality of first coordinates and a plurality of second coordinates are associated with each other, the plurality of first coordinates being coordinates of a plurality of first pixels in the first captured image, and the plurality of second coordinates being coordinates of a plurality of second pixels in the second captured image that correspond to the plurality of first pixels; calculating first external parameters representing one or both of the position and orientation of the second camera relative to the first camera based on the first internal parameters, the second internal parameters, and the first correspondence; obtaining a plurality of third coordinates, the plurality of third coordinates being three-dimensional coordinates on the projection surface of a first partial image included in both the first captured image and the second captured image in the projection image, based on the first internal parameters, the second internal parameters, the first correspondence, and the first external parameters; generating a second correspondence in which the plurality of first coordinates and the plurality of third coordinates are associated with each other; generating a third correspondence in which a plurality of fourth coordinates and the plurality of first coordinates are associated with each other, the plurality of fourth coordinates being coordinates of a plurality of third pixels in a display panel of the projector; and generating a fourth correspondence in which the plurality of third coordinates and the plurality of fourth coordinates are associated with each other based on the second correspondence and the third correspondence.

[0007] A program product according to one aspect of the present disclosure causes a computer to perform the following processes: projecting a projected image from a projector onto a projection surface; capturing the projected image with a first camera having first internal parameters to obtain a first captured image; capturing the projected image with a second camera having second internal parameters to obtain a second captured image; generating a first correspondence in which a plurality of first coordinates and a plurality of second coordinates correspond to each other, the plurality of first coordinates being coordinates of a plurality of first pixels in the first captured image, and the plurality of second coordinates being coordinates of a plurality of second pixels in the second captured image that correspond to the plurality of first pixels; calculating first external parameters representing one or both of the position and orientation of the second camera relative to the first camera based on the first internal parameters, the second internal parameters, and the first correspondence; obtaining a plurality of third coordinates, the plurality of third coordinates being three-dimensional coordinates on the projection surface of a first partial image included in both the first captured image and the second captured image in the projected image, based on the first internal parameters, the second internal parameters, the first correspondence, and the first external parameters; generating a second correspondence in which the plurality of first coordinates and the plurality of third coordinates correspond to each other; generating a third correspondence in which a plurality of fourth coordinates and the plurality of first coordinates correspond to each other, the plurality of fourth coordinates being coordinates of a plurality of third pixels in a display panel of the projector; and generating a fourth correspondence in which the plurality of third coordinates and the plurality of fourth coordinates correspond to each other based on the second correspondence and the third correspondence. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 FIG. 1 is a diagram schematically showing a system used in the projection method according to the first embodiment.

[0009] Figure 2 FIG. 2 is a block diagram of a projector used in the system according to the first embodiment.

[0010] Figure 3 FIG. 3 is a flowchart showing the procedure of the projection method according to the first embodiment.

[0011] Figure 4 FIG. 4 is a flowchart showing the measurement procedure of the shape of the projection surface.

[0012] Figure 5 FIG. 5 is a diagram for explaining the relationship between the coordinates in the display panel of the projected image and the coordinates in the first captured image and the second captured image.

[0013] Figure 6 FIG. 6 is a diagram for explaining a first partial image and a second partial image of the projected image.

[0014] Figure 7 FIG. 7 is a diagram for explaining the shape measurement of the second partial image of the projected image.

[0015] Figure 8 is a flowchart showing an adjustment process of a projected image.

[0016] Figure 9 is a diagram for explaining a plurality of reference points of a projected image in a display panel.

[0017] Figure 10 is a diagram for explaining three-dimensional coordinates of a mark at a first timing.

[0018] Figure 11 is a flowchart showing a recovery process.

[0019] Figure 12 is a diagram for explaining three-dimensional coordinates of a mark at a second timing.

[0020] Figure 13 is a diagram for explaining three-dimensional coordinates of marks at a first timing and a second timing.

[0021] Figure 14 is a diagram for explaining adjustment of a projected image.

[0022] Figure 15 is a diagram showing an overview of a system used in a projection method according to a second embodiment.

[0023] Figure 16 is a block diagram of a projector used in the system according to the second embodiment.

[0024] Figure 17 is a diagram for explaining projection of a projected image onto a projection surface.

[0025] Figure 18 is a diagram for explaining a relationship between coordinates of a projected image in a display panel and coordinates in a first captured image and a second captured image.

[0026] Figure 19 is a diagram for explaining shape measurement of a third partial image of a projected image.

[0027] Reference numeral description

[0028] 10-1: Projector; 10-2: Projector; 11: Storage device; 12: Processing device; 12a: Projection control unit; 12b: Shooting control unit; 12c: Calibration unit; 12d: Calibration unit; 13: Communication device; 14: Image processing circuit; 15: Optical device; 15a: Light source; 15b: Display panel; 15b-1: Display panel; 15b-2: Display panel; 15c: Optical system; 16: Operating device; 20-1: First camera; 20-2: Second camera; 30: Terminal device; 100: System; 100A: System; D1: First shooting data; D2: Second shooting data; DC1: First correspondence information; DC2: Second correspondence information; DC3: Third correspondence information; DC4: Fourth correspondence information; DP1: First coordinate information; DP2: Second coordinate information; DP3: Third coordinate information; DP4: Fourth coordinate information; DP5: Fifth coordinate information; DP6: Fifth transformation coordinate information; DP7: Shape information; DP8: First marker coordinate information; DP9: Second marker coordinate information; DP10: Shape information; F: Function; G1: Projected image; G2: Projected image; GG: Image group; GG1: First captured image; GG2: Second captured image; IMG1: Video data; IMG2: Video data; MK: Marker; MK-1: Marker; MK-2: Marker; P0: Third pixel; P0-1: Third pixel; P0-2: Fourth pixel; P1: First pixel; P1-1: First pixel; P1-2: First pixel; P2: Second pixel; P2-1: Second pixel; P2-2: Second pixel; PC1: First internal parameter information; PC2: Second internal parameter information; PC3: Third internal parameter information; PC4: First external parameter information; PC5: Second external parameter information; PR: Reference point; PR-1: Reference point; PR-2: Reference point; PR1: Program; PR2: Program; R1-1: First partial image; R1-2: Second partial image; R1-3: Region; R2-1: Partial image; R2-2: Third partial image; R2-3: Region; RC: Region; RC1: Region; RC2: Region; RG: Overlapping region; S100: Step; S101: Step; S102: Step; S103: Step; S104: Step; S105: Step; S106: Step; S107: Step; S108: Step; S109: Step; S110: Step; S111: Step; S112: Step; S200: Step; S201: Step S202: Step; S203: Step; S204: Step; S300: Step; S400: Step; S401: Step; S402: Step; S403: Step; S404: Step; S405: Step; S406: Step; S500: Step; SC: Projection surface; f: Focal length. Detailed implementation mode

[0029] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the accompanying drawings. In addition, in the drawings, the sizes and scales of the respective parts are appropriately different from the actual ones, and there are also parts schematically shown for easy understanding. In addition, the scope of the present disclosure is not limited to these embodiments as long as the gist of the present disclosure is not particularly limited in the following description.

[0030] 1. First Embodiment

[0031] 1-1. Outline of the System

[0032] Figure 1 FIG. is a schematic diagram showing the system 100 used in the projection method of the first embodiment. The system 100 is a projection system that projects a projection image G1 onto a projection surface SC.

[0033] The projection surface SC is the surface of an object such as a screen. In Figure 1 the example shown, the projection surface SC is a concave surface curved in such a way that the center of the projection surface SC is located inside the left and right sides. In addition, the shape of the projection surface SC is not limited to Figure 1 the example shown. For example, the projection surface SC may be a convex surface curved in such a way that the center of the projection surface SC is located in front of the left and right sides.

[0034] As Figure 1 shown, the system 100 includes a projector 10-1, a first camera 20-1, a second camera 20-2, and a terminal device 30.

[0035] The projector 10-1 is a display device that projects the projection image G1 shown in the video data IMG1 output from the terminal device 30 onto the projection surface SC. In Figure 1 the example shown, the projection image G1 is projected onto the area shifted to the left side in Figure 1 . In addition, the projection position of the projection image G1 with respect to the projection surface SC is not limited to Figure 1 the example shown and is arbitrary.

[0036] The projector 10-1 of the present embodiment has a function of controlling the operations of the first camera 20-1 and the second camera 20-2 and a function of adjusting the shape of the projection image G1 using the shooting results of the first camera 20-1 and the second camera 20-2.

[0037] The first camera 20-1 and the second camera 20-2 are digital cameras each having a shooting element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

[0038] The first camera 20-1 photographs the area RC1. The area RC1 is an area that includes the projected image G1 projected onto the projection surface SC. On the other hand, the second camera 20-2 photographs the area RC2. The area RC2 is an area that includes only a part of the projected image G1 projected onto the projection surface SC. In this way, the area RC1 and the area RC2 have an overlapping area RC. In addition, the first camera 20-1 may also be a component of the projector 10-1. In addition, the second camera 20-2 may also be a component of a projector different from the projector 10-1.

[0039] The terminal device 30 is a computer having a function of supplying the image data IMG1 to the projector 10-1. In Figure 1 the example shown, the terminal device 30 is a notebook computer. In addition, the terminal device 30 is not limited to a notebook computer. For example, it may also be a desktop computer, a smart phone, a tablet terminal, etc., and may also be a video playback device, a DVD (Digital Versatile Disk) player, a Blu-ray Disc player, a hard disk recorder, a television tuner device, a set-top box of CATV (Cable television), a video game console, etc.

[0040] 1-2. Projector

[0041] Figure 2 is a block diagram of the projector 10-1 used in the system 100 of the first embodiment. In Figure 2 it, in addition to the projector 10-1, the connection states of the first camera 20-1 and the second camera 20-2 with respect to the projector 10-1 are also shown.

[0042] As Figure 2 shown, the projector 10-1 has a storage device 11, a processing device 12, a communication device 13, an image processing circuit 14, an optical device 15, and an operation device 16. They are connected in a manner that enables mutual communication.

[0043] The storage device 11 is a storage device that stores programs executed by the processing device 12 and data processed by the processing device 12. The storage device 11 is configured to include, for example, a hard disk drive or a semiconductor memory. In addition, part or all of the storage device 11 may also be provided in an external storage device or a server, etc., of the projector 10-1.

[0044] In the storage device 11, a program PR1, first captured data D1, second captured data D2, first internal parameter information PC1, second internal parameter information PC2, third internal parameter information PC3, first external parameter information PC4, second external parameter information PC5, first coordinate information DP1, second coordinate information DP2, third coordinate information DP3, fourth coordinate information DP4, fifth coordinate information DP5, fifth transformed coordinate information DP6, shape information DP7, first marker coordinate information DP8, second marker coordinate information DP9, first correspondence information DC1, second correspondence information DC2, third correspondence information DC3, and fourth correspondence information DC4 are stored.

[0045] The program PR1 is a program for executing the projection method described in detail later.

[0046] The first captured data D1 is information representing a first captured image GG1, which will be described later, obtained by capturing a projection image G1 with the first camera 20-1.

[0047] The second captured data D2 is information representing a second captured image GG2, which will be described later, obtained by capturing a projection image G1 with the second camera 20-2.

[0048] The first internal parameter information PC1 is information representing the first internal parameters of the first camera 20-1. The first internal parameters are, for example, known parameters representing optical characteristics such as the optical center, focal length, and aberration of the first camera 20-1.

[0049] The second internal parameter information PC2 is information representing the second internal parameters of the second camera 20-2. The second internal parameters are, for example, known parameters representing optical characteristics such as the optical center, focal length, and aberration of the second camera 20-2.

[0050] The third internal parameter information PC3 is information representing the internal parameters of the projector 10-1, that is, the third internal parameters. The third internal parameters are, for example, parameters representing optical characteristics such as the field of view angle and optical center of the optical device 15 of the projector 10-1. In addition, the field of view angle varies due to the projection ratio or optical zoom. The optical center varies due to lens shift or lens displacement.

[0051] The first external parameter information PC4 is information representing the first external parameters, which represent one or both of the position and orientation of the second camera 20-2 relative to the first camera 20-1.

[0052] The second external parameter information PC5 is information representing the second external parameters, which represent one or both of the position and orientation of the projector 10-1 relative to the first camera 20-1.

[0053] The first coordinate information DP1 is information representing the coordinates of a plurality of first pixels, i.e., a plurality of first coordinates, in a first captured image GG1 described later shown in the first captured data D1. The first pixels are pixels of the first camera 20-1. The first coordinates are the coordinates of the first pixels in the coordinate system of the first camera 20-1.

[0054] The second coordinate information DP2 is information representing the coordinates of a plurality of second pixels, i.e., a plurality of second coordinates, corresponding to the plurality of first pixels in a second captured image GG2 shown in the second captured data D2. The second pixels are pixels of the second camera 20-2 and are pixels corresponding to the first pixels. The second coordinates are the coordinates of the second pixels in the coordinate system of the second camera 20-2.

[0055] The third coordinate information DP3 is information representing the three-dimensional coordinates, i.e., a plurality of third coordinates, of a first partial image R1-1 described later included in both the first captured image GG1 described later shown in the first captured data D1 and the second captured image GG2 described later shown in the second captured data D2 on a projection plane SC in a projection image G1. In addition, in this specification, the three-dimensional coordinates on the projection plane SC refer to coordinates in a world coordinate system, which is a three-dimensional coordinate system set in an actual space where the projection plane SC is provided or a virtual three-dimensional space corresponding to the actual space. Hereinafter, this three-dimensional coordinate system may be simply referred to as the "three-dimensional coordinate system". In addition, the three-dimensional coordinates may be simply referred to as the "three-dimensional coordinates".

[0056] The fourth coordinate information DP4 is information representing the coordinates of a plurality of third pixels, i.e., a plurality of fourth coordinates, in a display panel 15b of the projector 10-1. The third pixels are pixels of the display panel 15b. The fourth coordinates are the coordinates of the third pixels in the coordinate system of the display panel 15b.

[0057] The fifth coordinate information DP5 is information representing the coordinates of a plurality of reference points PR in the projection image G1 in the display panel 15b, i.e., a plurality of fifth coordinates.

[0058] The fifth transformed coordinate information DP6 is information representing the coordinates obtained by transforming the plurality of fifth coordinates shown in the fifth coordinate information DP5 into three-dimensional coordinates.

[0059] The shape information DP7 is information representing the three-dimensional shape on the projection plane SC of a second partial image R1-2 that is included in the first captured image GG1 shown in the first captured data D1 but not included in the second captured image GG2 shown in the second captured data D2 in the projected image G1. Additionally, the shape information DP7 may also include information representing the three-dimensional shape on the projection plane SC of a first partial image R1-1 (described later) that is included in both the first captured image GG1 (described later) shown in the first captured data D1 and the second captured image GG2 (described later) shown in the second captured data D2 in the projected image G1.

[0060] The first marker coordinate information DP8 is information representing the three-dimensional coordinates of a marker MK (described later) at a first timing.

[0061] The second marker coordinate information DP9 is information representing the three-dimensional coordinates of the marker MK (described later) at a second timing after the first timing.

[0062] The first correspondence information DC1 is information representing a first correspondence that establishes a correspondence between a plurality of first coordinates shown in the first coordinate information DP1 and a plurality of second coordinates shown in the second coordinate information DP2. That is, the first correspondence shown in the first correspondence information DC1 is the correspondence between the coordinate system of the first camera 20-1 and the coordinate system of the second camera 20-2.

[0063] The second correspondence information DC2 is information representing a second correspondence that establishes a correspondence between a plurality of first coordinates shown in the first coordinate information DP1 and a plurality of third coordinates shown in the third coordinate information DP3. That is, the second correspondence shown in the second correspondence information DC2 is the correspondence between the coordinate system of the first camera 20-1 and the three-dimensional coordinate system.

[0064] The third correspondence information DC3 is information representing a third correspondence that establishes a correspondence between a plurality of fourth coordinates shown in the fourth coordinate information DP4 and a plurality of first coordinates shown in the first coordinate information DP1. That is, the third correspondence shown in the third correspondence information DC3 is the correspondence between the coordinate system of the first camera 20-1 and the coordinate system of the display panel 15b of the projector 10-1.

[0065] The fourth correspondence information DC4 is information representing a fourth correspondence that establishes a correspondence between a plurality of third coordinates shown in the third coordinate information DP3 and a plurality of fourth coordinates shown in the fourth coordinate information DP4. That is, the fourth correspondence shown in the fourth correspondence information DC4 is the correspondence between the three-dimensional coordinate system and the coordinate system of the display panel 15b of the projector 10-1.

[0066] The processing device 12 is a processing device having functions of controlling respective parts of the projector 10-1 and processing various data. The processing device 12 is configured to include a processor such as a CPU (Central Processing Unit), for example. In addition, the processing device 12 may be constituted by a single processor or may be constituted by a plurality of processors. Further, part or all of the functions of the processing device 12 may be implemented by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). Additionally, the processing device 12 may be integrated with at least a part of the image processing circuit 14.

[0067] The communication device 13 is a communication device capable of communicating with various devices, obtaining the image data IMG1 from the terminal device 30, or communicating with the first camera 20-1 and the second camera 20-2, respectively. For example, the communication device 13 is a wired communication device such as a wired LAN (Local Area Network), a USB (Universal Serial Bus), or an HDMI (High Definition Multimedia Interface), or a wireless communication device such as an LPWA (Low Power Wide Area), a wireless LAN including Wi-Fi, or Bluetooth. "HDMI", "Wi-Fi", and "Bluetooth" are registered trademarks, respectively.

[0068] The image processing circuit 14 is a circuit that performs necessary processing on the image data IMG1 from the communication device 13 and inputs it to the optical device 15. The image processing circuit 14 has, for example, a frame memory (not shown), loads the image data IMG1 into the frame memory, appropriately performs various processes such as resolution conversion processing, size adjustment processing, and distortion correction processing, and inputs it to the optical device 15. Here, in these various processes, one or both of the aforementioned fourth correspondence information DC4 and shape information DP7 are appropriately used. In addition, the image processing circuit 14 may also perform processes such as OSD (On Screen Display) processing that generates image information for menu display or operation guidance, etc., and synthesizes it into the image data IMG1 as needed.

[0069] The optical device 15 is a device that projects image light onto the projection surface SC. The optical device 15 includes a light source 15a, a display panel 15b, and an optical system 15c.

[0070] The light source 15a includes, for example, a halogen lamp, a xenon lamp, an ultra-high pressure mercury lamp, an LED (Light Emitting Diode), or a laser light source, etc., and emits red, green, and blue lights respectively. The display panel 15b is a light modulator including three light modulation elements arranged corresponding to red, green, and blue. Each light modulation element includes, for example, a transmissive liquid crystal panel, a reflective liquid crystal panel, or a DMD (Digital Micromirror Device), etc., and generates image lights of various colors by modulating the lights of the corresponding colors. The image lights of various colors generated by the display panel 15b are synthesized by a color synthesis optical system to become full-color image lights. The optical system 15c is a projection optical system including a projection lens, etc., which images and projects the full-color image lights from the display panel 15b onto the projection surface SC.

[0071] The operation device 16 is a device that accepts operations from the user. For example, the operation device 16 includes an operation panel (not shown) and an infrared receiver for the remote controller. The operation panel is provided on the outer housing of the projector 10-1 and outputs a signal based on the operation from the user. The infrared receiver for the remote controller receives the infrared signal from a remote controller (not shown), decodes the infrared signal, and outputs a signal based on the operation of the remote controller. In addition, the operation device 16 is provided as needed and can be omitted.

[0072] In the above projector 10-1, the processing device 12 functions as a projection control unit 12a, a shooting control unit 12b, and a correction unit 12c by executing the program PR1 stored in the storage device 11. Therefore, the processing device 12 includes a projection control unit 12a, a shooting control unit 12b, and a correction unit 12c.

[0073] The projection control unit 12a controls the operations of the image processing circuit 14 and the optical device 15. More specifically, the projection control unit 12a projects the projection image G1 onto the projection surface SC by controlling the operation of the optical device 15.

[0074] The shooting control unit 12b controls the operations of the first camera 20-1 and the second camera 20-2 respectively. More specifically, the shooting control unit 12b obtains the first shooting data D1 by causing the first camera 20-1 to shoot the projection image G1 projected onto the projection surface SC, or obtains the second shooting data D2 by causing the second camera 20-2 to shoot the projection image G1 projected onto the projection surface SC, and stores the obtained first shooting data D1 and second shooting data D2 in the storage device 11.

[0075] The correction unit 12c uses the first shooting data D1, the second shooting data D2, the first internal parameter information PC1, and the second internal parameter information PC2 to perform correction for adjusting the shape of the projection image G1.

[0076] More specifically, the calibration unit 12c generates first correspondence information DC1 based on the first captured data D1 and the second captured data D2, and stores the generated first correspondence information DC1 in the storage device 11.

[0077] In addition, the calibration unit 12c generates first external parameter information PC4 based on the first internal parameter information PC1, the second internal parameter information PC2, and the first correspondence information DC1, and stores the generated first external parameter information PC4 in the storage device 11. This generation is performed for each first period. Thus, the first external parameter information PC4 is updated for each first period.

[0078] Furthermore, the calibration unit 12c generates third coordinate information DP3 based on the first internal parameter information PC1, the second internal parameter information PC2, the first correspondence information DC1, and the first external parameter information PC4, and stores the generated third coordinate information DP3 in the storage device 11.

[0079] In addition, the calibration unit 12c generates second correspondence information DC2 based on the first coordinate information DP1 and the third coordinate information DP3, and stores the generated second correspondence information DC2 in the storage device 11.

[0080] Furthermore, the calibration unit 12c generates third correspondence information DC3 based on the first coordinate information DP1 and the fourth coordinate information DP4, and stores the generated third correspondence information DC3 in the storage device 11.

[0081] In addition, the calibration unit 12c generates fourth correspondence information DC4 based on the second correspondence information DC2 and the third correspondence information DC3, and stores the generated fourth correspondence information DC4 in the storage device 11.

[0082] Moreover, the calibration unit 12c generates second external parameter information PC5 and third internal parameter information PC3 based on the fourth correspondence information DC4, and stores the generated second external parameter information PC5 and third internal parameter information PC3 in the storage device 11.

[0083] In addition, the calibration unit 12c generates shape information DP7 based on the second external parameter information PC5, the third internal parameter information PC3, and the third correspondence information DC3, and stores the generated shape information DP7 in the storage device 11.

[0084] Further, when the optical system 15c of the projector 10-1 is adjusted, the calibration unit 12c updates one or both of the second external parameter information PC5 and the third internal parameter information PC3 based on the results of the projection images G1 projected after adjusting the optical system 15c, which are captured by the first camera 20-1 and the second camera 20-2.

[0085] In addition, the calibration unit 12c generates fifth coordinate information DP5 based on a plurality of reference points PR, which will be described later, in the projection image G1, and stores the generated fifth coordinate information DP5 in the storage device 11.

[0086] Further, the calibration unit 12c generates fifth transformed coordinate information DP6 by transforming the fifth coordinate information DP5 using the fourth correspondence information DC4, and stores the generated fifth transformed coordinate information DP6 in the storage device 11.

[0087] In addition, the calibration unit 12c generates first marker coordinate information DP8 based on the fifth coordinate information DP5 and the second correspondence information DC2, and stores the generated first marker coordinate information DP8 in the storage device 11.

[0088] Further, the calibration unit 12c generates second marker coordinate information DP9 based on the fifth transformed coordinate information DP6 and the second correspondence information DC2, and stores the generated second marker coordinate information DP9 in the storage device 11.

[0089] In addition, the calibration unit 12c adjusts the projection image G1 based on the first marker coordinate information DP8, the second marker coordinate information DP9, the fifth transformed coordinate information DP6, the second internal parameter information PC2, and the second external parameter information PC5. Here, the calibration unit 12c generates the fifth coordinate information DP5 according to the result of receiving an operation from the user to move the plurality of reference points PR, which will be described later, in order to adjust the shape of the projection image G1.

[0090] 1-3. Control Method

[0091] Figure 3 is a flowchart showing the process of the projection method according to the first embodiment. This projection method is performed by the processing device 12 executing the program PR1 using the above system 100. As Figure 3 shown, this projection method includes steps S100 to step S500.

[0092] Specifically, first, in step S100, the shape of the projection surface SC is measured. Through this measurement, on the basis of obtaining the first captured data D1 and the second captured data D2, the third internal parameter information PC3, the first external parameter information PC4, the second external parameter information PC5, the first coordinate information DP1, the second coordinate information DP2, the third coordinate information DP3, the fourth coordinate information DP4, the shape information DP7, the first correspondence information DC1, the second correspondence information DC2, the third correspondence information DC3, and the fourth correspondence information DC4 are obtained by using the first internal parameter information PC1 and the second internal parameter information PC2. Regarding the details of step S100, it will be described later based on Figures 4 to 7 is described.

[0093] After step S100, in step S200, the projection image G1 is adjusted. Through this adjustment, the fifth coordinate information DP5, the fifth transformed coordinate information DP6, and the first marker coordinate information DP8 are obtained. Regarding the details of this obtaining, it will be described later based on Figures 8 to 10 is described.

[0094] Then, in step S300, it is determined whether the projection image G1 needs to be readjusted. For example, when the change amount of the positional relationship of the projector 10-1 with respect to the projection surface SC is equal to or greater than a specified amount, it is determined that the projection image G1 needs to be readjusted. In addition, it is possible to determine that the projection image G1 needs to be readjusted according to an instruction from the user, or it is possible to determine that the projection image G1 needs to be readjusted according to the output of a sensor or the like that detects the change amount of the positional relationship of the projector 10-1 with respect to the projection surface SC.

[0095] Step S300 is repeated (step S300: No) until it is determined that the projection image G1 needs to be readjusted. When it is determined that the projection image G1 needs to be readjusted (step S300: Yes), in step S400, the projection image G1 is readjusted. As a result, the projection state of the projection image G1 with respect to the projection surface SC is restored from the state at the second timing to the state at the first timing. Regarding the details of this restoration, it will be described later based on Figures 11 to 14 is described.

[0096] After step S400, in step S500, it is determined whether there is an end instruction. This determination is made, for example, by whether there is an end instruction based on an operation from the user.

[0097] In the case where there is no end instruction (step S500: No), the aforementioned step S300 is executed. As a result, the aforementioned steps S300 and S400 are repeated until there is an end instruction. On the other hand, in the case where there is an end instruction (step S500: Yes), the process ends.

[0098] Figure 4 It is a flowchart showing a measurement process of the shape of the projection surface SC. As Figure 4 shown, Figure 3 the step S100 shown contains steps S101 to S112.

[0099] Specifically, in step S101, the projection image G1 is projected from the projector 10-1 onto the projection surface SC. This projection is performed by the projection control unit 12a controlling the operation of the projector 10-1.

[0100] In step S102, the first captured image GG1 described later is obtained by capturing the projection image G1 using the first camera 20-1. This obtaining is performed by the imaging control unit 12b controlling the operation of the first camera 20-1. The obtained first captured image GG1 is stored in the storage device 11 as the first captured data D1.

[0101] In step S103, the second captured image GG2 described later is obtained by capturing the projection image G1 using the second camera 20-2. This obtaining is performed by the imaging control unit 12b controlling the operation of the second camera 20-2. The obtained second captured image GG2 is stored in the storage device 11 as the second captured data D2. In addition, step S103 can be executed before step S102 or during the period of repeating with step S102.

[0102] In step S104, the first correspondence is generated. This generation is performed by the calibration unit 12c based on the first coordinate information DP1 and the second coordinate information DP2. The obtained first correspondence is stored in the storage device 11 as the first correspondence information DC1.

[0103] In step S105, the first external parameter is calculated. This calculation is performed by the calibration unit 12c based on the first internal parameter indicated by the first internal parameter information PC1, the second internal parameter indicated by the second internal parameter information PC2, and the first correspondence indicated by the first correspondence information DC1. The calculated first external parameter is stored in the storage device 11 as the first external parameter information PC4.

[0104] In step S106, a plurality of third coordinates are obtained. The calibration unit 12c obtains a plurality of third coordinates based on the first internal parameter indicated by the first internal parameter information PC1, the second internal parameter indicated by the second internal parameter information PC2, the first correspondence indicated by the first correspondence information DC1, and the first external parameter indicated by the first external parameter information PC4. The obtained plurality of third coordinates are stored in the storage device 11 as the third coordinate information DP3.

[0105] Step S107 generates the second correspondence. This generation is performed by the calibration unit 12c based on the first coordinate information DP1 and the second coordinate information DP2. The generated second correspondence is stored in the storage device 11 as the second correspondence information DC2.

[0106] Step S108 generates the third correspondence. This generation is performed by the calibration unit 12c based on the fourth coordinate information DP4 and the first coordinate information DP1. The generated third correspondence is stored in the storage device 11 as the third correspondence information DC3. In addition, step S108 can be performed as long as the fourth coordinate information DP4 and the first coordinate information DP1 are generated, and can also be executed before step S107.

[0107] Step S109 generates the fourth correspondence. This generation is performed by the calibration unit 12c based on the second correspondence indicated by the second correspondence information DC2 and the third correspondence indicated by the third correspondence information DC3.

[0108] Step S110 calculates the second external parameter. This calculation is performed by the calibration unit 12c based on the fourth correspondence indicated by the fourth correspondence information DC4. The calculated second external parameter is stored in the storage device 11 as the second external parameter information PC5.

[0109] Step S111 calculates the third internal parameter. This calculation is performed by the calibration unit 12c based on the fourth correspondence indicated by the fourth correspondence information DC4. The calculated third internal parameter is stored in the storage device 11 as the third internal parameter information PC3. In addition, step S111 can be executed before step S110, or can also be executed during the period of repeating with step S110.

[0110] Step S112 obtains the shape information DP7. The shape information DP7 is obtained by the calibration unit 12c based on the second external parameter indicated by the second external parameter information PC5, the third internal parameter indicated by the third internal parameter information PC3, and the third correspondence indicated by the third correspondence information DC3. The obtained shape information DP7 is stored in the storage device 11.

[0111] Hereinafter, based on Figures 5 to 7 Steps S101 to S112 outlined above will be described in detail.

[0112] Figure 5 is a diagram for explaining the relationship between the coordinates of the projected image G1 in the display panel 15b and the coordinates in the first captured image GG1 and the second captured image GG2. In Figure 5In [it], the correspondence relationships are shown among the pixels of the display panel 15b that displays the projected image G1, i.e., the third pixel P0, the pixels of the first captured image GG1 shown in the first captured data D1, i.e., the first pixel P1, and the pixels of the second captured image GG2 shown in the second captured data D2, i.e., the second pixel P2.

[0113] In step S101, the projected image G1 is projected onto the projection surface SC. At this time, as Figure 5 shown in the upper part of [it], the projected image G1 is displayed on the display panel 15b. In Figure 5 the example shown in [it], the projected image G1 is an image in which a plurality of letters are arranged. In addition, the projected image G1 is an image for detecting the correspondence between the first pixel P1 and the second pixel P2 and the third pixel P0, and is not limited to Figure 5 the example shown in [it], and may also be a phase shift pattern, a Gray code pattern, a pattern image, etc.

[0114] In step S102, the first camera 20-1 captures the projected image G1 projected onto the projection surface SC, and thereby, as Figure 5 shown in the lower left part of [it], the first captured image GG1 shown in the first captured data D1 is obtained.

[0115] In step S103, the second camera 20-2 captures the projected image G1 projected onto the projection surface SC, and thereby, as Figure 5 shown in the lower right part of [it], the second captured image GG2 shown in the second captured data D2 is obtained.

[0116] In step S104, first, first coordinate information DP1 is generated based on the coordinates of a plurality of first pixels P1 in the coordinate system of the first captured image GG1, i.e., a plurality of first coordinates, and second coordinate information DP2 is generated based on the coordinates of a plurality of second pixels P2 in the coordinate system of the second captured image GG2, i.e., a plurality of second coordinates. Then, in step S104, based on the first coordinate information DP1 and the second coordinate information DP2, first correspondence relationship information DC1 is generated that represents the first correspondence relationship that establishes the correspondence between the plurality of first coordinates and the plurality of second coordinates.

[0117] Here, the plurality of first pixels P1 and the plurality of second pixels P2 correspond to each other. Therefore, the corresponding first pixel P1 and the second pixel P2 correspond to the same third pixel P0. The correspondence between such a plurality of first pixels P1 and a plurality of second pixels P2 is detected by a known measurement method. In addition, in step S104, along with the detection of the correspondence between the plurality of first pixels P1 and the plurality of second pixels P2, fourth coordinate information DP4 is obtained that represents the coordinates of a plurality of third pixels P0 in the coordinate system of the display panel 15b, i.e., the fourth coordinates. In addition, in Figure 5In the figure, for the sake of convenience of explanation, one each of the first pixel P1, the second pixel P2, and the third pixel P0 is shown respectively.

[0118] In step S105, taking the coordinate system of the first camera 20-1 as the reference coordinate, based on the first internal parameter information PC1, the second internal parameter information PC2, and the first correspondence information DC1, one or both of the position and orientation of the second camera 20-2 relative to the first camera 20-1 are estimated. Thereby, the first external parameter is obtained. In this estimation, for example, a known method such as the five-point algorithm like OpenCV is used. Here, this position is represented by a three-dimensional vector (tx, ty, tz), for example. In addition, this orientation is represented by a three-dimensional vector (rx, ry, rz) or a 3×3 rotation matrix, for example.

[0119] Here, in the cameras of the first camera 20-1 or the second camera 20-2, when the coordinates in the image sensor are set as (u, v) and the normalized coordinates obtained by normalizing these coordinates with the focal length f of this camera are (x, y), these coordinates have the following relationship.

[0120]

[0121] In the above formula, (c u , c v ) is the optical center coordinates of this camera. The internal parameters of this camera are represented by the internal parameter matrix A, which is the product of the optical center coordinates (c u , c v ) and the focal length f. In addition, the internal parameters of this camera can be obtained by multiplying the internal parameter matrix A by other parameters such as lens distortion coefficients.

[0122] Figure 6 is a diagram for explaining the first partial image R1-1 and the second partial image R1-2 of the projected image G1. In Figure 6 , in the first captured image GG1, the part of the contour of the second captured image GG2 that is included in the first captured image GG1 is represented by a hollow dotted line, and the part of the contour of the first captured image GG1 that is included in the second captured image GG2 is represented by a thick dotted line.

[0123] As Figure 6As shown, the projected image G1 in the first captured image GG1 is divided into a first partial image R1-1 and a second partial image R1-2 by the hollow dotted line representing a part of the contour of the second captured image GG2. The first partial image R1-1 is the area belonging to the above-mentioned area RC. That is, the first partial image R1-1 is the area included in both the first captured image GG1 and the second captured image GG2 in the projected image G1 projected onto the projection plane SC. On the other hand, the second partial image R1-2 is the area that is not included in the second captured image GG2 but is included in the first captured image GG1 in the projected image G1 projected onto the projection plane SC.

[0124] In this way, the first partial image R1-1 is captured by both the first camera 20-1 and the second camera 20-2. In addition, after step S105, as described above, the first internal parameter information PC1, the second internal parameter information PC2, and the second external parameter information PC5 have been obtained. Therefore, the multiple three-dimensional coordinates of the first partial image R1-1 on the projection plane SC can be calculated by triangulation.

[0125] In step S106, the multiple third coordinates calculated by such triangulation are obtained as the third coordinate information DP3.

[0126] In this way, after obtaining the third coordinate information DP3 in step S106, in step S107, based on the first coordinate information DP1 and the third coordinate information DP3, the second correspondence information DC2 representing the second correspondence that establishes the correspondence between the multiple first coordinates and the multiple third coordinates is generated.

[0127] In step S108, based on the fourth coordinate information DP4 and the first coordinate information DP1, the third correspondence information DC3 representing the third correspondence that establishes the correspondence between the multiple fourth coordinates and the multiple first coordinates is generated.

[0128] In this way, after obtaining the third correspondence information DC3 in step S108, in step S109, based on the second correspondence information DC2 and the third correspondence information DC3, the fourth correspondence information DC4 representing the fourth correspondence that establishes the correspondence between the multiple third coordinates and the multiple fourth coordinates is generated.

[0129] Thus, after obtaining the fourth correspondence information DC4 in step S109, in step S110, based on the fourth correspondence information DC4, the second external parameter information PC5 representing one or both of the position and orientation of the projector 10-1 relative to the first camera 20-1 is calculated. This calculation uses, for example, the method described in the literature (Zhang, Zhengyou. "A flexible new technique for camera calibration." IEEE Transactions on pattern analysis and machine intelligence 22.11 (2000): 1330-1334.).

[0130] In addition, after obtaining the fourth correspondence information DC4 in step S109, in step S111, based on the fourth correspondence information DC4, the third internal parameter information PC3 representing the third internal parameter which is the internal parameter of the projector 10-1 is calculated. In this calculation, similar to step S110, for example, the method described in the literature (Zhang, Zhengyou. "A flexible new technique for camera calibration." IEEE Transactions on pattern analysis and machine intelligence 22.11 (2000): 1330-1334.) is used.

[0131] Figure 7 It is a diagram for explaining the shape measurement of the second partial image R1-2 of the projected image G1. In Figure 7 In the first captured image GG1, the contour of the region R1-3 where the captured region of the first camera 20-1 overlaps with the display region of the display panel 15b is indicated by a one-dot chain line.

[0132] The region R1-3 is included in the first captured image GG1 and includes the first partial image R1-1 and the second partial image R1-2. In addition, after steps S110 and S111, as described above, the third internal parameter information PC3, the second external parameter information PC5, and the third correspondence information DC3 have been obtained. Therefore, similar to the triangulation measurement between the projection plane SC and the first camera 20-1 and the second camera 20-2 in step S106 above, by the triangulation measurement between the projection plane SC and the first camera 20-1 and the projector 10-1, a plurality of three-dimensional coordinates on the projection plane SC can be calculated.

[0133] In step S112, based on the result calculated by such triangulation measurement, shape information DP7 is obtained. The shape information DP7 represents the three-dimensional shape on the projection plane SC of the second partial image R1-2 that is not included in the second captured image GG2 but is included in the first captured image GG1 in the projected image G1.

[0134] Figure 8 is a flowchart showing the adjustment process of the projected image G1. As Figure 8 shown, Figure 3 the step S200 shown includes steps S201 to S204.

[0135] Specifically, in step S200, first, in step S201, it is determined whether there is an operation on a plurality of reference points PR described below in the projected image G1. This determination is made based on whether the correction unit 12c has received an operation by the user to move the plurality of reference points PR. If this operation is received, it is determined that there is an operation on the plurality of reference points PR in the projected image G1. In this way, step S201 receives from the user an operation to move the plurality of reference points PR in order to adjust the shape of the projected image G1.

[0136] Step S201 is repeated (step S201: No) until there is an operation on the plurality of reference points PR in the projected image G1. When there is an operation on the plurality of reference points PR in the projected image G1 (step S201: Yes), in step S202, the coordinates of the plurality of reference points PR in the display panel 15b, that is, the plurality of fifth coordinates, are determined. The plurality of fifth coordinates are stored in the storage device 11 as the fifth coordinate information DP5. In this way, step S202 determines the plurality of fifth coordinates after the operation of moving the plurality of reference points PR in order to adjust the shape of the projected image G1.

[0137] After step S202, in step S203, the plurality of fifth coordinates indicated by the fifth coordinate information DP5 are transformed into three-dimensional coordinates. This transformation is performed by the correction unit 12c using the fourth correspondence relationship indicated by the fourth correspondence information DC4. Through this transformation, the fifth transformation coordinate information DP6 is generated, and the generated fifth transformation coordinate information DP6 is stored in the storage device 11.

[0138] After step S203, in step S204, the three-dimensional coordinates of a marker MK described below set on the projection plane SC are obtained. This obtaining is performed by transforming the coordinates of the marker MK in the first captured image GG1 into three-dimensional coordinates using the second correspondence relationship indicated by the second correspondence information DC2. The obtained third-dimensional coordinates are stored in the storage device 11 as the first marker coordinate information DP8.

[0139] Hereinafter, based on Figure 11 andFigure 12 The above general steps S201 to S204 will be described in detail.

[0140] Figure 9 It is a diagram for explaining a plurality of reference points PR of the projected image G1 in the display panel 15b. In Figure 9 it, an example of the way in which a plurality of reference points PR are arranged in a lattice pattern on the display panel 15b is illustrated. In Figure 9 in the example shown, the plurality of reference points PR include a plurality of reference points PR arranged along a portion corresponding to the outer edge of the projection plane SC in the display panel 15b. In addition, the arrangement and number of the reference points PR in the display panel 15b are not limited to Figure 9 the example shown, and are arbitrary.

[0141] Each reference point PR can be moved by a user operation as needed. When two or more reference points PR selected by the user among the plurality of reference points PR are moved, as the interval between the plurality of reference points PR changes, a portion of the projected image G1 corresponding to the interval is deformed. Thus, the shape of the projected image G1 can be adjusted to match the shape of the projection plane SC.

[0142] In step S201, it is determined whether there is such an operation. After such step S201, in step S202, fifth coordinate information DP5 representing the coordinates of the plurality of reference points PR in the display panel 15b, that is, a plurality of fifth coordinates, is obtained.

[0143] After such step S202, in step S203, the fifth coordinate information DP5 is transformed using the fourth correspondence shown in the fourth correspondence information DC4, thereby generating fifth transformed coordinate information DP6.

[0144] Figure 10 It is a diagram for explaining the three-dimensional coordinates of the marker MK at the first timing. In Figure 10 it, in a three-dimensional coordinate system set in the actual space where the projection plane SC is provided or a virtual three-dimensional space corresponding to the actual space, the above-mentioned plurality of reference points PR and a plurality of markers MK detected from the first captured image GG1 and the second captured image GG2 are shown.

[0145] The plurality of markers MK are markers set at arbitrary positions on the projection plane SC. In addition, the shape, position, and number of the markers MK on the projection plane SC are not limited to Figure 10 the example shown, and are arbitrary. However, when the shape of the marker MK is not a shape capable of discriminating the posture of the projection plane SC, the marker MK is set at three or more arbitrary positions on the projection plane SC.

[0146] In step S204, after detecting the coordinates of the marker MK in the first captured image GG1 using a known technique such as the phase shift method or image recognition technology, the coordinates are transformed using the second correspondence information DC2, thereby generating first marker coordinate information DP8 representing the three-dimensional coordinates of the marker MK at the first timing.

[0147] Figure 11 is a flowchart showing the restoration process. As Figure 11 shown, Figure 3 the step S400 shown includes steps S401 to S406.

[0148] Specifically, first, in step S401, it is determined whether the first period has elapsed. This determination is made by the calibration unit 12c based on whether a predetermined timing after the last generation timing of the first external parameter information PC4 has been reached. In the case where the predetermined timing after the last generation timing of the first external parameter information PC4 has been reached, it is determined that the first period has elapsed. The predetermined timing is, for example, a timing arbitrarily set by the user, a timing at which the output of a sensor such as a temperature sensor or a vibration sensor becomes a predetermined state, or a timing at which a predetermined period has elapsed after the last generation timing of the first external parameter information PC4.

[0149] In the case where it is determined that the first period has elapsed (step S401: Yes), in step S402, the first external parameter is updated. Thus, the first external parameter is updated every first period. This update is performed by executing the aforementioned steps S101, S102, S103, S104, S105 again. The updated first external parameter is stored in the storage device 11 as the first external parameter information PC4.

[0150] After step S402, or in the case where it is determined that the first period has not elapsed (step S401: No), in step S403, it is determined whether an adjustment of the optical system 15c has been performed. This determination is made by whether the calibration unit 12c has performed an adjustment such as lens displacement of the optical system 15c. In the case where an adjustment such as lens displacement of the optical system 15c has been performed, it is determined that an adjustment of the optical system 15c has been performed.

[0151] In the case where the optical system 15c is adjusted (step S403: Yes), in step S404, the parameters related to the optical system 15c are updated. The parameter is one or both of the second external parameter and the third internal parameter. This update is performed by executing one or both of the aforementioned steps S111 and S112 again. Thus, in the case where the optical system 15c of the projector 10-1 is adjusted, based on the results of the projection image G1 projected after adjusting the optical system 15c captured by the first camera 20-1 and the second camera 20-2, one or both of the second external parameter shown in the second external parameter information PC5 and the third internal parameter shown in the third internal parameter information PC3 are updated. The updated second external parameter is stored in the storage device 11 as the second external parameter information PC5. In addition, the updated third internal parameter is stored in the storage device 11 as the third internal parameter information PC3.

[0152] After step S404, or in the case where the optical system 15c is not adjusted (step S403: No), in step S405, the three-dimensional coordinates of the marker MK at the second timing after the first timing are obtained. This acquisition is performed after the calibration unit 12c uses the first camera 20-1 and the second camera 20-2 to capture the projection surface SC again, and based on the captured image, through the same processing as in the above step S204. The obtained three-dimensional coordinates are stored in the storage device 11 as the second marker coordinate information DP9.

[0153] After step S405, in step S406, the projection image G1 is adjusted. This adjustment is performed by the calibration unit 12c based on the three-dimensional coordinates of the marker MK at the first timing, the three-dimensional coordinates of the marker MK at the second timing, the coordinates after transforming the multiple fifth coordinates shown in the fifth coordinate information DP5 into three-dimensional coordinates, the second internal parameter shown in the second internal parameter information PC2, and the second external parameter shown in the second external parameter information PC5.

[0154] Hereinafter, based on Figures 12 to 14 Step S406 in the above steps S401 to S406 will be described in detail.

[0155] Figure 12 This is a diagram for explaining the three-dimensional coordinates of the marker MK at the second timing. In Figure 12 it shows a plurality of markers MK detected based on the shooting result in step S405 in a three-dimensional coordinate system set in the actual space where the projection surface SC is provided or in a virtual three-dimensional space corresponding to the actual space.

[0156] Figure 13 This is a diagram for explaining the three-dimensional coordinates of the marker MK at the first timing and the second timing. In Figure 13In the three-dimensional coordinate system set in the actual space provided with the projection plane SC or the virtual three-dimensional space corresponding to the actual space, a plurality of markers MK detected based on the first captured image GG1 and the second captured image GG2 are denoted as marker MK-1, and a plurality of markers MK detected based on the imaging result in step S405 are denoted as marker MK-2.

[0157] In step S406, first, based on the coordinates of such a plurality of markers MK-1 and the coordinates of a plurality of markers MK-2, a function F representing the change in the position and orientation of the projection plane SC from the first timing to the second timing is obtained. The function F is a function representing a three-dimensional affine transformation composed of three-dimensional translation, three-dimensional rotation, and scale transformation.

[0158] Figure 14 It is a diagram for explaining the adjustment of the projection image G1. In Figure 14 In the three-dimensional coordinate system set in the actual space provided with the projection plane SC or the virtual three-dimensional space corresponding to the actual space, a plurality of reference points PR at the first timing are denoted as a plurality of reference points PR-1, and a plurality of reference points PR at the second timing are denoted as a plurality of reference points PR-2.

[0159] In step S406, as described above, after obtaining the function F representing the change in the position and orientation of the projection plane SC from the first timing to the second timing, by using this function F, the transformation from a plurality of reference points PR-1 to a plurality of reference points PR-2 is performed. Thereby, the projection image G1 is adjusted.

[0160] As described above, the above projection method includes step S101, step S102, step S103, step S104, step S105, step S106, step S107, step S108, and step S109. Thereby, a fourth correspondence relationship that can be used for various adjustments can be obtained.

[0161] As described above, the projection method of the present embodiment includes step S110 and step S111. By executing step S110, even if one or both of the position and orientation of the projector 10-1 relative to the first camera 20-1 change, various adjustments can be accurately performed based on the second external parameter shown in the second external parameter information PC5. In addition, by executing step S111, even if the optical characteristics of the projector 10-1 change, various adjustments can be accurately performed based on the third internal parameter shown in the third internal parameter information PC3. In addition, one of step S110 and step S111 may be omitted.

[0162] In addition, as described above, the projection method of the present embodiment includes step S112. Thus, in addition to the first partial image R1-1, three-dimensional information of the projection surface SC of the second partial image R1-2 can also be obtained.

[0163] Moreover, as described above, the projection method of the present embodiment includes step S402. Thus, it is possible to cope with changes in the positions of the first camera 20-1 and the second camera 20-2.

[0164] In addition, as described above, the projection method of the present embodiment includes step S404. Thus, it is possible to cope with the case where the optical setting of the projector 10-1 is changed.

[0165] Moreover, as described above, the projection method of the present embodiment includes step S202, step S203, step S204, step S405, and step S406. Thus, even if the projection image G1 at the second timing is shifted relative to the projection image G1 at the first timing on the projection surface SC, it is possible to return to the state of the projection image G1 at the first timing.

[0166] As described above, the projection method of the present embodiment includes step S201. Then, step S202 determines a plurality of fifth coordinates after the operation of moving a plurality of reference points PR to adjust the shape of the projection image G1. Thus, it is possible to maintain or restore the projection image G1 whose shape has been adjusted by the user.

[0167] 2. Second Embodiment

[0168] Hereinafter, a second embodiment of the present disclosure will be described. For elements having the same functions and operations as those in the first embodiment in the following exemplified modes, the reference numerals used in the description of the first embodiment are used and the detailed descriptions thereof are appropriately omitted.

[0169] Figure 15 FIG. is a schematic diagram showing a system 100A used in the projection method of the second embodiment. The system 100A is a multi-projection system that projects an image group GG onto a projection surface SC.

[0170] The system 100A is configured in the same manner as the system 100 of the first embodiment, except that a projector 10-2 is added. However, the projector 10-1 is the host and controls the operation of the projector 10-2 as a slave unit.

[0171] The projector 10-2 is a display device that projects a projection image G2 shown in the video data IMG2 output from the terminal device 30 onto the projection surface SC. The projector 10-2 is configured in the same manner as the projector 10-1, except that it is a slave unit. In addition, the projector 10-2 may have a structure different from that of the projector 10-1 as long as its operation can be controlled by the projector 10-1.

[0172] The projected images G1 and G2 are arranged in the left - right direction in Figure 15 and are projected onto the projection surface SC as an image group GG in a connected state. In Figure 15 the example shown, the projected image G1 is projected onto the Figure 15 left - hand region of the projection surface SC, while on the other hand, the projected image G2 is projected onto the Figure 15 right - hand region of the projection surface SC. Also, the Figure 15 right - hand end portion in Figure 15 of the projected image G1 and the

[0173] left - hand end portion in

[0174] of the projected image G2 overlap and join together in the overlapping region RG.

[0175] In the present embodiment, the second camera 20 - 2 photographs the region RC2 including the projected image G2.

[0176] Figure 16 is a block diagram of the projector 10 - 1 used in the system 100A of the second embodiment. The projector 10 - 1 is configured in the same manner as the projector 10 - 1 of the first embodiment, except that the program PR2 is used instead of the program PR1 of the first embodiment. Also, in Figure 16 the structure of the projector 10 - 1 is representatively shown, but the structure of the projector 10 - 2 is the same as that of the projector 10 - 1 except that it is a slave unit. Therefore, regarding the configuration of the projector 10 - 2, in the following description of the components, the video data IMG1 may be replaced with the video data IMG2. Hereinafter, regarding the structural elements of the projector 10 - 1, the structural elements of the projector 10 - 1 and the projector 10 - 2 may be distinguished by adding the suffix “-1” to the reference numerals of the structural elements of the projector 10 - 1 or adding the suffix “-2” to the reference numerals of the structural elements of the projector 10 - 2.

[0177] The program PR2 is a program for executing the projection method of the present embodiment and is stored in the storage device 11.

[0178] In the projector 10-1 of the present embodiment, the processing device 12 functions as a projection control unit 12a, a shooting control unit 12b, and a correction unit 12d by executing the program PR2.

[0179] The correction unit 12d is the same as the correction unit 12c of the first embodiment except for additionally performing correction to adjust the shape of the projected image G2. That is, the correction unit 12d uses the first shooting data D1, the second shooting data D2, the first internal parameter information PC1, and the second internal parameter information PC2 to perform correction to adjust the shapes of the projected images G1 and G2.

[0180] More specifically, the correction unit 12d generates the shape information DP10 by the same method as the generation of the shape information DP7, and stores the generated shape information DP10 in the storage device 11.

[0181] The shape information DP10 is information representing the three-dimensional shape on the projection surface SC of a third partial image R2-2 described later, which is not included in the first captured image GG1 shown in the first captured data D1 but is included in the second captured image GG2 shown in the second captured data D2 of the projected image G2.

[0182] In the present embodiment, when the second captured data D2 is obtained by shooting the projected image G1 with the second camera 20-2, not only the projected image G1 but also the projected image G2 is projected on the projection surface SC. Therefore, in the second captured image GG2 shown in the second captured data D2, in addition to the projected image G1, the projected image G2 is also displayed. In addition, when the first captured data D1 is obtained by shooting the projected image G1 with the first camera 20-1, not only the projected image G1 but also the projected image G2 may be projected on the projection surface SC. In this case, in the first captured image GG1 shown in the first captured data D1, not only the projected image G1 but also the projected image G2 is displayed.

[0183] Hereinafter, based on Figures 17 to 19 matters related to the generation of the shape information DP10 and the adjustment of the projected image G2 will be described.

[0184] Figure 17 is a diagram for explaining the projection of the projected images G1 and G2 onto the projection surface SC. In step S101 of the present embodiment, as Figure 17 shown, the projected image G1 is projected from the projector 10-1 onto the projection surface SC, and the projected image G2 is projected from the projector 10-2 onto the projection surface SC.

[0185] The projected image G2, like the projected image G1, is an image for detecting the correspondence between the pixels of the first captured image GG1 and the pixels of the second captured image GG2 shown in the second captured data D2 with respect to the pixels of the display panel 15b.

[0186] In Figure 17 the example shown, the projected images G1 and G2 are images that can be recognizably displayed in the region RC. Thus, steps S102 and S103 can be executed in a state where the projected image G1 and the projected image G2 are simultaneously projected onto the projection surface SC. In addition, the specific display content of the projected images G1 and G2 is not limited to Figure 17 the example shown and is arbitrary. In addition, the projected images G1 and G2 may not be recognizable in the region RC. In this case, steps S102 and S103 are respectively executed in a state where the projected image G2 is not projected onto the projection surface SC and the projected image G1 is projected, and in a state where the projected image G1 is not projected onto the projection surface SC and the projected image G2 is projected.

[0187] Figure 18 is a diagram for explaining the relationship between the coordinates of the projected images G1 and G2 in the display panels 15b-1 and 15b-2 and the coordinates in the first captured image GG1 and the second captured image GG2. In Figure 18 it shows the correspondence between the third pixel P0-1 which is the pixel of the display panel 15b-1 that displays the projected image G1, the fourth pixel P0-2 which is the pixel of the display panel 15b-2 that displays the projected image G2, the first pixels P1-1 and P1-2 which are the pixels of the first captured image GG1 shown in the first captured data D1, and the second pixels P2-1 and P2-2 which are the pixels of the second captured image GG2 shown in the second captured data D2.

[0188] In step S101 of the present embodiment, the projected images G1 and G2 are projected onto the projection surface SC. At this time, as shown in the upper left of Figure 18 , similar to the first embodiment, the projected image G1 is displayed on the display panel 15b-1. In addition, as shown in the upper right of Figure 18 , the projected image G2 is displayed on the display panel 15b-2.

[0189] In step S102 of the present embodiment, the first camera 20-1 captures the projected images G1 and G2 projected onto the projection surface SC. Thus, as shown in the lower left of Figure 18 , the first captured image GG1 shown in the first captured data D1 is obtained. In addition, in step S102, the projected image G2 may not be projected onto the projection surface SC.

[0190] In step S103 of the present embodiment, the second camera 20-2 captures the projection images G1 and G2 projected onto the projection surface SC, whereby, as shown on the lower right side of Figure 18 , the second captured image GG2 shown in the second captured data D2 is obtained.

[0191] In step S104 of the present embodiment, first, first coordinate information DP1 is generated based on the coordinates of a plurality of first pixels P1-1 and P1-2 in the coordinate system of the first captured image GG1, that is, a plurality of first coordinates, and second coordinate information DP2 is generated based on the coordinates of a plurality of second pixels P2-1 and P2-2 in the coordinate system of the second captured image GG2, that is, a plurality of second coordinates. Then, in step S104 of the present embodiment, based on the first coordinate information DP1 and the second coordinate information DP2, first correspondence relationship information DC1 representing a first correspondence relationship that establishes a correspondence between the plurality of first coordinates and the plurality of second coordinates is generated.

[0192] Here, the plurality of first pixels P1-1 and the plurality of second pixels P2-1 correspond to each other in the same manner as the correspondence between the plurality of first pixels P1 and the plurality of second pixels P2 in the first embodiment. Therefore, the corresponding first pixel P1-1 and second pixel P2-1 correspond to the same third pixel P0-1. In addition, the plurality of first pixels P1-2 correspond to the plurality of second pixels P2-2. Therefore, the corresponding first pixel P1-2 and second pixel P2-2 correspond to the same fourth pixel P0-2. Furthermore, in Figure 18 , for ease of explanation, one of each of the first pixels P1-1, P1-2, the second pixels P2-1, P2-2, and the third pixels P0-1, P0-2 is illustrated respectively.

[0193] Figure 19 is a diagram for explaining the shape measurement of the third partial image R2-2 of the projection image G2. In Figure 19 , in the second captured image GG2, the portion of the contour of the second captured image GG2 that is included in the first captured image GG1 is represented by a hollow dotted line, and the portion of the contour of the first captured image GG1 that is included in the second captured image GG2 is represented by a thick dotted line. And, in Figure 19 , in the second captured image GG2, the contour of the region R2-3 where the capture region of the second camera 20-2 and the display region of the display panel 15b-2 overlap is shown by a dashed-dotted line.

[0194] As Figure 19As shown, the projected image G2 in the second captured image GG2 is divided into a partial image R2-1 and a third partial image R2-2 by a thick dashed line representing a part of the contour of the first captured image GG1. The partial image R2-1 is an area belonging to the above-mentioned area RC. That is, the partial image R2-1 is an area included in both the first captured image GG1 and the second captured image GG2 in the projected image G2 projected onto the projection plane SC. On the other hand, the third partial image R2-2 is an area that is not included in the first captured image GG1 but is included in the second captured image GG2 in the projected image G2 projected onto the projection plane SC.

[0195] The area R2-3 is included in the second captured image GG2 and includes the partial image R2-1 and the third partial image R2-2. In step S112 of the present embodiment, in the same manner as calculating the three-dimensional shape of the second partial image R1-2 of the projected image G1 on the projection plane SC, the three-dimensional shape of the third partial image R2-2 of the projected image G2 on the projection plane SC is calculated. Thereby, the shape information DP10 is obtained.

[0196] According to the above second embodiment, various adjustments of the projectors 10-1 and 10-2 can also be performed.

[0197] 3. Modification

[0198] Each of the above-exemplified methods can be variously modified. The following exemplify specific modification methods that can be applied to the above-mentioned methods. Two or more methods arbitrarily selected from the following exemplifications can be appropriately combined within a non-contradictory range.

[0199] 3-1. Modification 1

[0200] In the above embodiment, a method in which the processing device 12 of the projector 10-1 executes the programs PR1 and PR2 is exemplified, but it is not limited to this method. For example, the programs PR1 and PR2 can also be executed by a processing device of a computer that is communicably connected to the projector 10-1 and the first camera 20-1.

[0201] 3-2. Modification 2

[0202] In the above-mentioned embodiment, a method in which the fourth correspondence relationship is used for adjusting the projected image G1 is exemplified, but it is not limited to this method. For example, the fourth correspondence relationship can be used to display a pattern such as a grid pattern with uniformity on the projection plane SC, or can be used to reflect the three-dimensional shape model of the projection plane SC observed from the first camera 20-1 onto three-dimensional image editing software or the like, and then draw a picture on the model, or display it on a PC monitor or the like or cause the projector 10-1 to project, and see how it looks when observing the picture from the projector 10-1.

[0203] 4. Supplementary Notes

[0204] The summary of the present disclosure is noted below.

[0205] (Supplementary Note 1) The projection method according to the first aspect of the present disclosure includes: projecting a projection image from a projector onto a projection surface; capturing the projection image with a first camera having first internal parameters to obtain a first captured image; capturing the projection image with a second camera having second internal parameters to obtain a second captured image; generating a first correspondence relationship in which a plurality of first coordinates and a plurality of second coordinates correspond to each other, the plurality of first coordinates being the coordinates of a plurality of first pixels in the first captured image, and the plurality of second coordinates being the coordinates of a plurality of second pixels corresponding to the plurality of first pixels in the second captured image; calculating a first external parameter representing one or both of the position and orientation of the second camera relative to the first camera based on the first internal parameters, the second internal parameters, and the first correspondence relationship; obtaining a plurality of third coordinates, the plurality of third coordinates being the three-dimensional coordinates of a first partial image included in both the first captured image and the second captured image in the projection surface based on the first internal parameters, the second internal parameters, the first correspondence relationship, and the first external parameter; generating a second correspondence relationship in which the plurality of first coordinates and the plurality of third coordinates correspond to each other; generating a third correspondence relationship in which a plurality of fourth coordinates and the plurality of first coordinates correspond to each other, the plurality of fourth coordinates being the coordinates of a plurality of third pixels in the display panel of the projector; and generating a fourth correspondence relationship in which the plurality of third coordinates and the plurality of fourth coordinates correspond to each other based on the second correspondence relationship and the third correspondence relationship.

[0206] In the above aspect, a fourth correspondence relationship that can be used for various adjustments can be obtained.

[0207] (Supplementary Note 2) In the second aspect, which is a preferred example of the first aspect, the projection method includes: calculating a second external parameter representing one or both of the position and orientation of the projector relative to the first camera and a third internal parameter that is an internal parameter of the projector based on the fourth correspondence relationship. In the above aspect, even if one or both of the position and orientation of the projector relative to the first camera change, various adjustments can be performed with high accuracy based on the second external parameter. In addition, even if the optical characteristics of the projector change, various adjustments can be performed with high accuracy based on the third internal parameter.

[0208] (Supplementary Note 3) In the third mode, which is a preferred example of the first mode or the second mode, the projection method includes: obtaining shape information according to the second external parameter, the third internal parameter, and the third correspondence relationship, where the shape information represents the three-dimensional shape on the projection surface of the second partial image that is not included in the second captured image but is included in the first captured image in the projection image. In the above mode, in addition to the first partial image, the three-dimensional information of the second partial image on the projection surface can also be obtained.

[0209] (Supplementary Note 4) In the fourth mode, which is a preferred example of any one of the first mode to the third mode, the projection method includes: updating the first external parameter every first period. In the above mode, the position changes of the first camera and the second camera can be dealt with.

[0210] (Supplementary Note 5) In the fifth mode, which is a preferred example of the second mode, the projection method includes: when the optical system of the projector is adjusted, updating one or both of the second external parameter and the third internal parameter according to the result of capturing the projection image projected after adjusting the optical system by using the first camera and the second camera. In the above mode, the situation where the optical setting of the projector has changed can be dealt with.

[0211] (Supplementary Note 6) In the sixth mode, which is a preferred example of the second mode, the projection method includes: determining a plurality of fifth coordinates that are the coordinates of a plurality of reference points in the display panel in the projection image; transforming the plurality of fifth coordinates into three-dimensional coordinates according to the fourth correspondence relationship; obtaining the three-dimensional coordinates of the marker at the first timing according to the second correspondence relationship; obtaining the three-dimensional coordinates of the marker at the second timing after the first timing according to the second correspondence relationship; and adjusting the projection image according to the three-dimensional coordinates of the marker at the first timing, the three-dimensional coordinates of the marker at the second timing, the coordinates obtained by transforming the plurality of fifth coordinates into three-dimensional coordinates, the second internal parameter, and the second external parameter. In the above mode, even if the projection image at the second timing is shifted relative to the projection image at the first timing on the projection surface, it can be returned to the state of the projection image at the first timing.

[0212] (Supplementary Note 7) In the seventh mode, which is a preferred example of the sixth mode, the projection method includes: accepting an operation by the user to move the plurality of reference points in order to adjust the shape of the projection image, and determining the plurality of fifth coordinates after the operation. In the above mode, the maintenance or restoration of the projection image whose shape has been adjusted by the user can be performed.

[0213] (Supplementary Note 8) The system according to the eighth aspect, which is a preferred example of the present disclosure, includes: an optical device of a projector; a first camera; a second camera; and a processing device that controls operations of the optical device, the first camera, and the second camera. The processing device performs the following processes: projecting a projection image from the projector onto a projection surface; capturing the projection image with the first camera having first internal parameters to obtain a first captured image; capturing the projection image with the second camera having second internal parameters to obtain a second captured image; generating a first correspondence in which a plurality of first coordinates and a plurality of second coordinates are corresponding, where the plurality of first coordinates are coordinates of a plurality of first pixels in the first captured image, and the plurality of second coordinates are coordinates of a plurality of second pixels corresponding to the plurality of first pixels in the second captured image; calculating first external parameters representing one or both of the position and orientation of the second camera relative to the first camera based on the first internal parameters, the second internal parameters, and the first correspondence; obtaining a plurality of third coordinates, which are three-dimensional coordinates on the projection surface of a first partial image included in both the first captured image and the second captured image in the projection image, based on the first internal parameters, the second internal parameters, the first correspondence, and the first external parameters; generating a second correspondence in which the plurality of first coordinates and the plurality of third coordinates are corresponding; generating a third correspondence in which a plurality of fourth coordinates and the plurality of first coordinates are corresponding, where the plurality of fourth coordinates are coordinates of a plurality of third pixels in a display panel of the projector; and generating a fourth correspondence in which the plurality of third coordinates and the plurality of fourth coordinates are corresponding based on the second correspondence and the third correspondence.

[0214] In the above aspect, a fourth correspondence that can be used for various adjustments can be obtained.

[0215] (Supplementary Note 9) The program according to the ninth aspect, which is a preferred example of the present disclosure, causes a computer to execute the following processes: projecting a projection image from a projector onto a projection surface; capturing the projection image with a first camera having first internal parameters to obtain a first captured image; capturing the projection image with a second camera having second internal parameters to obtain a second captured image; generating a first correspondence in which a plurality of first coordinates and a plurality of second coordinates correspond to each other, the plurality of first coordinates being coordinates of a plurality of first pixels in the first captured image, and the plurality of second coordinates being coordinates of a plurality of second pixels in the second captured image that correspond to the plurality of first pixels; calculating first external parameters representing one or both of the position and orientation of the second camera relative to the first camera based on the first internal parameters, the second internal parameters, and the first correspondence; obtaining a plurality of third coordinates, which are three-dimensional coordinates on the projection surface of a first partial image included in both the first captured image and the second captured image in the projection image, based on the first internal parameters, the second internal parameters, the first correspondence, and the first external parameters; generating a second correspondence in which the plurality of first coordinates and the plurality of third coordinates correspond to each other; generating a third correspondence in which a plurality of fourth coordinates and the plurality of first coordinates correspond to each other, the plurality of fourth coordinates being coordinates of a plurality of third pixels in a display panel of the projector; and generating a fourth correspondence in which the plurality of third coordinates and the plurality of fourth coordinates correspond to each other based on the second correspondence and the third correspondence.

[0216] In the above aspect, a fourth correspondence that can be used for various adjustments can be obtained.

Claims

1. A projection method, comprising: projecting a projection image from a projector onto a projection surface; The projection image is photographed by a first camera having first internal parameters to obtain a first photographed image; photographing the projection image by a second camera having second internal parameters to obtain a second photographed image; generating a first correspondence relationship between a plurality of first coordinates and a plurality of second coordinates, wherein the plurality of first coordinates are coordinates of a plurality of first pixels in the first captured image, and the plurality of second coordinates are coordinates of a plurality of second pixels in the second captured image corresponding to the plurality of first pixels; Calculating a first external parameter representing one or both of a position and a posture of the second camera relative to the first camera based on the first internal parameter, the second internal parameter, and the first correspondence relationship; Based on the first internal parameter, the second internal parameter, the first corresponding relationship, and the first external parameter, a plurality of third coordinates are obtained, wherein the plurality of third coordinates are three-dimensional coordinates of a first partial image included in both the first captured image and the second captured image in the projected image on the projection surface; generating a second correspondence relationship that establishes correspondence between the plurality of first coordinates and the plurality of third coordinates; generating a third correspondence relationship that establishes a plurality of fourth coordinates corresponding to the plurality of first coordinates, wherein the plurality of fourth coordinates are coordinates of a plurality of third pixels in a display panel of the projector; as well as Based on the second correspondence relationship and the third correspondence relationship, a fourth correspondence relationship is generated in which the plurality of third coordinates are associated with the plurality of fourth coordinates.

2. The projection method according to claim 1, wherein: The projection method includes calculating, based on the fourth correspondence relationship, a second external parameter indicating one or both of a position and a posture of the projector relative to the first camera and a third internal parameter which is an internal parameter of the projector.

3. The projection method according to claim 2, wherein: The projection method includes: obtaining shape information based on the second external parameter, the third internal parameter and the third corresponding relationship, wherein the shape information represents the three-dimensional shape of the second partial image of the projected image that is not included in the second captured image but included in the first captured image on the projection surface.

4. The projection method according to claim 1, wherein: The projection method includes: updating the first external parameter every first period.

5. The projection method according to claim 2, wherein: The projection method includes: when the optical system of the projector is adjusted, updating one or both of the second external parameter and the third internal parameter based on the result of photographing the projection image projected after adjusting the optical system using the first camera and the second camera.

6. The projection method according to claim 2, wherein: The projection method comprises: determining a plurality of fifth coordinates as coordinates of a plurality of reference points in the projected image in the display panel; According to the fourth corresponding relationship, transforming the plurality of fifth coordinates into three-dimensional coordinates; According to the second corresponding relationship, the three-dimensional coordinates of the marker at the first timing are obtained; acquiring the three-dimensional coordinates of the marker at a second timing after the first timing based on the second correspondence relationship; and The projection image is adjusted based on the three-dimensional coordinates of the marker at the first timing, the three-dimensional coordinates of the marker at the second timing, coordinates obtained by transforming the plurality of fifth coordinates into three-dimensional coordinates, the second internal parameter, and the second external parameter.

7. The projection method according to claim 6, wherein: The projection method includes: accepting an operation from a user to move the plurality of reference points in order to adjust the shape of the projection image; The plurality of fifth coordinates are determined after the operation.

8. A projection system comprising: Optics of the projector; Camera No. 1; a second camera; and a processing device that controls the operations of the optical device, the first camera, and the second camera, The processing device performs the following processing: projecting a projection image from the projector onto a projection surface; photographing the projection image by the first camera having first internal parameters to obtain a first photographed image; photographing the projection image by the second camera having second internal parameters to obtain a second photographed image; generating a first correspondence relationship between a plurality of first coordinates and a plurality of second coordinates, wherein the plurality of first coordinates are coordinates of a plurality of first pixels in the first captured image, and the plurality of second coordinates are coordinates of a plurality of second pixels in the second captured image corresponding to the plurality of first pixels; Calculating a first external parameter representing one or both of a position and a posture of the second camera relative to the first camera based on the first internal parameter, the second internal parameter, and the first correspondence relationship; Based on the first internal parameter, the second internal parameter, the first corresponding relationship, and the first external parameter, a plurality of third coordinates are obtained, wherein the plurality of third coordinates are three-dimensional coordinates of a first partial image included in both the first captured image and the second captured image in the projected image on the projection surface; generating a second correspondence relationship that establishes correspondence between the plurality of first coordinates and the plurality of third coordinates; generating a third correspondence relationship that establishes a plurality of fourth coordinates corresponding to the plurality of first coordinates, wherein the plurality of fourth coordinates are coordinates of a plurality of third pixels in a display panel of the projector; as well as Based on the second correspondence relationship and the third correspondence relationship, a fourth correspondence relationship is generated in which the plurality of third coordinates are associated with the plurality of fourth coordinates.

9. A program product that causes a computer to execute the following processing: projecting a projection image from a projector onto a projection surface; The projection image is photographed by a first camera having first internal parameters to obtain a first photographed image; photographing the projection image by a second camera having second internal parameters to obtain a second photographed image; generating a first correspondence relationship between a plurality of first coordinates and a plurality of second coordinates, wherein the plurality of first coordinates are coordinates of a plurality of first pixels in the first captured image, and the plurality of second coordinates are coordinates of a plurality of second pixels in the second captured image corresponding to the plurality of first pixels; Calculating a first external parameter representing one or both of a position and a posture of the second camera relative to the first camera based on the first internal parameter, the second internal parameter, and the first correspondence relationship; Based on the first internal parameter, the second internal parameter, the first corresponding relationship, and the first external parameter, a plurality of third coordinates are obtained, wherein the plurality of third coordinates are three-dimensional coordinates of a first partial image included in both the first captured image and the second captured image in the projected image on the projection surface; generating a second correspondence relationship that establishes correspondence between the plurality of first coordinates and the plurality of third coordinates; generating a third correspondence relationship that establishes a plurality of fourth coordinates corresponding to the plurality of first coordinates, wherein the plurality of fourth coordinates are coordinates of a plurality of third pixels in a display panel of the projector; as well as Based on the second correspondence relationship and the third correspondence relationship, a fourth correspondence relationship is generated in which the plurality of third coordinates are associated with the plurality of fourth coordinates.

Images