Projection method, system and program

Abstract

To provide a projection method that can omit chores of users.SOLUTION: A projection method includes: projecting a projection image onto a projection surface from a projector; acquiring a photographing image taking a photograph of the projection image by a camera having a first internal parameter; generating a first correspondence relation having coordinates of a plurality of first pixels in the photographing image and coordinates of a plurality of second pixels on a display panel of the projector associated with each other; acquiring a state information indicative of a state of an optical system of the projector; estimating a second internal parameter of the optical system on the basis of a second correspondence relation having the second internal parameter and the sate of the optical system associated with each other, and the state information; and estimating an external parameter indicative of one or both of a position and posture of the projector with respect to the camera on the basis of the first internal parameter, the first correspondence relation and the second internal parameter.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a projection method, system, and program. [Background technology]

[0002] For example, Patent Document 1 discloses a technology in which a red calibration pattern is projected from a projector onto a calibration plate with a yellow pattern printed on it, the calibration plate is photographed with a camera, and the internal parameters of the projector are determined based on the photographed image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2008 / 0285843 Summary of the Invention [Problem to be solved by the invention]

[0004] The internal parameters of a projector are used to determine, for example, one or both of the position and attitude of the projector relative to a camera as external parameters of the projector. In Patent Document 1, determining the internal parameters requires preparing not only a projector and a camera but also a calibration plate, which is time-consuming for the user. [Means for solving the problem]

[0005] A projection method according to one embodiment of the present disclosure includes: projecting a drawn image drawn on a display panel of a projector onto a projection surface from the projector as a projected image; acquiring a captured image by capturing the projected image with a camera having a first internal parameter; generating a first correspondence relationship in which a plurality of first coordinates, which are coordinates of a plurality of first pixels constituting the projected image in the captured image, correspond to a plurality of second coordinates, which are coordinates of a plurality of second pixels constituting the drawn image; acquiring status information indicating a state of an optical system of the projector; estimating the second internal parameter based on the status information and a second correspondence relationship in which a second internal parameter, which is an internal parameter of the optical system, corresponds to a state of the optical system; and estimating an external parameter indicating one or both of a position and an attitude of the projector relative to the camera based on the first internal parameter, the first correspondence relationship, and the second internal parameter.

[0006] A system according to one embodiment of the present disclosure is a system including a projector and a camera having a first internal parameter, wherein the projector projects a projection image from the projector onto a projection surface, acquires a captured image by capturing the projection image with the camera, generates a first correspondence relationship in which a plurality of first coordinates that are coordinates of a plurality of first pixels that constitute the projection image in the captured image correspond to a plurality of second coordinates that are coordinates of a plurality of second pixels that constitute the projection image on a display panel of the projector, acquires status information indicating a state of an optical system of the projector, estimates the second internal parameter based on the status information and a second correspondence relationship in which a second internal parameter that is an internal parameter of the optical system corresponds to the state of the optical system, and estimates an external parameter that indicates one or both of a position and an attitude of the projector relative to the camera based on the first internal parameter, the first correspondence relationship, and the second internal parameter.

[0007] A program according to one embodiment of the present disclosure causes a computer to execute the following steps: project a projection image from a projector onto a projection surface; acquire a captured image by capturing the projection image with a camera having first internal parameters; generate a first correspondence relationship in which a plurality of first coordinates, which are coordinates of a plurality of first pixels that constitute the projection image in the captured image, correspond to a plurality of second coordinates, which are coordinates of a plurality of second pixels that constitute the projection image on a display panel of the projector; acquire status information indicating a state of an optical system of the projector; estimate the second internal parameters based on the status information and a second correspondence relationship in which second internal parameters, which are internal parameters of the optical system, correspond to the state of the optical system; and estimate external parameters indicating one or both of a position and an attitude of the projector relative to the camera based on the first internal parameters, the first correspondence relationship, and the second internal parameters. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing an outline of a system used in a projection method according to a first embodiment. [Figure 2] FIG. 2 is a block diagram of a projector used in the system according to the first embodiment. [Figure 3] 4 is a flowchart showing the flow of a projection method according to the first embodiment. [Figure 4] 5 is a flowchart showing the flow of measuring the shape of the projection surface in the first embodiment. [Figure 5] 10A and 10B are diagrams for explaining the relationship between the coordinates of a projected image on a display panel and the coordinates of a captured image. [Figure 6] FIG. 10 is a diagram for explaining an example of a second correspondence relationship. [Figure 7] 10 is a flowchart showing the flow of adjustment of a projection image. [Figure 8] 10 is a diagram for explaining a plurality of reference points on a display panel of a projection image. FIG. [Figure 9] FIG. 10 is a diagram for explaining three-dimensional coordinates of a marker at a first timing. [Figure 10] 10 is a flowchart showing a restoration flow. [Figure 11] FIG. 10 is a diagram for explaining three-dimensional coordinates of the marker at a second timing. [Figure 12] 10A and 10B are diagrams for explaining three-dimensional coordinates of a marker at a first timing and a second timing. [Figure 13] FIG. 10 is a diagram for explaining adjustment of a projection image. [Figure 14] FIG. 10 is a diagram showing an outline of a system used in a projection method according to a second embodiment. [Figure 15] FIG. 10 is a block diagram of a projector used in a system according to a second embodiment. [Figure 16] 10 is a flowchart showing the flow of measuring the shape of the projection surface in the second embodiment. [Figure 17] 10A and 10B are diagrams for explaining shape measurement of a first portion and a second portion of a projection surface. DETAILED DESCRIPTION OF THE INVENTION

[0009] Preferred embodiments of the present disclosure will be described below with reference to the accompanying drawings. Note that the dimensions and scale of each part in the drawings may differ from the actual dimensions and are shown schematically to facilitate understanding. Furthermore, the scope of the present disclosure is not limited to these embodiments unless otherwise specified in the following description to the effect that the present disclosure is limited.

[0010] 1. First embodiment 1-1. System Overview 1 is a diagram showing an outline of a system 100 used in a projection method according to the first embodiment. The system 100 is a projection system that projects a projection image G onto a projection surface SC.

[0011] The projection surface SC is the surface of an object such as a screen. In the example shown in Fig. 1, the projection surface SC is flat. The shape of the projection surface SC is not limited to the example shown in Fig. 1, and may be, for example, a surface having a concavely curved portion as in the second embodiment described below, or a surface having a convexly curved portion.

[0012] As shown in FIG. 1, the system 100 includes a projector 10, a camera 20, and a terminal device 30.

[0013] The projector 10 is a display device that projects a projection image G indicated by video data IMG output from the terminal device 30 onto a projection surface SC. In the example shown in Fig. 1, the projection image G is projected onto an area that includes the projection surface SC. Note that the projection position of the projection image G relative to the projection surface SC is not limited to the example shown in Fig. 1 and is arbitrary.

[0014] The projector 10 of this embodiment has a function of controlling the operation of the camera 20 and a function of adjusting the shape of the projection image G using the imaging results of the camera 20. The camera 20 is a digital camera having an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 20 images an area RC. The area RC is an area that includes the projection image G projected onto the projection surface SC. Note that the camera 20 may be a component of the projector 10.

[0015] The terminal device 30 is a computer that has a function of supplying video data IMG to the projector 10. In the example shown in Fig. 1, the terminal device 30 is a notebook computer. Note that the terminal device 30 is not limited to a notebook computer, and may be, for example, a desktop computer, a smartphone, a tablet terminal, a video playback device, a DVD (Digital Versatile Disk) player, a Blu-ray Disc player, a hard disk recorder, a television tuner device, a CATV (Cable television) set-top box, a video game console, or the like.

[0016] 1-2.Projector 2 is a block diagram of the projector 10 used in the system 100 according to the first embodiment. In addition to the projector 10, Fig. 2 shows the connection state of the camera 20 and the terminal device 30 to the projector 10.

[0017] 2, the projector 10 has a storage device 11, a processing device 12, a communication device 13, an image processing circuit 14, an optical device 15, and an operation device 16. These are connected to each other so that they can communicate with each other.

[0018] The storage device 11 is a storage device that stores programs executed by the processing device 12 and data processed by the processing device 12. The storage device 11 includes, for example, a hard disk drive or a semiconductor memory. Note that part or all of the storage device 11 may be provided in an external storage device or server outside the projector 10.

[0019] The memory device 11 stores a program PR1, imaging data D1, status information D2, first internal parameter information PC1, second internal parameter information PC2, external parameter information PC3, first coordinate information DP1, second coordinate information DP2, third coordinate information DP3, fifth coordinate information DP5, fifth transformed coordinate information DP6, first marker coordinate information DP8, second marker coordinate information DP9, first correspondence relationship information DC1, second correspondence relationship information DC2, third correspondence relationship information DC3, and correction information DC0.

[0020] The program PR1 is a program for executing a projection method, which will be described in detail later.

[0021] The imaging data D1 is information indicating a captured image GG, which will be described later, acquired by capturing the projection image G with the camera 20.

[0022] The status information D2 is information that indicates the status of the optical system 15c of the projector 10. The status of the optical system 15c of the projector 10 refers to the status of optical characteristics such as optical distortion, lens shift amount, and throw ratio, and is indicated by information such as the number of steps of the motor that drives the optical system 15c, the current or voltage applied to the motor, and the detection value of a sensor that detects the position of the lens included in the optical system 15c. The lens included in the optical system 15c refers to the projection lens. Here, the optical distortion differs depending on the lens shift amount, and is therefore mapped for each lens shift amount.

[0023] The first internal parameter information PC1 is information indicating first internal parameters of the camera 20. The first internal parameters are known parameters that indicate optical characteristics such as the optical center, focal length, and aberration of the camera 20. Hereinafter, the first internal parameters indicated by the first internal parameter information PC1 may be simply referred to as "first internal parameters."

[0024] The second internal parameter information PC2 is information indicating second internal parameters, which are internal parameters of the projector 10. The second internal parameters are parameters that represent optical characteristics such as the angle of view and optical center of the optical device 15 of the projector 10, for example. The angle of view varies depending on the throw ratio or optical zoom. The optical center varies depending on the lens offset or lens shift. Hereinafter, the second internal parameters indicated by the second internal parameter information PC2 may be simply referred to as the "second internal parameters."

[0025] The external parameter information PC3 is information indicating external parameters that indicate one or both of the position and attitude of the projector 10 relative to the camera 20. Below, the external parameters indicated by the external parameter information PC3 may be simply referred to as "external parameters."

[0026] The first coordinate information DP1 is information indicating a plurality of first coordinates, which are coordinates of a plurality of first pixels in a captured image GG (described later) indicated by the imaging data D1. The first pixels are pixels of the camera 20. The first coordinates are coordinates of the first pixels in a coordinate system of the captured image GG (described later) acquired by the camera 20. The first coordinates are two-dimensional coordinates.

[0027] The second coordinate information DP2 is information indicating a plurality of second coordinates, which are the coordinates of a plurality of second pixels on the display panel 15b of the projector 10. The second pixels are pixels on the display panel 15b. The second coordinates are the coordinates of the second pixels in the coordinate system of the display panel 15b. The second coordinates are two-dimensional coordinates. A drawn image based on the video data IMG is drawn on the display panel 15b, and the drawn image is projected as a projection image G via a projection lens. The pixels that make up the drawn image are the second pixels.

[0028] The third coordinate information DP3 is information indicating a plurality of third coordinates, which are three-dimensional coordinates of the projection image G on the projection surface SC. In this specification, the three-dimensional coordinates on the projection surface SC are coordinates in a world coordinate system, which is a three-dimensional coordinate system set in the real space where the projection surface SC is installed or in a virtual three-dimensional space corresponding to the real space. Hereinafter, the three-dimensional coordinate system may be simply referred to as a "three-dimensional coordinate system." The three-dimensional coordinates may also be simply referred to as "three-dimensional coordinates."

[0029] The fifth coordinate information DP5 is information indicating a plurality of fifth coordinates which are the coordinates of a plurality of reference points PR (to be described later) of the projection image G on the display panel 15b of the projector 10. The fifth coordinates are the coordinates of pixels corresponding to the reference points PR (to be described later) in the coordinate system of the display panel 15b.

[0030] The fifth converted coordinate information DP6 is information indicating coordinates obtained by converting the plurality of fifth coordinates indicated by the fifth coordinate information DP5 into three-dimensional coordinates.

[0031] The first marker coordinate information DP8 is information indicating the three-dimensional coordinates of a marker MK, which will be described later, at a first timing.

[0032] The second marker coordinate information DP9 is information indicating the three-dimensional coordinates of a marker MK, which will be described later, at a second timing that is later than the first timing.

[0033] The first correspondence information DC1 is information indicating a first correspondence between a plurality of first coordinates indicated by the first coordinate information DP1 and a plurality of second coordinates indicated by the second coordinate information DP2. That is, the first correspondence indicated by the first correspondence information DC1 is a correspondence between the coordinate system of a captured image GG (described later) acquired by the camera 20 and the coordinate system of the display panel 15b of the projector 10. The first correspondence is a correspondence between a plurality of first coordinates, which are the coordinates of a plurality of first pixels constituting the projected image G in the captured image GG, and a plurality of second coordinates, which are the coordinates of a plurality of second pixels constituting the drawn image. Hereinafter, the first correspondence indicated by the first correspondence information DC1 may be simply referred to as the "first correspondence."

[0034] The second correspondence information DC2 is information indicating a second correspondence between second internal parameters, which are internal parameters of the optical system 15c, and the state of the optical system 15c. The second correspondence information DC2 includes information indicating the second correspondence for each temperature of the optical system 15c. Hereinafter, the second correspondence indicated by the second correspondence information DC2 may be simply referred to as the "second correspondence." The second correspondence information DC2 is obtained using results of determining the second correspondence through experiments or the like, and is stored in advance in the storage device 11.

[0035] The third correspondence information DC3 is information indicating a third correspondence relationship in which a plurality of first coordinates indicated by the first coordinate information DP1 correspond to a plurality of third coordinates indicated by the third coordinate information DP3. That is, the third correspondence relationship indicated by the third correspondence information DC3 is a correspondence relationship between a coordinate system of a captured image GG (described below) acquired by the camera 20 and a three-dimensional coordinate system. Hereinafter, the third correspondence relationship indicated by the third correspondence information DC3 may be simply referred to as the "third correspondence relationship."

[0036] The correction information DC0 is information indicating a correspondence relationship between a plurality of second coordinates indicated by the second coordinate information DP2 and a plurality of third coordinates indicated by the third coordinate information DP3. That is, the correspondence relationship indicated by the correction information DC0 is a correspondence relationship between a three-dimensional coordinate system and the coordinate system of the display panel 15b of the projector 10.

[0037] The processing device 12 has the function of controlling each unit of the projector 10 and the function of processing various data. The processing device 12 includes, for example, a processor such as a CPU (Central Processing Unit). The processing device 12 may be configured with a single processor or multiple processors. Some or all of the functions of the processing device 12 may be realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The processing device 12 may be integrated with at least a part of the image processing circuit 14.

[0038] The communication device 13 is a communication device capable of communicating with various devices, and acquires video data IMG from the terminal device 30 and communicates with the camera 20. For example, the communication device 13 is a wired communication device such as a wired LAN (Local Area Network), USB (Universal Serial Bus), or HDMI (High Definition Multimedia Interface), or a wireless communication device such as LPWA (Low Power Wide Area), wireless LAN including Wi-Fi, or Bluetooth. "HDMI," "Wi-Fi," and "Bluetooth" are all registered trademarks.

[0039] The image processing circuit 14 is a circuit that performs necessary processing on the video data IMG from the communication device 13 and inputs the data to the optical device 15. The image processing circuit 14 has, for example, a frame memory (not shown), and loads the video data IMG into the frame memory, appropriately performs various processes such as resolution conversion, resizing, and distortion correction, and inputs the data to the optical device 15. The aforementioned correction information DC0 is used appropriately in these various processes. Note that the image processing circuit 14 may also perform processes such as OSD (On Screen Display) processing, where necessary, that generates image information for menu display or operation guides, etc., and combines it with the video data IMG.

[0040] The optical device 15 is a device that projects image light onto the projection surface SC, and includes a light source 15a, a display panel 15b, and an optical system 15c.

[0041] The light source 15a includes a light source such as a halogen lamp, a xenon lamp, an ultra-high pressure mercury lamp, an LED (Light Emitting Diode), or a laser light source, and emits red, green, and blue light, respectively. The display panel 15b is an optical modulator including three light modulation elements corresponding to red, green, and blue. Each light modulation element includes, for example, a transmissive liquid crystal panel, a reflective liquid crystal panel, or a DMD (Digital Mirror Device), and generates image light of each color by modulating light of the corresponding color. The image light of each color generated by the display panel 15b is combined by a color combining optical system to become full-color image light. The optical system 15c is a projection optical system including a projection lens and the like that forms and projects the full-color image light from the display panel 15b onto a projection surface SC.

[0042] The operation device 16 is a device that accepts operations from the user. For example, the operation device 16 includes an operation panel and a remote control receiver, not shown. The operation panel is provided on the exterior housing of the projector 10, and outputs a signal based on an operation from the user. The remote control receiver receives an infrared signal from a remote control, not shown, decodes the infrared signal, and outputs a signal based on the operation of the remote control. The operation device 16 may be provided as needed, or may be omitted.

[0043] In the above-described projector 10, the processing device 12 functions as a projection control unit 12a, an imaging control unit 12b, and a correction unit 12c by executing the program PR1 stored in the storage device 11. Therefore, the processing device 12 includes the projection control unit 12a, the imaging control unit 12b, and the correction unit 12c.

[0044] The projection control unit 12a controls the operations of the image processing circuit 14 and the optical device 15. More specifically, the projection control unit 12a controls the operation of the optical device 15 to project the projection image G onto the projection surface SC.

[0045] The imaging control unit 12b controls the operation of the camera 20. More specifically, the imaging control unit 12b causes the camera 20 to capture a projection image G projected onto the projection surface SC, thereby acquiring imaging data D1, and stores the acquired imaging data D1 in the storage device 11.

[0046] The correction unit 12c performs correction to adjust the shape of the projection image G.

[0047] More specifically, the correction unit 12c generates first coordinate information DP1 and second coordinate information DP2 based on the imaging data D1, generates first correspondence information DC1 based on the first coordinate information DP1 and the second coordinate information DP2, and stores the generated first correspondence information DC1 in the storage device 11.

[0048] The correction unit 12c also acquires the state information D2, generates second internal parameter information PC2 based on the second correspondence information DC2 and the state information D2, and stores the generated second internal parameter information PC2 in the storage device 11. Here, the correction unit 12c monitors the state of optical characteristics of the optical system 15c, such as optical distortion, lens shift amount, and throw ratio, and updates the state information D2 when the state changes. Note that if information related to the state information D2 is stored in the storage device 11 or another storage device 11 separately from the state information D2, or if a sensor for detecting the state of the optical system 15c is included, the correction unit 12c may update the state information D2 when performing step S104, which will be described later.

[0049] Furthermore, the correction unit 12c generates external parameter information PC3 based on the first internal parameter information PC1, the first correspondence relationship information DC1, and the second internal parameter information PC2, and stores the generated external parameter information PC3 in the storage device 11. Here, the correction unit 12c estimates the external parameters using a second correspondence relationship according to the temperature of the optical system 15c.

[0050] In addition, the correction unit 12c generates third coordinate information DP3 based on the first internal parameter information PC1, the second internal parameter information PC2, the first correspondence relationship information DC1, and the external parameter information PC3, and stores the generated third coordinate information DP3 in the storage device 11.

[0051] Furthermore, the correction unit 12c generates third correspondence relationship information DC3 based on the first coordinate information DP1 and the third coordinate information DP3, and causes the storage device 11 to store the generated third correspondence relationship information DC3.

[0052] Furthermore, the correction unit 12c generates correction information DC0 based on the first correspondence relationship information DC1 and the third correspondence relationship information DC3, and causes the storage device 11 to store the generated correction information DC0.

[0053] Furthermore, when the optical system 15c of the projector 10 is adjusted, the correction unit 12c updates one or both of the second internal parameter information PC2 and the external parameter information PC3 based on the results of capturing, with the camera 20, the projection image G that is projected after adjusting the optical system 15c.

[0054] Furthermore, the corrector 12c generates fifth coordinate information DP5 based on a plurality of reference points PR in the projection image G, which will be described later, and causes the storage device 11 to store the generated fifth coordinate information DP5.

[0055] Furthermore, the correction unit 12c generates fifth transformed coordinate information DP6 by transforming the fifth coordinate information DP5 using the correction information DC0, and causes the storage device 11 to store the generated fifth transformed coordinate information DP6.

[0056] Furthermore, the correction unit 12c generates first marker coordinate information DP8 based on the fifth coordinate information DP5 and the third correspondence relationship information DC3, and causes the storage device 11 to store the generated first marker coordinate information DP8.

[0057] Furthermore, the corrector 12c generates second marker coordinate information DP9 based on the fifth converted coordinate information DP6 and the third correspondence relationship information DC3, and causes the storage device 11 to store the generated second marker coordinate information DP9.

[0058] Furthermore, the correction unit 12c adjusts the projection image G based on the first marker coordinate information DP8, the second marker coordinate information DP9, the fifth converted coordinate information DP6, the second camera parameter information, and the external parameter information PC3. Here, the correction unit 12c generates the fifth coordinate information DP5 based on the result of receiving an operation from the user to move a plurality of reference points PR, which will be described later, in order to adjust the shape of the projection image G.

[0059] 1-3.Control method 3 is a flowchart showing the flow of the projection method according to the first embodiment. The projection method is performed by causing a processing device 12, which is an example of a "computer," to execute a program PR1 using a system 100 including the above-described projector 10 and camera 20. As shown in FIG. 3, the projection method includes steps S100 to S500.

[0060] More specifically, first, the shape of the projection surface SC is measured in step S100. Details of step S100 will be explained later with reference to FIGS.

[0061] After step S100, in step S200, the projection image G is adjusted. Through this adjustment, fifth coordinate information DP5, fifth converted coordinate information DP6, and first marker coordinate information DP8 are acquired. Details of this acquisition will be described later with reference to FIGS. 7 to 9.

[0062] Then, in step S300, it is determined whether or not readjustment of the projection image G is necessary. For example, if the amount of change in the positional relationship of the projector 10 with respect to the projection surface SC is equal to or greater than a predetermined amount, it is determined that readjustment of the projection image G is necessary. Note that it may be determined that readjustment of the projection image G is necessary based on an instruction from the user, or it may be determined that readjustment of the projection image G is necessary based on the output of a sensor or the like that detects the amount of change in the positional relationship of the projector 10 with respect to the projection surface SC.

[0063] Step S300 is repeated until it is determined that readjustment of the projection image G is necessary (step S300: NO), and if it is determined that readjustment of the projection image G is necessary (step S300: YES), the projection image G is readjusted in step S400, thereby restoring the projection state of the projection image G onto the projection surface SC from the state at the second timing to the state at the first timing. Details of this restoration will be described later with reference to FIGS. 10 to 13.

[0064] After step S400, in step S500, it is determined whether or not an instruction to end the process has been issued. This determination is made based on, for example, whether or not an instruction to end the process has been issued based on an operation by the user.

[0065] If there is no instruction to end the process (step S500: NO), the above-mentioned step S300 is executed. As a result, the above-mentioned steps S300 and S400 are repeated until an instruction to end the process is given. On the other hand, if an instruction to end the process is given (step S500: YES), the process ends.

[0066] 4 is a flowchart showing the flow of measuring the shape of the projection surface SC in Embodiment 1. Step S100 shown in the above-mentioned FIG. 3 includes steps S101 to S109 as shown in FIG.

[0067] More specifically, in step S101, the projector 10 projects the projection image G onto the projection surface SC. This projection is performed by the projection control unit 12a controlling the operation of the projector 10.

[0068] In step S102, a captured image GG, which will be described later, is acquired by capturing the projection image G using the camera 20. This acquisition is performed by the imaging control unit 12b controlling the operation of the camera 20. The captured image GG thus captured is stored in the storage device 11 as imaging data D1.

[0069] In step S103, a first correspondence relationship is generated. This generation is performed by the corrector 12c associating a plurality of first coordinates, which are the coordinates of a plurality of first pixels that form the projection image G in the captured image GG described below, with a plurality of second coordinates, which are the coordinates of a plurality of second pixels that form the projection image G on the display panel 15b of the projector 10. The generated first correspondence relationship is stored in the storage device 11 as first correspondence relationship information DC1.

[0070] In step S104, the state information D2 is acquired. This acquisition is performed by the correction unit 12c detecting the state of the optical system 15c. The acquired state information D2 is stored in the storage device 11.

[0071] In step S105, the second internal parameters are estimated. This estimation is performed by the correction unit 12c based on the second correspondence relationship and the state information D2. The estimated second internal parameters are stored in the storage device 11 as second internal parameter information PC2.

[0072] In step S106, external parameters indicating one or both of the position and attitude of the projector 10 relative to the camera 20 are estimated. This estimation is performed by the correction unit 12c based on the first internal parameters, the first correspondence relationship, and the second internal parameters. The estimated external parameters are stored in the storage device 11 as external parameter information PC3.

[0073] In step S107, a plurality of third coordinates, which are three-dimensional coordinates of the projection image G on the projection surface SC, are calculated by the correction unit 12c based on the first internal parameter, the second internal parameter, the first correspondence relationship, and the external parameter. The calculated plurality of third coordinates are stored in the storage device 11 as third coordinate information DP3.

[0074] In step S108, a third correspondence relationship is generated. This is generated by the corrector 12c associating a plurality of first coordinates with a plurality of third coordinates. The generated third correspondence relationship is stored in the storage device 11 as third correspondence relationship information DC3.

[0075] In step S109, a correspondence relationship is generated in which the plurality of second coordinates indicated by the second coordinate information DP2 correspond to the plurality of third coordinates indicated by the third coordinate information DP3. This is generated by the correction unit 12c based on the first correspondence relationship indicated by the first correspondence relationship information DC1 and the third correspondence relationship indicated by the third correspondence relationship information DC3. The generated correspondence relationship is stored in the storage device 11 as correction information DC0.

[0076] The above outline of steps S101 to S110 will be described in detail below with reference to FIGS.

[0077] Fig. 5 is a diagram illustrating the relationship between the coordinates of the projected image G on the display panel 15b and the coordinates of the captured image GG. Fig. 5 shows the correspondence between a first pixel P1, which is a pixel of the captured image GG represented by the imaging data D1, and a second pixel P0, which is a pixel of the display panel 15b that displays the projected image G.

[0078] In step S101, the projection image G is projected onto the projection surface SC. At this time, the projection image G is displayed on the display panel 15b as shown in the upper part of Fig. 5. In the example shown in Fig. 5, the projection image G is an image in which a plurality of alphabetic characters are arranged. Note that the projection image G is an image for detecting the correspondence of the first pixel P1 to the second pixel P0, and is not limited to the example shown in Fig. 5, and may be a phase shift pattern, a Gray code pattern, a pattern image, etc.

[0079] In step S102, the camera 20 captures the projection image G projected onto the projection surface SC, thereby obtaining a captured image GG indicated by the imaging data D1, as shown in the lower left part of Fig. 5. In addition to the projection surface SC and the projection image G, the captured image GG also includes a plurality of markers MK, which will be described later.

[0080] In step S103, first, first coordinate information DP1 is generated based on a plurality of first coordinates, which are the coordinates of a plurality of first pixels P1 in the coordinate system of the captured image GG, and second coordinate information DP2 is generated based on a plurality of second coordinates, which are the coordinates of a plurality of second pixels on the display panel 15b. Then, in step S103, first correspondence information indicating a first correspondence relationship in which the plurality of first coordinates are associated with the plurality of second coordinates is generated based on the first coordinate information DP1 and the second coordinate information DP2.

[0081] Here, the plurality of first pixels P1 and the plurality of second pixels P0 correspond to each other. Such correspondence between the plurality of first pixels P1 and the plurality of second pixels P0 is detected by a known measurement method. Note that, for convenience of explanation, FIG. 5 illustrates one each of the first pixel P1 and the second pixel P0.

[0082] In step S104, the correction unit 12c acquires the state information D2 by reading it from the storage device 11. Then, in step S105, the correction unit 12c estimates the second internal parameters indicated by the second internal parameter information PC2 based on the second correspondence indicated by the second correspondence information DC2 and the state information D2.

[0083] Fig. 6 is a diagram for explaining an example of the second correspondence indicated by the second correspondence information DC2. In the example shown in Fig. 6, the second correspondence information DC2 includes information DC2a, DC2b, and DC2c.

[0084] The information DC2a indicates the relationship between the horizontal lens shift amount of the optical system 15c and the position of the lens center, as shown on the left side of Fig. 6. Therefore, the horizontal lens center position of the optical system 15c can be estimated based on the horizontal lens shift amount indicated by the state information D2 and the relationship indicated by the information DC2a.

[0085] Information DC2b indicates the relationship between the amount of lens shift in the vertical direction of optical system 15c and the position of the lens center, as shown in the center of Fig. 6. Therefore, the position of the lens center in the vertical direction of optical system 15c can be estimated based on the amount of lens shift in the vertical direction indicated by status information D2 and the relationship indicated by information DC2b.

[0086] The information DC2c indicates the relationship between the optical zoom amount and the focal length of the optical system 15c, as shown on the right side of Fig. 6. Therefore, the focal length of the optical system 15c can be estimated based on the optical zoom amount indicated by the state information D2 and the relationship indicated by the information DC2c.

[0087] Here, the optical characteristics of the optical system 15c vary depending on the temperature of the optical system 15c. Therefore, a plurality of pieces of information DC2a, DC2b, and DC2c may be provided, for example, for each predetermined temperature range. When a plurality of sets of information DC2a, DC2b, and DC2c are provided for each predetermined temperature range, the corrector 12c selects and uses one set of information DC2a, DC2b, and DC2c from the plurality of sets of information DC2a, DC2b, and DC2c based on the detection result of a temperature sensor (not shown).

[0088] The information DC2a, DC2b, and DC2c may be in the form of a function such as a linear function, or may be in the form of a look-up table. Note that the information DC2a, DC2b, and DC2c are not limited to the example shown in Fig. 6 and may include, for example, information regarding optical distortion.

[0089] As described above, in step S105, the second internal parameters indicated by the second internal parameter information PC2 are estimated using the state information D2. Note that the final estimation result in step S105 may be the result of estimating the second internal parameters using the state information D2, or may be a value optimized by processing such as minimizing reprojection error using the estimated value as an initial value.

[0090] In step S106, the coordinate system of the captured image GG (described later) acquired by the camera 20 is used as the reference coordinate system, and external parameters indicating one or both of the position and orientation of the projector 10 relative to the camera 20 are estimated based on the first internal parameter, the first correspondence relationship, and the second internal parameter. For this estimation, a known method such as a five-point algorithm in OpenCV or the like is used. Here, the position is represented by, for example, a three-dimensional vector (tx, ty, tz). Furthermore, the orientation is represented by, for example, a three-dimensional vector (rx, ry, rz) or a 3×3 rotation matrix.

[0091] Here, in camera 20, when the coordinates on the image sensor are (u, v) and the normalized coordinates obtained by normalizing these coordinates with the focal length f of the camera are (x, y), these coordinates have the following relationship:

number

[0092] In the above equation, (c u ,c v ) is the optical center coordinate of the camera. The internal parameters of the camera 20 are the optical center coordinate (c u ,c v ) and the focal length f. The internal parameters of the camera 20 may be obtained by multiplying the internal parameter matrix A by other parameters such as lens distortion coefficients.

[0093] In step S107, the acquired second internal parameter information PC2, external parameter information PC3, and first correspondence relationship information DC1 are used to calculate a plurality of three-dimensional coordinates on the projection surface SC by triangulation between the projection surface SC, the camera 20, and the projector 10. The plurality of third coordinates calculated by such triangulation are acquired as third coordinate information DP3.

[0094] After acquiring the third coordinate information DP3 in step S107 in this manner, in step S108, third correspondence information DC3 indicating a third correspondence in which multiple first coordinates are associated with multiple third coordinates is generated based on the first coordinate information DP1 and the third coordinate information DP3.

[0095] After obtaining the first correspondence information DC1 in step S108 in this manner, in step S109, correction information DC0 indicating the correspondence between multiple second coordinates and multiple third coordinates is generated based on the third correspondence information DC3 and the first correspondence information DC1.

[0096] 7 is a flowchart showing the flow of adjusting the projection image G. Step S200 shown in FIG. 3 includes steps S201 to S204, as shown in FIG.

[0097] More specifically, in step S200, first, in step S201, it is determined whether or not an operation has been performed on a plurality of reference points PR in the projection image G, which will be described later. This determination is made by the correction unit 12c based on whether or not an operation by the user to move the plurality of reference points PR has been accepted. If such an operation has been accepted, it is determined that an operation has been performed on the plurality of reference points PR in the projection image G. In this way, in step S201, an operation to move the plurality of reference points PR in order to adjust the shape of the projection image G is accepted from the user.

[0098] Step S201 is repeated until an operation is performed on a plurality of reference points PR in the projection image G (step S201: NO). If an operation is performed on a plurality of reference points PR in the projection image G (step S201: YES), a plurality of fifth coordinates, which are the coordinates of the plurality of reference points PR on the display panel 15b, are identified in step S202. The plurality of fifth coordinates are stored in the storage device 11 as fifth coordinate information DP5. In this way, step S202 identifies a plurality of fifth coordinates after an operation to move the plurality of reference points PR in order to adjust the shape of the projection image G.

[0099] After step S202, in step S203, the plurality of fifth coordinates indicated by the fifth coordinate information DP5 are converted into three-dimensional coordinates. This conversion is performed by the correction unit 12c using the correspondence indicated by the correction information DC0. Through this conversion, fifth converted coordinate information DP6 is generated, and the generated fifth converted coordinate information DP6 is stored in the storage device 11.

[0100] After step S203, in step S204, the three-dimensional coordinates of a marker MK (described later) set on the projection surface SC are acquired. This acquisition is performed by converting the coordinates of the marker MK in the captured image GG into three-dimensional coordinates using the third correspondence indicated by the third correspondence information DC3. The acquired three-dimensional coordinates are stored in the storage device 11 as first marker coordinate information DP8.

[0101] The above outline of steps S201 to S204 will be described in detail below with reference to FIGS.

[0102] Fig. 8 is a diagram illustrating a plurality of reference points PR on the display panel 15b of the projection image G. Fig. 8 illustrates an example in which the plurality of reference points PR are arranged in a grid pattern on the display panel 15b. In the example shown in Fig. 8, the plurality of reference points PR include a plurality of reference points PR arranged along a portion of the display panel 15b that corresponds to the outer edge of the projection surface SC. Note that the arrangement and number of reference points PR on the display panel 15b are not limited to the example shown in Fig. 8 and are arbitrary.

[0103] Each reference point PR can be moved by a user operation as needed. When two or more reference points PR selected by the user are moved among the multiple reference points PR, the portion of the projected image G corresponding to the interval between the multiple reference points PR is deformed in accordance with the change in the interval between the multiple reference points PR. This allows the shape of the projected image G to be adjusted to fit the shape of the projection surface SC.

[0104] In step S201, it is determined whether or not such an operation has been performed. After step S201, in step S202, fifth coordinate information DP5 indicating a plurality of fifth coordinates, which are the coordinates of a plurality of reference points PR on the display panel 15b, is acquired.

[0105] After step S202, in step S203, the fifth coordinate information DP5 is transformed using the correspondence indicated by the correction information DC0, thereby generating fifth transformed coordinate information DP6.

[0106] Fig. 9 is a diagram for explaining the three-dimensional coordinates of the markers MK at timing 1. Fig. 9 shows the above-mentioned multiple reference points PR and multiple markers MK detected based on the captured image GG in a three-dimensional coordinate system set in the real space where the projection surface SC is installed or in a virtual three-dimensional space corresponding to the real space.

[0107] The multiple markers MK are marks set at any positions on the projection surface SC. The shape, position, and number of the markers MK on the projection surface SC are not limited to the example shown in Fig. 9 and are arbitrary. However, if the shape of the markers MK is not a shape that allows the orientation of the projection surface SC to be determined, the markers MK are set at any three or more positions on the projection surface SC.

[0108] In step S204, the coordinates of the marker MK in the captured image GG are detected using known techniques such as a phase shift method or image recognition technology, and then the coordinates are converted using the third correspondence information DC3 to generate first marker coordinate information DP8 indicating the three-dimensional coordinates of the marker MK at the first timing.

[0109] Fig. 10 is a flowchart showing the flow of restoration. Step S400 shown in Fig. 3 includes steps S401 to S405, as shown in Fig. 10.

[0110] Specifically, first, in step S401, it is determined whether or not the optical system 15c has been adjusted. This determination is made based on whether or not the correction unit 12c has performed adjustments such as lens shift of the optical system 15c. If adjustments such as lens shift of the optical system 15c have been performed, it is determined that the optical system 15c has been adjusted.

[0111] If the optical system 15c has been adjusted (step S401: YES), the second internal parameters indicated by the second internal parameter information PC2 are updated in step S402. This update is performed by executing the above-mentioned step S105 again using the result of capturing, by the camera 20, the projection image G that is projected after the optical system 15c has been adjusted. As a result, when the optical system 15c of the projector 10 has been adjusted, the second internal parameters indicated by the second internal parameter information PC2 are updated based on the result of capturing, by the camera 20, the projection image G that is projected after the optical system 15c has been adjusted. The updated second internal parameters are stored in the storage device 11 as the second internal parameter information PC2.

[0112] After step S402, the external parameters indicated by the external parameter information PC3 are updated. This update is performed by executing the above-mentioned step S106 again using the results of capturing, by the camera 20, the projection image G that is projected after adjusting the optical system 15c. As a result, when the optical system 15c of the projector 10 is adjusted, the external parameters indicated by the external parameter information PC3 are updated based on the results of capturing, by the camera 20, the projection image G that is projected after adjusting the optical system 15c. The updated external parameters are stored in the storage device 11 as the external parameter information PC3.

[0113] After step S403, or if the optical system 15c has not been adjusted (step S401: NO), in step S404, the three-dimensional coordinates of the marker MK at a second timing that is later than the first timing are acquired. This acquisition is performed by the corrector 12c by capturing an image of the projection surface SC again using the camera 20, and then performing processing similar to that of step S204 based on the captured image. The acquired three-dimensional coordinates are stored in the storage device 11 as second marker coordinate information DP9.

[0114] After step S404, in step S405, the projection image G is adjusted. This adjustment is performed by the correction unit 12c based on the three-dimensional coordinates of the marker MK at the first timing, the three-dimensional coordinates of the marker MK at the second timing, coordinates obtained by converting the plurality of fifth coordinates indicated by the fifth coordinate information DP5 into three-dimensional coordinates, the first internal parameters indicated by the first internal parameter information PC1, the second internal parameters indicated by the second internal parameter information PC2, and the external parameters indicated by the external parameter information PC3.

[0115] Of the above steps S401 to S405, step S405 will be described in detail below with reference to FIGS.

[0116] Fig. 11 is a diagram for explaining the three-dimensional coordinates of the markers MK at timing 2. Fig. 11 shows a plurality of markers MK detected based on the imaging results in step S404 in a three-dimensional coordinate system set in the real space where the projection surface SC is installed or in a virtual three-dimensional space corresponding to the real space.

[0117] Fig. 12 is a diagram for explaining the three-dimensional coordinates of the markers MK at timing 1 and timing 2. In Fig. 12, in a three-dimensional coordinate system set in the real space where the projection surface SC is installed or in a virtual three-dimensional space corresponding to the real space, the multiple markers MK detected based on the captured image GG are shown as markers MK-1, and the multiple markers MK detected based on the imaging results in step S404 are shown as markers MK-2.

[0118] In step S405, first, a function F that represents the change in position and orientation of the projection surface SC from the first timing to the second timing is calculated based on the coordinates of the multiple markers MK-1 and the coordinates of the multiple markers MK-2. The function F represents a three-dimensional affine transformation consisting of three-dimensional translation, three-dimensional rotation, and scale transformation.

[0119] Fig. 13 is a diagram for explaining the adjustment of the projection image G. In Fig. 13, in a three-dimensional coordinate system set in the real space where the projection surface SC is installed or in a virtual three-dimensional space corresponding to the real space, the multiple reference points PR at the first timing are shown as multiple reference points PR-1, and the multiple reference points PR at the second timing are shown as multiple reference points PR-2.

[0120] In step S405, as described above, after determining the function F that represents the change in position and orientation of the projection surface SC from the first timing to the second timing, the multiple reference points PR-1 are converted to multiple reference points PR-2 using this function F. In this way, the projection image G is adjusted.

[0121] The above projection method includes steps S101 to S106. As a result, the extrinsic parameters are estimated using the second internal parameters estimated based on the second correspondence relationship and the state information D2, so there is no need to prepare a calibration plate for estimating the extrinsic parameters. This makes it possible to provide a projection method that saves the user effort.

[0122] As described above, the projection method of this embodiment includes steps S107 and S108, which make it possible to generate a third correspondence relationship required for various controls using the second correspondence relationship.

[0123] Furthermore, as described above, the external parameters are estimated using the second correspondence relationship according to the temperature of the optical system 15c, which allows the external parameters to be suitably adjusted according to changes in the optical characteristics of the optical system 15c caused by changes in the temperature.

[0124] Furthermore, the projection method of this embodiment includes steps S402 and S403, as described above, which allow the external parameters to be updated in response to changes in the state of the optical system 15c of the projector .

[0125] As described above, the projection method of this embodiment includes steps S202 to S204, steps S404, and step S405. This allows the projection image G at the second timing to be returned to the state of the projection image G at the first timing even if the projection image G at the second timing is displaced on the projection surface SC relative to the projection image G at the first timing.

[0126] 2. Second embodiment A second embodiment of the present disclosure will be described below. In the following exemplary embodiment, for elements whose actions and functions are similar to those of the first embodiment, the reference numerals used in the description of the first embodiment will be used, and detailed descriptions of each element will be omitted as appropriate.

[0127] 14 is a diagram showing an outline of a system 100A used in the projection method according to the second embodiment. The system 100A has the same configuration as the system 100 of the first embodiment, except that it includes a projector 10A instead of the projector 10 of the first embodiment.

[0128] In this embodiment, as shown in Fig. 14, the projection surface SC has a first portion R1 which is a flat surface and a second portion R2 which is a concave curved surface. In the example shown in Fig. 14, the projection surface SC is divided into the first portion R1 and the second portion R2 in the width direction. The first portion R1 is a flat surface similar to the part of the projection surface SC in the width direction in the first embodiment. The second portion R2 is a curved surface that is concavely curved from one end of the first portion R1 in the width direction toward the front side of the figure as it moves in the width direction.

[0129] 15 is a block diagram of a projector 10A used in a system 100A according to the second embodiment. The projector 10A is configured in the same way as the projector 10 of the first embodiment, except that a program PR2 is used instead of the program PR1 of the first embodiment.

[0130] In the projector 10A, the processing device 12 functions as a projection control unit 12a, an imaging control unit 12b, and a correction unit 12d by executing the program PR2 stored in the storage device 11. Therefore, the processing device 12 includes the projection control unit 12a, the imaging control unit 12b, and the correction unit 12d.

[0131] The corrector 12d is similar to the corrector 12c of the first embodiment except that, in estimating the external parameters, the corrector 12d does not use the relationship regarding the second part R2 of the first correspondence relationship but uses the relationship regarding the first part R1.

[0132] 16 is a flowchart showing the flow of measuring the shape of the projection surface SC in the second embodiment. The projection method of this embodiment is similar to the projection method of the first embodiment, except that it includes step S106A instead of step S106 of the first embodiment.

[0133] In step S106A, the external parameters are estimated. This estimation is performed by the correction unit 12d based on the first internal parameters, the first correspondence relationship, and the second internal parameters. However, during this estimation, the relationship regarding the first part R1 of the first correspondence relationship is used, not the relationship regarding the second part R2. This allows the external parameters to be estimated in the same way as in the first embodiment. The estimated external parameters are stored in the storage device 11 as external parameter information PC3.

[0134] Fig. 17 is a diagram for explaining shape measurement of a first portion R1 and a second portion R2 of a projection surface SC. Fig. 17 shows a captured image GG obtained by capturing a projected image G projected onto a projection surface SC having a first portion R1 and a second portion R2.

[0135] 17, the first portion R1 and the second portion R2 are each shown in the captured image GG overlapping the projected image G. Therefore, similar to the first embodiment, in step S107, multiple three-dimensional coordinates in the first portion R1 and the second portion R2 can be calculated by triangulation between the projection surface SC, the camera 20, and the projector 10 using the acquired second internal parameter information PC2, external parameter information PC3, and first correspondence relationship information DC1.

[0136] The second embodiment described above also provides a projection method that can save the user time and effort.

[0137] 3. Variations The above-described embodiments can be modified in various ways. Specific modifications that can be applied to the above-described embodiments are exemplified below. Two or more embodiments arbitrarily selected from the following examples can be combined as appropriate within the scope of not mutually contradictory.

[0138] 3-1. Variation 1 In the above-described embodiment, an example is given in which the processing device 12 of the projector 10 executes the programs PR1 and PR2, but this is not limited to this example, and for example, the processing device of a computer that is communicatively connected to the projector 10 and the camera 20 may execute the programs PR1 and PR2.

[0139] 3-2. Variation 2 In the above-described embodiment, an example is given in which the correspondence indicated by the correction information DC0 is used to adjust the projection image G, but this is not limited to this example. For example, the correspondence may be used to display a uniform grid-like pattern or the like on the projection surface SC, or may be used to reflect a three-dimensional model of the projection surface SC as seen from the camera 20 in three-dimensional image editing software or the like, and then draw a picture on the model, and display how the picture looks when viewed from the projector 10 on a PC monitor or the like, or project it on the projector 10.

[0140] 4. Notes A summary of this disclosure is provided below.

[0141] (Supplementary Note 1) A first aspect, which is a preferred example of the projection method of the present disclosure, includes projecting a drawn image drawn on a display panel of a projector onto a projection surface from the projector as a projected image; acquiring a captured image by capturing the projected image with a camera having a first internal parameter; generating a first correspondence relationship in which a plurality of first coordinates, which are coordinates of a plurality of first pixels constituting the projected image in the captured image, correspond to a plurality of second coordinates, which are coordinates of a plurality of second pixels constituting the drawn image; acquiring status information indicating a state of an optical system of the projector; estimating the second internal parameter based on the status information and a second correspondence relationship in which a second internal parameter, which is an internal parameter of the optical system, corresponds to a state of the optical system; and estimating an external parameter indicating one or both of a position and an attitude of the projector relative to the camera, based on the first internal parameter, the first correspondence relationship, and the second internal parameter.

[0142] In the above aspect, the external parameters are estimated using the second internal parameters estimated based on the second correspondence relationship and the state information, so there is no need to prepare a calibration plate for estimating the external parameters, and therefore a projection method that can save the user effort can be provided.

[0143] (Supplementary Note 2) In a second aspect which is a preferred example of the first aspect, the method includes determining a plurality of third coordinates which are three-dimensional coordinates of the projected image on the projection surface based on the first internal parameters, the second internal parameters, the first correspondence relationship, and the external parameters, and generating a third correspondence relationship in which the plurality of first coordinates and the plurality of third coordinates are associated with each other. In the above aspect, the second correspondence relationship can be used to generate a third correspondence relationship required for various controls.

[0144] (Supplementary Note 3) In the third aspect, which is a preferred example of the first or second aspect, the external parameters are further adjusted in accordance with the temperature of the optical system. In the above aspects, the external parameters can be suitably adjusted in accordance with changes in optical characteristics that accompany temperature changes in the optical system.

[0145] (Supplementary Note 4) In a fourth aspect which is a preferred example of the third aspect, when the optical system is adjusted, the second internal parameters are updated, and the external parameters are updated based on the first internal parameters, the first correspondence relationship, and the updated second internal parameters. In the above aspect, the external parameters can be updated in response to a change in the state of the optical system of the projector.

[0146] (Supplementary Note 5) In a fifth aspect, which is a preferred example of any of the second to fourth aspects, the method includes: identifying a plurality of fifth coordinates, which are coordinates of a plurality of reference points of the projected image on a display panel of the projector; converting the plurality of fifth coordinates into three-dimensional coordinates based on a correspondence relationship between the plurality of second coordinates and the plurality of third coordinates; acquiring the three-dimensional coordinates of the marker at a first timing based on the third correspondence relationship; acquiring the three-dimensional coordinates of the marker at a second timing that is later than the first timing based on the third correspondence relationship; and adjusting the projected image based on the three-dimensional coordinates of the marker at the first timing, the three-dimensional coordinates of the marker at the second timing, the coordinates obtained by converting the plurality of fifth coordinates into three-dimensional coordinates, the first internal parameter, the second internal parameter, and the external parameter. In the above aspect, even if the projected image at the second timing is displaced on the projection surface relative to the projected image at the first timing, it can be restored to the state of the projected image at the first timing.

[0147] (Appendix 6) A sixth aspect, which is a preferred example of the system of the present disclosure, is a system including a projector and a camera having a first internal parameter, wherein the projector projects a projection image from the projector onto a projection surface; acquires a captured image by capturing the projection image with the camera; generates a first correspondence relationship in which a plurality of first coordinates, which are coordinates of a plurality of first pixels constituting the projection image in the captured image, correspond to a plurality of second coordinates, which are coordinates of a plurality of second pixels constituting the projection image on a display panel of the projector; acquires status information indicating a state of an optical system of the projector; estimates the second internal parameter based on the status information and a second correspondence relationship in which a second internal parameter, which is an internal parameter of the optical system, corresponds to a state of the optical system; and estimates an external parameter indicating one or both of a position and an attitude of the projector relative to the camera based on the first internal parameter, the first correspondence relationship, and the second internal parameter.

[0148] In the above aspect, the external parameters are estimated using the second internal parameters estimated based on the second correspondence relationship and the state information, so there is no need to prepare a calibration plate for estimating the external parameters, and therefore a projection method that can save the user effort can be provided.

[0149] (Appendix 7) A seventh aspect, which is a preferred example of the program of the present disclosure, causes a computer to execute the following steps: projecting a projection image from a projector onto a projection surface; capturing the projection image with a camera having first internal parameters to obtain a captured image; generating a first correspondence relationship in which a plurality of first coordinates, which are coordinates of a plurality of first pixels that constitute the projection image in the captured image, correspond to a plurality of second coordinates, which are coordinates of a plurality of second pixels that constitute the projection image on a display panel of the projector; obtaining status information indicating a state of an optical system of the projector; estimating the second internal parameters based on the status information and a second correspondence relationship in which second internal parameters, which are internal parameters of the optical system, correspond to a state of the optical system; and estimating external parameters that indicate one or both of a position and an attitude of the projector relative to the camera based on the first internal parameters, the first correspondence relationship, and the second internal parameters.

[0150] In the above aspect, the external parameters are estimated using the second internal parameters estimated based on the second correspondence relationship and the state information, so there is no need to prepare a calibration plate for estimating the external parameters, and therefore a projection method that can save the user effort can be provided. [Explanation of symbols]

[0151] 10...Projector, 10A...Projector, 11...Storage device, 12...Processing device, 12a...Projection control unit, 12b...Imaging control unit, 12c...Correction unit, 12d...Correction unit, 13...Communication device, 14...Image processing circuit, 15...Optical device, 15a...Light source, 15b...Display panel, 15c...Optical system, 16...Operation device, 20...Camera, 30...Terminal device, 100...System, 100A...System, D1...Imaging data, D2...Status information, DC0...Correction information, DC 1...first correspondence information, DC2...second correspondence information, DC2a...information, DC2b...information, DC2c...information, DC3...third correspondence information, DP1...first coordinate information, DP2...second coordinate information, DP3...third coordinate information, DP5...fifth coordinate information, DP6...fifth converted coordinate information, DP8...first marker coordinate information, DP9...second marker coordinate information, G...projected image, GG...captured image, IMG...video data, MK...marker, MK-1...marker, MK-2...marker marker, P0...second pixel, P1...first pixel, PC1...first internal parameter information, PC2...second internal parameter information, PC3...external parameter information, PR...reference point, PR-1...reference point, PR-2...reference point, PR1...program, PR2...program, R1...first part, R2...second part, RC...area, S100...step, S101...step, S102...step, S103...step, S104...step, S105...step, S 106...step, S106A...step, S107...step, S108...step, S109...step, S110...step, S200...step, S201...step, S202...step, S203...step, S204...step, S300...step, S400...step, S401...step, S402...step, S403...step, S404...step, S405...step, S500...step, SC...projection surface.

Claims

1. projecting an image drawn on a display panel of a projector onto a projection surface from the projector as a projection image; capturing the projection image with a camera having first internal parameters to obtain a captured image; generating a first correspondence relationship in which a plurality of first coordinates, which are coordinates of a plurality of first pixels constituting the projected image in the captured image, and a plurality of second coordinates, which are coordinates of a plurality of second pixels constituting the drawn image, are associated with each other; acquiring status information indicating a status of an optical system of the projector; estimating the second internal parameter based on a second correspondence relationship in which a second internal parameter, which is an internal parameter of the optical system, and a state of the optical system are associated with each other, and based on the state information; estimating an extrinsic parameter indicating one or both of a position and an orientation of the projector relative to the camera based on the first intrinsic parameter, the first correspondence, and the second intrinsic parameter; Projection method.

2. determining a plurality of third coordinates, which are three-dimensional coordinates of the projected image on the projection surface, based on the first internal parameters, the second internal parameters, the first correspondence relationship, and the external parameters; generating a third correspondence relationship in which the plurality of first coordinates are associated with the plurality of third coordinates; The projection method according to claim 1 .

3. adjusting the external parameters in response to a temperature of the optical system. The projection method according to claim 1 .

4. updating the second internal parameters when the optical system is adjusted; updating the external parameters based on the first internal parameters, the first correspondence relationship, and the updated second internal parameters; The projection method according to claim 3 .

5. Identifying a plurality of fifth coordinates which are coordinates of a plurality of reference points of the projection image on the display panel of the projector; converting the plurality of fifth coordinates into three-dimensional coordinates based on a correspondence relationship between the plurality of second coordinates and the plurality of third coordinates; acquiring three-dimensional coordinates of the marker at a first timing based on the third correspondence relationship; acquiring three-dimensional coordinates of the marker at a second timing that is later than the first timing based on the third correspondence relationship; adjusting the projection image based on the three-dimensional coordinates of the marker at the first timing, the three-dimensional coordinates of the marker at the second timing, coordinates obtained by converting the plurality of fifth coordinates into three-dimensional coordinates, the first internal parameter, the second internal parameter, and the external parameter; The projection method according to claim 2 .

6. A projector and a camera having a first intrinsic parameter, The projector includes: projecting a projection image from the projector onto a projection surface; capturing the projection image with the camera to obtain a captured image; generating a first correspondence relationship in which a plurality of first coordinates, which are coordinates of a plurality of first pixels constituting the projection image in the captured image, and a plurality of second coordinates, which are coordinates of a plurality of second pixels constituting the projection image on a display panel of the projector, are associated with each other; acquiring status information indicating a status of an optical system of the projector; estimating the second internal parameter based on a second correspondence relationship in which a second internal parameter, which is an internal parameter of the optical system, and a state of the optical system are associated with each other, and based on the state information; and estimating an extrinsic parameter indicating one or both of a position and an orientation of the projector relative to the camera based on the first intrinsic parameter, the first correspondence, and the second intrinsic parameter. system.

7. Projecting a projection image from a projector onto a projection surface; capturing the projection image with a camera having first internal parameters to obtain a captured image; generating a first correspondence relationship in which a plurality of first coordinates, which are coordinates of a plurality of first pixels constituting the projection image in the captured image, and a plurality of second coordinates, which are coordinates of a plurality of second pixels constituting the projection image on a display panel of the projector, are associated with each other; acquiring status information indicating a status of an optical system of the projector; estimating the second internal parameter based on a second correspondence relationship in which a second internal parameter, which is an internal parameter of the optical system, and a state of the optical system are associated with each other, and based on the state information; and estimating an external parameter indicating one or both of a position and an attitude of the projector relative to the camera based on the first internal parameter, the first correspondence, and the second internal parameter. program.

Images