Method for adjusting projection image, projection system, and program for information processing

Abstract

To solve a problem in which, when external imaging apparatuses are used in place of imaging means built in projectors, a user needs to calibrate a parameter related to the positional relationship between the apparatuses.SOLUTION: A method for adjusting a projection image includes: picking up an image of a projection surface of a plane on which a first pattern image including at least four unit images is projected from a first projection device 11, thereby acquiring a first picked-up image corresponding to a first apparatus comprising a first lens; picking up an image of the projection surface, thereby acquiring a second picked-up image corresponding to a second apparatus comprising a second lens; acquiring a projective transformation matrix indicating conversion from a first coordinate system in the first apparatus into a second coordinate system in the second apparatus based on the first picked-up image and the second picked-up image; and acquiring a plane parameter of the projection surface by using the projective transformation matrix.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a projection image adjustment method, a projection system, and an information processing program. [Background technology]

[0002] When a projector projects a projection image onto a projection surface to display a display image, a technique may be used in which the projector estimates the three-dimensional shape of the projection surface and the position of the projection surface relative to the projector.

[0003] For example, Patent Document 1 discloses a technique for measuring the three-dimensional position of a projection surface for each of a plurality of projectors based on parameters acquired by using a projection means and an image capturing means built into the projector. In this technique, in addition to the internal parameters of the projection optical system and the internal parameters of the image capturing optical system, a parameter corresponding to the base line length of triangulation is stored in advance in a storage device. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-042653 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the technology according to Patent Document 1, as described above, each projector needs to store various parameters in advance. When an external imaging device is used instead of the imaging means built into each projector, the parameter corresponding to the base line length of triangulation changes depending on the arrangement position of the imaging device. However, in the technology according to Patent Document 1, it is difficult for each projector to store the changing parameters in advance. For this reason, the user needs to calibrate the parameters related to the positional relationship between the devices. This may result in a decrease in convenience for the user. [Means for solving the problem]

[0006] A method for adjusting a projection image according to one embodiment of the present invention includes: acquiring a first captured image corresponding to a first device having a first lens by capturing an image of a flat projection surface onto which a first pattern image including at least four unit images is projected from a first projection device; acquiring a second captured image corresponding to a second device having a second lens by capturing an image of the projection surface; acquiring a projection transformation matrix indicating any one of a transformation from a first coordinate system in the first device to a second coordinate system in the second device, a transformation from a third coordinate system in the first projection device to the first coordinate system in the first device, and a transformation from the first coordinate system in the first device to the third coordinate system in the first projector, based on the first captured image and the second captured image; acquiring plane parameters of the projection surface using the projection transformation matrix; and projecting a projection image adjusted based on the plane parameters onto the projection surface from the first projection device, wherein at least one of the first device and the second device is an imaging device.

[0007] A projection system according to one aspect of the present invention includes a first projection device, a first device having a first lens that captures an image of a flat projection surface onto which a first pattern image including at least four unit images is projected from the first projection device to generate a first captured image, and a second device having a second lens that captures an image of the projection surface to generate a second captured image, and the first projection device acquires the first captured image, acquires the second captured image, converts a first coordinate system in the first device to a second coordinate system in the second device based on the first captured image and the second captured image, and performs a conversion from the first captured image to the second coordinate system in the second device based on the first captured image and the second captured image. a fourth acquisition unit that acquires a projective transformation matrix indicating one of a transformation from a third coordinate system in the first device to the first coordinate system in the first device and a transformation from the first coordinate system in the first device to the third coordinate system in the first projector, a fourth acquisition unit that acquires plane parameters of the projection surface using the projective transformation matrix, and a processing device that projects a projection image adjusted based on the plane parameters onto the projection surface from the first projection device, wherein at least one of the first device and the second device is an imaging device.

[0008] An information processing program according to one aspect of the present invention includes: acquiring a first captured image corresponding to a first device having a first lens by capturing an image of a flat projection surface onto which a first pattern image including at least four unit images is projected from a first projection device; acquiring a second captured image corresponding to a second device having a second lens by capturing an image of the projection surface; and converting a first coordinate system in the first device to a second coordinate system in the second device based on the first captured image and the second captured image; to the first coordinate system in the first device, and a transformation from the first coordinate system in the first device to the third coordinate system in the first projection device; acquiring a projective transformation matrix indicating one of the transformations; acquiring plane parameters of the projection surface using the projective transformation matrix; and projecting a projection image adjusted based on the plane parameters from the first projection device onto the projection surface, wherein at least one of the first device and the second device is an imaging device. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing the configuration of a projection system 1. [Diagram 2] FIG. 2 is a block diagram showing the configuration of a first projector 10-1. [Diagram 3] FIG. 2 is an explanatory diagram showing an example of an optical system 110. [Figure 4] FIG. 4 is a functional block diagram showing the functions of a three-dimensional shape calculation unit 122. [Diagram 5] 13 is a flowchart showing a solution selection operation by a plane parameter acquisition unit 122-3B. [Figure 6] FIG. 4 is a functional block diagram showing the functions of a direction acquisition unit 124. [Figure 7] 13 is a diagram showing an example of a panel horizontal central axis direction vector HV1, a panel horizontal central axis direction vector HV2, and a vector AV that is an average of the two. FIG. [Figure 8] FIG. 4 is a functional block diagram showing the functions of an adjustment unit 125. [Figure 9]1 is a diagram showing an example of a projection area AR1 by a first projector 10-1, a projection area AR2 by a second projector 10-2, and a rectangle SQ with a maximum area. [Figure 10] FIG. 4 is an explanatory diagram of the operation of a coordinate value calculation unit 125-5. [Figure 11] FIG. 4 is an explanatory diagram of the operation of a coordinate value calculation unit 125-5. [Figure 12] FIG. 4 is an explanatory diagram of the operation of a coordinate value calculation unit 125-5. [Figure 13] FIG. 4 is a block diagram showing the configuration of a second projector 10-2. [Figure 14] 6 is a flowchart showing the operation of a first projector 10-1. [Figure 15] 6 is a flowchart showing the operation of a first projector 10-1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments for carrying out the present invention will be described with reference to the drawings. However, in each drawing, the dimensions and scale of each part are appropriately different from the actual ones. In addition, since the embodiments described below are preferred specific examples of the present invention, various technically preferable limitations are attached, but the scope of the present invention is not limited to these embodiments unless otherwise specified in the following description to the effect that the present invention is limited.

[0011] 1: First embodiment 1-1: Overall composition FIG. 1 is a block diagram showing a configuration of a projection system 1 according to a first embodiment. The projection system 1 includes a first projector 10-1, a second projector 10-2, a first image pickup device 20-1, and a second image pickup device 20-2. The first projector 10-1 and the second projector 10-2 are communicably connected to each other via a communication network NET. The first image pickup device 20-1 and the first projector 10-1 are communicably connected to each other. Similarly, the second image pickup device 20-2 and the second projector 10-2 are communicably connected to each other. A captured image captured by the second image pickup device 20-2 is output to the first projector 10-1 via the second projector 10-2. Note that the first image pickup device 20-1 may be connected to the communication network NET instead of being directly connected to the first projector 10-1. In this case, a captured image captured by the first image pickup device 20-1 is output to the first projector 10-1 via the communication network NET. Similarly, the second imaging device 20-2 may be connected to a communication network NET instead of being directly connected to the second projector 10-2. In this case, the captured image captured by the second imaging device 20-2 is output to the first projector 10-1 via the communication network NET. Note that the "first imaging device 20-1" is an example of a "first device." Also, the "second imaging device 20-2" is an example of a "second device."

[0012] The first projector 10-1 and the second projector 10-2 display a display image by projecting a projection image onto a projection surface such as a wall surface or a screen. In this embodiment, the first projector 10-1 and the second projector 10-2 execute a tiling display. Specifically, the first projector 10-1 projects a projection image PP1 onto the projection surface PF. The second projector 10-2 projects a projection image PP2 onto the projection surface PF. On the projection surface PF, the projection images PP1 and PP2 partially overlap each other. A single display image DP is displayed in the entire area which is the sum of the area of ​​the projection image PP1 and the area of ​​the projection image PP2. A part of the display image DP is included in the projection image PP1, and another part of the display image DP is included in the projection image PP2. A part of the display image DP included in the projection image PP1 and a part of the display image DP included in the projection image PP2 are partially overlapped with each other, so that a single display image DP is displayed on the projection surface PF.

[0013] In this embodiment, it is assumed that first projector 10-1 and second projector 10-2 are both placed substantially horizontally.

[0014] The first imaging device 20-1 captures the projection surface PF. Similarly, the second imaging device 20-2 captures the projection surface PF. The first projector 10-1 can acquire the three-dimensional shape of the projection surface PF based on the captured image of the projection surface PF captured by the first imaging device 20-1 and the captured image of the projection surface PF captured by the second imaging device 20-2. That is, it can be said that the first imaging device 20-1 and the second imaging device 20-2 measure the three-dimensional shape of the projection surface PF as one sensor 20. The first imaging device 20-1 includes a "first lens" as the above-mentioned "first device". Similarly, the second imaging device 20-2 includes a "second lens" as the above-mentioned "second device". Note that the first projector 10-1 may acquire the three-dimensional shape of the projection surface PF by using one stereo camera or one TOF (Time of Flight) camera instead of the first imaging device 20-1 and the second imaging device 20-2.

[0015] The first projector 10-1 adjusts the outer shape of the projection image PP1 projected from the first projector 10-1 and the outer shape of the projection image PP2 projected from the second projector 10-2 using measurement data related to the three-dimensional shape of the projection surface PF. As a result, the display image DP is displayed without being rotated within the projection surface PF. In particular, in this embodiment, when the display image DP is rectangular, the display image DP has one side perpendicular to the vertical direction within the projection surface PF and the other side perpendicular to the horizontal direction within the projection surface PF.

[0016] 1-2: Configuration of the first projector 2 is a block diagram showing the configuration of the first projector 10-1. The first projector 10-1 includes a projection device 11, a processing device 12, a storage device 13, and a communication device 14. The elements of the first projector 10-1 are connected to each other by one or more buses for communicating information. Furthermore, each element of the first projector 10-1 is composed of one or more devices, and some elements of the first projector 10-1 may be omitted.

[0017] The projection device 11 is a device that projects a projection image PP1 onto a projection surface PF such as a wall or a screen. The projection device 11 projects various images under the control of the processing device 12. As described later with reference to FIG. 3, the projection device 11 includes, for example, an illumination device 140, a liquid crystal panel 160, and a projection lens system 183, and modulates light from the illumination device 140 using the liquid crystal panel 160. The projection device 11 also projects the modulated light onto the projection surface PF via the projection lens system 183.

[0018] The processing device 12 is a processor that controls the entire first projector 10-1, and is composed of, for example, a single chip or multiple chips. The processing device 12 is composed of, for example, a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, and a register. Note that some or all of the functions of the processing device 12 may be realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The processing device 12 executes various processes in parallel or sequentially.

[0019] The storage device 13 is a recording medium readable by the processing device 12, and stores a plurality of programs including a control program PR1 executed by the processing device 12. The storage device 13 may be configured with at least one of, for example, a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a Random Access Memory (RAM), etc. The storage device 13 may be called a register, a cache, a main memory, a primary storage device, or the like.

[0020] The communication device 14 is hardware as a transmitting / receiving device for communicating with other devices. The communication device 14 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 14 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 14 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include those that comply with a wired LAN (Local Area Network), IEEE1394, or USB (Universal Serial Bus). Examples of the wireless communication interface include those that comply with a wireless LAN, Bluetooth (registered trademark), or the like.

[0021] 3 is an explanatory diagram showing an example of an optical system 110 provided in the projection device 11. The optical system 110 includes an illumination device 140, a separation optical system 150, three liquid crystal panels 160R, 160G, and 160B, and a projection optical system 180. Hereinafter, the liquid crystal panels 160R, 160G, and 160B may be collectively referred to as liquid crystal panels 160. The liquid crystal panel 160 is an example of a "display panel."

[0022] The illumination device 140 includes a white light source such as a halogen lamp.

[0023] Separation optical system 150 has three mirrors 151, 152, and 155 and dichroic mirrors 153 and 154 therein. Separation optical system 150 separates white light, which is visible light emitted from illumination device 140, into the three primary colors of red, green, and blue. Hereinafter, "red" will be referred to as "R", "green" as "G", and "blue" as "B".

[0024] For example, white light emitted from lighting device 140 is separated into three primary color light components of R wavelength region, G wavelength region, and B wavelength region by mirrors 151, 152, and 155 and dichroic mirrors 153 and 154 arranged inside separation optical system 150. Then, the light in the R wavelength region is guided to liquid crystal panel 160R, the light in the G wavelength region is guided to liquid crystal panel 160G, and the light in the B wavelength region is guided to liquid crystal panel 160B.

[0025] Specifically, dichroic mirror 154 transmits light in the R wavelength range out of the white light and reflects light in the G and B wavelength ranges. Dichroic mirror 153 transmits light in the B wavelength range out of the light in the G and B wavelength ranges reflected by dichroic mirror 154 and reflects light in the G wavelength range.

[0026] Here, the liquid crystal panels 160R, 160G, and 160B are used as spatial light modulators. Each of the liquid crystal panels 160R, 160G, and 160B has, for example, 800 data lines, 600 scanning lines, and pixels arranged in a matrix of 800 horizontal columns and 600 vertical rows. In each pixel, the polarization state of the transmitted light, which is the outgoing light relative to the incident light, is controlled according to the gradation. Note that the numbers of the scanning lines, data lines, and pixels of the liquid crystal panels 160R, 160G, and 160B described above are merely examples, and are not limited to the above examples.

[0027] The projection optical system 180 includes a dichroic prism 181, an optical path shift element 182, and a projection lens system 183. The light modulated by the liquid crystal panels 160R, 160G, and 160B is incident on the dichroic prism 181 from three directions. In this dichroic prism 181, the light in the R wavelength range and the light in the B wavelength range are refracted at 90 degrees, while the light in the G wavelength range travels straight. As a result, images of the primary colors R, G, and B are synthesized.

[0028] The light emitted from the dichroic prism 181 passes through the light path shift element 182 and reaches the projection lens system 183. For example, the light path shift element 182 is disposed between the dichroic prism 181 and the projection lens system 183.

[0029] The projection lens system 183 enlarges and projects the light emitted from the light path shift element 182, specifically, the composite image, onto a projection surface PF such as a screen. The liquid crystal panels 160R, 160G, and 160B receive light corresponding to the primary colors R, G, and B, respectively, via the dichroic mirrors 153 and 154.

[0030] 3 is merely an example. The optical system 110 may include a DMD panel instead of the liquid crystal panel 160, for example.

[0031] 2, the processing device 12 reads out and executes a control program PR1 from the storage device 13, thereby functioning as an acquisition unit 121, a three-dimensional shape calculation unit 122, an on-plane conversion unit 123, a direction acquisition unit 124, an adjustment unit 125, and a projection control unit 126. Note that the control program PR1 may be transmitted from another device, such as a server that manages the first projector 10-1, via the communication network NET.

[0032] The acquisition unit 121 acquires the pattern image and the projection images PP1 and PP2 from the storage device 13. In addition, the acquisition unit 121 acquires, via the communication device 14, from the second projector 10-2, the captured image captured by the second imaging device 20-2.

[0033] The three-dimensional shape calculation unit 122 calculates parameters related to the three-dimensional shape of the projection surface PF as viewed from the first imaging device 20-1. In other words, the three-dimensional shape calculation unit 122 calculates and acquires three-dimensional plane parameters of the projection surface PF for the first imaging device 20-1. Note that the "three-dimensional plane parameters of the projection surface PF" refer to coefficients a, b, and c when the projection surface PF is expressed by the formula ax+by+cz=1 in a three-dimensional coordinate system that is an XYZ coordinate system on the captured image captured by the first imaging device 20-1.

[0034] 4 is a functional block diagram showing functions of the three-dimensional shape calculation unit 122. The three-dimensional shape calculation unit 122 includes a correspondence relationship acquisition unit 122-1, an axial direction detection unit 122-2, and a plane orientation estimation unit 122-3. The correspondence relationship acquisition unit 122-1 includes a first captured image acquisition unit 122-1A and a second captured image acquisition unit 122-1B. The plane orientation estimation unit 122-3 includes a transformation matrix acquisition unit 122-3A and a plane parameter acquisition unit 122-3B. The first captured image acquisition unit 122-1A is an example of a "first acquisition unit." The second captured image acquisition unit 122-1B is an example of a "second acquisition unit."

[0035] The correspondence acquisition unit 122-1 acquires the correspondence between the camera image coordinate system in the first imaging device 20-1 and the panel image coordinate system in the projection device 11 provided in the first projector 10-1, and the correspondence between the camera image coordinate system in the first imaging device 20-1 and the panel image coordinate system in the projection device 11 provided in the second projector 10-2. In this specification, the camera image coordinate system in the first imaging device 20-1 is referred to as the "first camera image coordinate system". Similarly, the camera image coordinate system in the second imaging device 20-2 is referred to as the "second camera image coordinate system". In this specification, the panel image coordinate system in the projection device 11 provided in the first projector 10-1 is referred to as the "first panel image coordinate system". Similarly, the panel image coordinate system in the projection device 11 provided in the second projector 10-2 is referred to as the "second panel image coordinate system". In other words, correspondence relationship acquisition section 122-1 acquires the correspondence relationship between the first camera image coordinate system and the first panel image coordinate system, and the correspondence relationship between the first camera image coordinate system and the second panel image coordinate system.

[0036] More specifically, the first projector 10-1 projects a pattern image acquired by the acquisition unit 121 onto the projection surface PF. Examples of the pattern image include a checkered pattern, a Gaussian dot pattern, and a circle pattern. The first imaging device 20-1 captures the pattern image projected onto the projection surface PF. The first captured image acquisition unit 122-1A included in the correspondence relationship acquisition unit 122-1 acquires a captured image of the pattern image captured by the first imaging device 20-1. The correspondence relationship acquisition unit 122-1 executes pattern detection on the captured image. For example, when the pattern image is a checkered pattern, the correspondence relationship acquisition unit 122-1 acquires the coordinate values ​​of the grid on the checkered pattern. When the pattern image is a Gaussian dot pattern, the correspondence relationship acquisition unit 122-1 acquires the coordinate values ​​of the location where the luminance is maximum. When the pattern image is a circle pattern, the correspondence relationship acquisition unit 122-1 acquires the coordinate values ​​of the center of the circle. Correspondence acquisition unit 122-1 acquires the correspondence between these coordinate values ​​on the captured image and these coordinate values ​​on liquid crystal panel 160 provided in first projector 10-1. That is, correspondence acquisition unit 122-1 acquires the correspondence between these coordinate values ​​in the first camera image coordinate system and these coordinate values ​​in the first panel image coordinate system.

[0037] Similarly, the second projector 10-2 projects a pattern image onto the projection surface PF. The first captured image acquisition unit 122-1A included in the correspondence relationship acquisition unit 122-1 acquires a captured image of the pattern image captured by the first imaging device 20-1. The correspondence relationship acquisition unit 122-1 executes pattern detection on the captured image. The correspondence relationship acquisition unit 122-1 acquires a correspondence relationship between coordinate values ​​in the first camera image coordinate system and coordinate values ​​in the second panel image coordinate system based on the pattern detection.

[0038] Further, the second captured image acquisition section 122-1B included in the correspondence relationship acquisition section 122-1 acquires a captured image of a pattern image similarly captured by the second imaging device 20-2. The correspondence relationship acquisition section 122-1 executes pattern detection on the captured image. Based on the pattern detection, the correspondence relationship acquisition section 122-1 acquires a correspondence relationship between the coordinate values ​​in the second camera image coordinate system and the coordinate values ​​in the first panel image coordinate system, and a correspondence relationship between the coordinate values ​​in the second camera image coordinate system and the coordinate values ​​in the second panel image coordinate system.

[0039] The correspondence acquisition unit 122-1 acquires, for each of the first imaging device 20-1 and the second imaging device 20-2, the correspondence between the coordinate values ​​in the camera image coordinate system and the coordinate values ​​in the panel image coordinate system of the projectors, among all the projectors, whose imaging range includes the display image DP displayed by projecting the projection image PP onto the projection surface PF.

[0040] The axis direction detector 122-2 detects, in the camera image coordinate system, the panel horizontal central axis direction, which is the direction of the axis that corresponds to the horizontal central axis in the panel image coordinate system.

[0041] Specifically, the axial direction detector 122-2 acquires the positions of at least two points on the horizontal central axis, which is a vertical axis passing through the optical center, on the liquid crystal panel 160 provided in the first projector 10-1 in the camera image coordinate system. On the liquid crystal panel 160, the horizontal central axis passes through the intersection of the optical axis of the projection lens system 183 and the liquid crystal panel 160. In addition, the liquid crystal panel 160 has two sides that are both parallel to the horizontal central axis and two sides that are both perpendicular to the horizontal central axis.

[0042] When the axial direction detection unit 122-2 acquires two points on the horizontal central axis, which is a vertical axis passing through the optical center, on the liquid crystal panel 160 provided in the first projector 10-1, the axial direction detection unit 122-2 determines a vector that extends from top to bottom on the horizontal central axis connecting these two corresponding points in the camera image coordinate system and has a length normalized to 1 as the panel horizontal central axis direction vector in the camera image coordinate system. On the other hand, when the axial direction detection unit 122-2 acquires three or more points on the horizontal central axis, which is a vertical axis passing through the optical center, on the liquid crystal panel 160 provided in the first projector 10-1, the axial direction detection unit 122-2 determines a vector that extends from top to bottom on a line obtained by linear approximation using a method such as the least squares method for a point group of these three or more points and has a length normalized to 1 as the panel horizontal central axis direction vector in the camera image coordinate system.

[0043] The plane orientation estimation unit 122-3 estimates the orientation of the projection surface PF relative to the first imaging device 20-1. As described above, the plane orientation estimation unit 122-3 includes a transformation matrix acquisition unit 122-3A and a plane parameter acquisition unit 122-3B. The transformation matrix acquisition unit 122-3A is an example of a "third acquisition unit." The plane parameter acquisition unit 122-3B is an example of a "fourth acquisition unit."

[0044] The transformation matrix acquisition unit 122-3A converts the coordinate values ​​of the corresponding points in the first camera image coordinate system used when the correspondence acquisition unit 122-1 acquires the above correspondence into coordinate values ​​of the first camera normalized coordinate system, which is a normalized coordinate system in the first imaging device 20-1. Here, the "normalized coordinate system" is a coordinate system in an XY plane that passes through a point with a length of 1 in the depth direction from the optical origin on the optical axis of the first imaging device 20-1. In the normalized coordinate system, image distortion caused by the camera lens is removed. The normalized coordinate system has the optical center as the origin on the captured image captured by the first imaging device 20-1. In addition, the transformation matrix acquisition unit 122-3A converts the coordinate values ​​of the corresponding points in the second camera image coordinate system used by the correspondence acquisition unit 122-1 into coordinate values ​​of the second camera normalized coordinate system, which is a normalized coordinate system in the second imaging device 20-2.

[0045] Furthermore, the transformation matrix acquisition unit 122-3A calculates and acquires a projection transformation matrix from the first camera normalized coordinate system to the second camera normalized coordinate system using the coordinate values ​​of the first camera normalized coordinate system and the coordinate values ​​of the second camera normalized coordinate system. If the coordinates of a point in the first camera normalized coordinate system are (x1, y1) and the coordinates of a point corresponding to the point in the second camera normalized coordinate system are (x2, y2), the projection transformation matrix H is expressed by the following formula (1).

number

[0046] In a first captured image captured by the first imaging device 20-1, coordinate values ​​obtained by converting the coordinate values ​​of four pairs of corresponding points corresponding to four unit images into coordinate values ​​in the first camera normalized coordinate system of the first imaging device 20-1 are an example of a "first converted coordinate value group." Also, in a second captured image captured by the second imaging device 20-2, coordinate values ​​obtained by converting the coordinate values ​​of four pairs of corresponding points corresponding to four unit images into coordinate values ​​in the second camera normalized coordinate system of the second imaging device 20-2 are an example of a "second converted coordinate value group."

[0047] The plane parameter acquisition section 122-3B uses the above-mentioned projective transformation matrix H to acquire the plane parameters of the projection plane PF.

[0048] By performing singular value decomposition on the projective transformation matrix H, the position and orientation of the second imaging device 20-2 in three-dimensional coordinates relative to the first imaging device 20-1 and the three-dimensional plane parameters of the projection surface PF relative to the first imaging device 20-1 are calculated. However, according to the singular value decomposition, two sets of solutions are derived as sets of the position and orientation of the second imaging device 20-2 in three-dimensional coordinates relative to the first imaging device 20-1 and the three-dimensional plane parameters of the projection surface PF relative to the first imaging device 20-1. Therefore, the plane parameter acquisition unit 122-3B selects the solution in which the position of the second imaging device 20-2 relative to the first imaging device 20-1 indicated by each of the two solutions is closer to the position included in the layout information indicating the arrangement of the first imaging device 20-1 and the second imaging device 20-2.

[0049] Here, the "layout information" indicates, for example, the vertical or horizontal positional relationship between the first imaging device 20-1 and the second imaging device 20-2. Note that the direction from the first imaging device 20-1 to the second imaging device 20-2 is an example of the "first direction."

[0050] As described above, the first projector 10-1 and the second projector 10-2 are used for tiling, and therefore are installed side by side, either horizontally or vertically. Therefore, the plane parameter acquisition unit 122-3B can acquire the positional relationship between the first projector 10-1 and the second projector 10-2 by comparing the projection center coordinates of the first projector 10-1 and the second projector 10-2 on the captured image. In addition, when the first imaging device 20-1 is attached to the first projector 10-1 and the second imaging device 20-2 is attached to the second projector 10-2, the plane parameter acquisition unit 122-3B can calculate the positional relationship between the first imaging device 20-1 and the second imaging device 20-2 based on the positional relationship between the first projector 10-1 and the second projector 10-2.

[0051] For example, the first imaging device 20-1 captures an image of the projection surface PF onto which a pattern image is projected from the projection device 11 included in the first projector 10-1. The first imaging device 20-1 also captures an image of the projection surface PF onto which a pattern image is projected from the projection device 11 included in the second projector 10-2. Here, the pattern image projected from the projection device 11 included in the first projector 10-1 is an example of a "first pattern image". The pattern image projected from the projection device 11 included in the second projector 10-2 is an example of a "second pattern image". The captured image captured by the first imaging device 20-1 of the projection surface PF onto which the first pattern image is projected is an example of a "first captured image". The captured image captured by the first imaging device 20-1 of the projection surface PF onto which the second pattern image is projected is an example of a "third captured image". The plane parameter acquisition unit 122-3B acquires a first direction representing the direction from the first pattern image to the second pattern image in an imaged image coordinate system that defines a position within the imaged image captured by the first imaging device 20-1, based on the first captured image and the third captured image.

[0052] The layout information may be stored in the storage device 13 of the first projector 10-1. The layout information basically indicates the positional relationship between the first imaging device 20-1 and the second imaging device 20-2. However, when the first imaging device 20-1 is attached to the first projector 10-1 and the second imaging device 20-2 is attached to the second projector 10-2 as described above, the layout information may be layout information indicating the arrangement of the first projector 10-1 and the second projector 10-2. The layout information may be information manually set by the user of the projection system 1. In other words, the first projector 10-1 may receive an operation to specify the first direction from the user of the projection system 1.

[0053] 5 is a flowchart showing the solution selection operation by the plane parameter acquisition unit 122-3B. The plane parameters of the projection plane PF for the first imaging device 20-1 included in the first solution are expressed as (a, b, c)=(aA ,b A ,c A ), and the plane parameters of the projection plane PF for the first imaging device 20-1 included in the second solution are (a, b, c)=(a B ,b B ,c B )

[0054] In step S1, the processing device 12, functioning as the plane parameter acquisition unit 122-3B, determines whether or not the second imaging device 20-2 is located to the right of the first imaging device 20-1 when facing the projection surface PF in the layout information. If the determination result in step S1 is positive, that is, if the second imaging device 20-2 is located to the right of the first imaging device 20-1 (YES in step S1), the processing device 12 executes the process of step S2. If the determination result in step S1 is negative, that is, if the second imaging device 20-2 is located to the left of the first imaging device 20-1 (NO in step S1), the processing device 12 executes the process of step S6.

[0055] In step S2, the processing device 12, functioning as the plane parameter acquisition unit 122-3B, determines whether or not the second imaging device 20-2 is located to the right of the first imaging device 20-1 toward the projection surface PF in the first solution, and whether or not the second imaging device 20-2 is located to the left of the first imaging device 20-1 toward the projection surface PF in the second solution. If the determination result in step S2 is positive, that is, if the second imaging device 20-2 is located to the right of the first imaging device 20-1 toward the projection surface PF in the first solution, and if the second imaging device 20-2 is located to the left of the first imaging device 20-1 toward the projection surface PF in the second solution (YES in step S2), the processing device 12 executes the process of step S3. On the other hand, if the judgment result of step S2 is negative, i.e., in the first solution, the second imaging device 20-2 is to the left of the first imaging device 20-1 toward the projection surface PF, or in the second solution, the second imaging device 20-2 is to the right of the first imaging device 20-1 toward the projection surface PF (NO in step S2), the processing device 12 executes the processing of step S4.

[0056] In step S3, the processing device 12 selects the first solution by functioning as the plane parameter acquisition unit 122-3B. That is, the processing device 12 selects the first solution by determining (a, b, c)=(a A ,b A ,c A ).

[0057] In step S4, the processing device 12, functioning as the plane parameter acquisition unit 122-3B, determines whether or not the second imaging device 20-2 is located to the right of the first imaging device 20-1 toward the projection surface PF in the second solution, and whether or not the second imaging device 20-2 is located to the left of the first imaging device 20-1 toward the projection surface PF in the first solution. If the determination result in step S4 is positive, that is, if the second imaging device 20-2 is located to the right of the first imaging device 20-1 toward the projection surface PF in the second solution, and if the second imaging device 20-2 is located to the left of the first imaging device 20-1 toward the projection surface PF in the first solution (YES in step S4), the processing device 12 executes the process of step S3. On the other hand, if the judgment result of step S4 is negative, i.e., in the second solution, the second imaging device 20-2 is to the left of the first imaging device 20-1 toward the projection surface PF, or in the first solution, the second imaging device 20-2 is to the right of the first imaging device 20-1 toward the projection surface PF (NO in step S4), the processing device 12 executes the processing of step S5.

[0058] In step S5, the processing device 12 functions as the plane parameter acquisition unit 122-3B and determines that the positional relationship between the first imaging device 20-1 and the second imaging device 20-2 is indistinguishable. In this case, the processing device 12 may stop operating.

[0059] In step S6, the processing device 12, functioning as the plane parameter acquisition unit 122-3B, determines whether or not the second imaging device 20-2 is on the left side of the first imaging device 20-1 toward the projection surface PF in the first solution, and whether or not the second imaging device 20-2 is on the right side of the first imaging device 20-1 toward the projection surface PF in the second solution. If the determination result in step S6 is positive, that is, if the second imaging device 20-2 is on the left side of the first imaging device 20-1 toward the projection surface PF in the first solution, and if the second imaging device 20-2 is on the right side of the first imaging device 20-1 toward the projection surface PF in the second solution (YES in step S6), the processing device 12 executes the process of step S7. On the other hand, if the judgment result of step S6 is negative, i.e., in the first solution, the second imaging device 20-2 is to the right of the first imaging device 20-1 toward the projection surface PF, or in the second solution, the second imaging device 20-2 is to the left of the first imaging device 20-1 toward the projection surface PF (NO in step S6), the processing device 12 executes the processing of step S8.

[0060] In step S7, the processing device 12 selects the second solution by functioning as the plane parameter acquisition unit 122-3B. That is, the processing device 12 selects the second solution by determining (a, b, c)=(a B ,b B ,c B ).

[0061] In step S8, the processing device 12, functioning as the plane parameter acquisition unit 122-3B, determines whether or not the second imaging device 20-2 is located to the left of the first imaging device 20-1 toward the projection surface PF in the second solution, and whether or not the second imaging device 20-2 is located to the right of the first imaging device 20-1 toward the projection surface PF in the first solution. If the determination result in step S8 is positive, that is, if the second imaging device 20-2 is located to the left of the first imaging device 20-1 toward the projection surface PF in the second solution, and if the second imaging device 20-2 is located to the right of the first imaging device 20-1 toward the projection surface PF in the first solution (YES in step S8), the processing device 12 executes the process of step S7. On the other hand, if the judgment result of step S8 is negative, i.e., in the second solution, the second imaging device 20-2 is to the right of the first imaging device 20-1 toward the projection surface PF, or in the first solution, the second imaging device 20-2 is to the left of the first imaging device 20-1 toward the projection surface PF (NO in step S8), the processing device 12 executes the processing of step S5.

[0062] In other words, the processing device 12 acquires the plane parameters of the projection surface PF indicated by the first solution when the direction from the first imaging device 20-1 to the second imaging device 20-2 indicated by the first solution coincides with the above-mentioned first direction and the direction from the first imaging device 20-1 to the second imaging device 20-2 indicated by the second solution does not coincide with the above-mentioned first direction.

[0063] 2, the on-plane conversion unit 123 converts the two-dimensional panel horizontal central axis direction vector in the camera image coordinate system detected by the axis direction detection unit 122-2 into a three-dimensional panel horizontal central axis direction vector on the projection surface PF, using the plane parameters of the projection surface PF acquired by the plane orientation estimation unit 122-3. Specifically, the on-plane conversion unit 123 converts the two-dimensional panel horizontal central axis vector related to the first projector 10-1 on the first camera image coordinate system into a two-dimensional panel horizontal central axis direction vector related to the first projector 10-1 in the first camera normalized coordinate system, using the internal parameters of the first imaging device 20-1. This conversion process is the same as the conversion process executed by the plane orientation estimation unit 122-3. Furthermore, the on-plane conversion unit 123 converts the two-dimensional panel horizontal central axis direction vector of the first projector 10-1 into a three-dimensional panel horizontal central axis direction vector of the first projector 10-1 on the projection surface PF using plane parameters of the projection surface PF for the first imaging device 20-1. Specifically, in a three-dimensional coordinate system with the optical center of the first imaging device 20-1 as the origin, when there is a plane that satisfies ax+by+cz=1 and the coordinate value of a point (X, Y, Z) on the plane observed in the first camera normalized coordinate system is (x1, y1), the following formula (2) is established.

number

[0064] By using a similar technique, the on-plane conversion unit 123 converts a two-dimensional panel horizontal central axis direction vector relating to the second projector 10-2 in the first camera image coordinate system into a three-dimensional panel horizontal central axis direction vector relating to the second projector 10-2 on the projection surface PF.

[0065] The direction acquisition unit 124 calculates and acquires vectors in three mutually orthogonal directions on the projection surface PF. Fig. 6 is a functional block diagram showing the function of the direction acquisition unit 124. The direction acquisition unit 124 includes a normal direction acquisition unit 124-1, a vertical direction acquisition unit 124-2, and a horizontal direction acquisition unit 124-3.

[0066] The normal direction acquisition unit 124-1 acquires the normal direction of the projection surface PF by using the plane parameters acquired by the plane parameter acquisition unit 122-3B. As described above, when the plane parameters of the projection surface PF are (a, b, c), the normal vector n (n x ,n y ,n z ) is calculated by the following formula (3).

number

[0067] The vertical direction acquisition unit 124-2 calculates a vector that is the average of the three-dimensional panel horizontal central axis direction vector of the first projector 10-1 on the projection surface PF and the three-dimensional panel horizontal central axis direction vector of the second projector 10-2 on the projection surface PF, which are output from the on-plane conversion unit 123. Fig. 7 is a diagram showing an example of the three-dimensional panel horizontal central axis direction vector HV1 of the first projector 10-1 on the projection surface PF, the three-dimensional panel horizontal central axis direction vector HV2 of the second projector 10-2 on the projection surface PF, and a vector AV that is the average of both.

[0068] Specifically, the vertical direction acquisition unit 124-2 calculates the average element of the elements of the three-dimensional panel horizontal central axis direction vector HV1 associated with the first projector 10-1 on the projection surface PF and the elements of the three-dimensional panel horizontal central axis direction vector HV2 associated with the second projector 10-2 on the projection surface PF. A vector AV having an average element of the elements of both panel horizontal central axis vectors HV is a vertical vector within the projection surface PF. This vertical vector within the projection surface PF is referred to as a "vertical vector" in this specification. This vertical vector is also referred to as v(v x ,v y ,v z The vertical direction acquisition unit 124-2 acquires the vertical direction in the projection plane PF based on the vertical vector v.

[0069] As described above, the first projector 10-1 and the second projector 10-2 are installed substantially horizontally, but the roll rotation components of both projectors 10 are not 0. The vertical direction acquisition unit 124-2 averages these to compensate for the roll rotation as much as possible. Furthermore, when the projection system 1 performs tiling using three or more projectors 10 instead of two, the variation in the roll rotation is more averaged and the roll rotation is more compensated for.

[0070] The horizontal direction acquisition unit 124-3 acquires a horizontal direction that is perpendicular to the normal direction and the vertical direction on the projection surface PF. Specifically, the horizontal direction acquisition unit 124-3 acquires the normal vector n (n x ,n y ,n z ) and the vertical vector v(v x ,v y ,v z ) and normalize the cross product with the horizontal vector h(h x ,h y ,h z )

[0071] In Fig. 2, the adjustment unit 125 adjusts the shape of the projection images PP1 and PP2 including a part of the display image DP so that a rectangular display image DP having one side perpendicular to the vertical direction and the other side perpendicular to the horizontal direction is displayed on the projection surface PF. Fig. 8 is a functional block diagram showing the functions of the adjustment unit 125. The adjustment unit 125 includes a transformation matrix calculation unit 125-1, a projection area detection unit 125-2, a coordinate system conversion unit 125-3, a search unit 125-4, a coordinate value calculation unit 125-5, and a geometric transformation unit 125-6.

[0072] The transformation matrix calculation unit 125-1 calculates a transformation matrix from the first camera coordinate system, which is a three-dimensional coordinate system seen from the first imaging device 20-1, to a three-dimensional coordinate system when the projection surface PF is seen from the front. x ,n y ,n z ), vertical vector v(v x ,v y ,v z ), and the horizontal vector h(h x ,h y ,h z ) is defined by the following formula (4).

number

[0073] The projection area detection unit 125-2 detects the projection area of ​​each projector 10 on the image captured by the first imaging device 20-1. Specifically, the projection area detection unit 125-2 extracts the coordinate values ​​of four lattice points closest to the coordinates corresponding to the four corners of the liquid crystal panel 160 of each projector 10 from the coordinate values ​​of the corresponding point group on the image captured by the first imaging device 20-1 acquired by the correspondence acquisition unit 122-1. The area surrounded by these four lattice points almost coincides with the projection area. Note that the projection area detection unit 125-2 may calculate in advance a projection transformation matrix between the first camera image coordinate system and the panel image coordinate system of each projector 10, and acquire the coordinate values ​​of the four corner points without margins by projecting the coordinate values ​​of the four corner points of the liquid crystal panel 160 onto the first camera image coordinate system.

[0074] The coordinate system conversion unit 125-3 converts the coordinate values ​​of the projection area in the first camera image coordinate system into coordinate values ​​in the projection plane coordinate system. Specifically, the coordinate system conversion unit 125-3 converts the coordinate values ​​of the four corner points of the projection area in the first camera image coordinate system into coordinate values ​​of the four corner points of the projection area in the first camera normalized coordinate system by using the internal parameters of the first imaging device 20-1. This conversion process is the same as the conversion process executed by the plane orientation estimation unit 122-3. Furthermore, the coordinate system conversion unit 125-3 converts the coordinate values ​​of the four corner points of the projection area in the first camera normalized coordinate system into coordinate values ​​of the four corner points of the projection area in the first camera coordinate system by using the plane parameters (a, b, c). This conversion process is the same as the conversion process executed by the on-plane conversion unit 123. Furthermore, the coordinate system conversion unit 125-3 converts the coordinate values ​​of the four corner points of the projection area in the first camera coordinate system into the coordinate values ​​of the four corner points of the projection area in the projection surface coordinate system by using the conversion matrix R. Specifically, if the coordinate values ​​of the four corner points of the projection area in the first camera coordinate system are (X1, Y1, Z1), the coordinate system conversion unit 125-3 converts the coordinate values ​​of the four corner points of the projection area in the three-dimensional projection surface coordinate system (X S ,Y S ,Z S ) is calculated using the following formula (5).

number

[0075] The search unit 125-4 searches for a rectangle with the maximum area inscribed in the entire area, which is the sum of the projection area by the first projector 10-1 and the projection area by the second projector 10-2, on the projection surface PF. FIG. 9 is a diagram showing an example of the projection area AR1 by the first projector 10-1, the projection area AR2 by the second projector 10-2, and the rectangle SQ with the maximum area. The search unit 125-4 may draw a plurality of rectangles SQ in the entire area and select the rectangle SQ with the maximum area among the plurality of rectangles SQ. Alternatively, the search unit 125-4 may determine the rectangle SQ with the maximum area using dynamic programming. The aspect ratio of the rectangle SQ searched for at this time may be an aspect ratio designated in advance by the user. Alternatively, if there is no particular designation, the search unit 125-4 may determine the rectangle SQ with the maximum area in the entire area regardless of the aspect ratio. Searching unit 125-4 stores the coordinates of the four corners of rectangle SQ determined by the above method in the two-dimensional projection surface coordinate system as a corrected connected area in the projection surface coordinate system.

[0076] Coordinate value calculation unit 125-5 calculates coordinate values ​​of the four corners of the corrected connected area in the first panel image coordinate system of first projector 10-1 and in the second panel image coordinate system of second projector 10-2, using the coordinate values ​​of the four corners of the corrected connected area stored by search unit 125-4. Figures 10 to 12 are explanatory diagrams of the operation of coordinate value calculation unit 125-5.

[0077] First, the coordinate value calculation unit 125-5 divides the rectangle SQ as the corrected connected region shown in Fig. 9 into two rectangles SQ1 and SQ2 that match the aspect ratios of the liquid crystal panels 160 of the first projector 10-1 and the second projector 10-2 as shown in Fig. 10. At this time, the coordinate value calculation unit 125-5 makes the left side of the rectangle SQ1 match the left side of the rectangle SQ, and the right side of the rectangle SQ2 match the right side of the rectangle SQ. In addition, the coordinate value calculation unit 125-5 sets the coordinates of the four corners of each of the rectangles SQ1 and SQ2 as the corrected four corner coordinates in the projection surface coordinate system.

[0078] Next, coordinate value calculation unit 125-5 obtains the coordinate values ​​of the four corner coordinates of projection area AR1 before correction in the first panel image coordinate system and the coordinate values ​​of the four corner coordinates of projection area AR1' after correction in the projection surface coordinate system. At this time, the coordinate values ​​of the four corner coordinates of projection area AR1 before correction in the first panel image coordinate system can be obtained from the panel resolution of first projector 10-1.

[0079] Next, the coordinate value calculation unit 125-5 calculates a projective transformation matrix H1 based on the correspondence relationship between the coordinate values ​​of the four corners of the projection area AR1 before correction in the first panel image coordinate system and the coordinate values ​​of the four corners of the projection area AR1' after correction in the projection surface coordinate system. The projective transformation matrix H1 is a projective transformation matrix from the projection surface coordinate system to the first panel image coordinate system.

[0080] Finally, as shown in FIG. 11, the coordinate value calculation unit 125-5 uses the projection transformation matrix H1 to project the corrected four corner coordinates of the rectangle SQ1 in the projection surface coordinate system into the first panel image coordinate system, thereby calculating the corrected four corner coordinates of the rectangle SQ1' in the first panel image coordinate system, which is the final output.

[0081] Similarly, the coordinate value calculation unit 125-5 acquires the coordinate values ​​of the four corner coordinates of the projection area AR2 before correction in the second panel image coordinate system and the coordinate values ​​of the four corner coordinates of the projection area AR2' after correction in the projection surface coordinate system. At this time, the coordinate values ​​of the four corner coordinates of the projection area AR2 before correction in the second panel image coordinate system can be obtained from the panel resolution of the second projector 10-2.

[0082] Next, the coordinate value calculation unit 125-5 calculates a projective transformation matrix H2 based on the correspondence relationship between the coordinate values ​​of the four corners of the projection area AR2 before correction in the second panel image coordinate system and the coordinate values ​​of the four corners of the projection area AR2' after correction in the projection surface coordinate system. The projective transformation matrix H2 is a projective transformation matrix from the projection surface coordinate system to the second panel image coordinate system.

[0083] Finally, as shown in Figure 12, the coordinate value calculation unit 125-5 uses the projection transformation matrix H2 to project the corrected four corner coordinates of the rectangle SQ2 in the projection surface coordinate system into the second panel image coordinate system, thereby calculating the corrected four corner coordinates of the rectangle SQ2' in the second panel image coordinate system, which is the final output.

[0084] In FIG. 8, geometric transformation section 125-6 performs geometric transformation on the projected image using the corrected four corner coordinates of rectangle SQ1' and the corrected four corner coordinates of rectangle SQ2' calculated by coordinate value calculation section 125-5.

[0085] 2, the projection control unit 126 causes the projection device 11 to project the above-mentioned pattern image onto the projection surface PF. In addition, the projection control unit 126 outputs the above-mentioned pattern image to the second projector 10-2, and then causes the second projector 10-2 to project the pattern image onto the projection surface PF.

[0086] Furthermore, the projection control unit 126 causes the projection device 11 to project the projection image adjusted by the adjustment unit 125 onto the projection surface PF. Furthermore, the projection control unit 126 outputs the projection image adjusted by the adjustment unit 125 to the second projector 10-2, and then causes the second projector 10-2 to project the projection image onto the projection surface PF. Specifically, the projection control unit 126 causes the projection device 11 to project the projection image corrected to the shape of a rectangle SQ1' shown in FIG. 11 onto the projection surface PF. Similarly, the projection control unit 126 causes the second projector 10-2 to project the projection image corrected to the shape of a rectangle SQ2' shown in FIG. 12 onto the projection surface PF.

[0087] Although not shown, the first projector 10-1 also has other functions that are included in a normal projector.

[0088] 1-3: Configuration of the second projector FIG. 13 is a block diagram showing the configuration of the second projector 10-2. For the sake of simplicity, the following description will use the same reference numerals for components of the second projector 10-2 that are the same as those of the first projector 10-1, and will omit detailed descriptions of their functions. The second projector 10-2 includes a projection device 11, a processing device 12A, a storage device 13A, and a communication device 14. The components of the second projector 10-2 are connected to each other by a single or multiple buses for communicating information. The components of the second projector 10-2 are each composed of a single device or multiple devices, and some components of the second projector 10-2 may be omitted.

[0089] The processing device 12A is a processor that controls the entire second projector 10-2, and is configured, for example, by a single chip or multiple chips. The processing device 12A is configured, for example, by a CPU including an interface with peripheral devices, an arithmetic unit, and a register. Note that some or all of the functions of the processing device 12A may be realized by hardware such as a DSP, an ASIC, a PLD, or an FPGA. The processing device 12A executes various processes in parallel or sequentially.

[0090] The storage device 13A is a recording medium readable by the processing device 12A, and stores a plurality of programs including the control program PR1A executed by the processing device 12A. The storage device 13A may be composed of at least one of, for example, a ROM, an EPROM, an EEPROM, a RAM, etc. The storage device 13A may be called a register, a cache, a main memory, a primary storage device, or the like.

[0091] The processing device 12A functions as an acquisition unit 121A and a projection control unit 126A by reading and executing the control program PR1A from the storage device 13A. Note that the control program PR1A may be transmitted from another device, such as a server that manages the second projector 10-2, via the communication network NET.

[0092] The acquiring unit 121A acquires a pattern image from the first projector 10-1 via the communication device 14. In addition, the acquiring unit 121A acquires a projection image adjusted by the first projector 10-1 from the first projector 10-1 via the communication device 14.

[0093] The projection control unit 126A causes the projection device 11 to project the pattern image acquired by the acquisition unit 121A onto the projection surface PF. In addition, the projection control unit 126A causes the projection device 11 to project the projection image acquired by the acquisition unit 121A and adjusted by the first projector 10-1 onto the projection surface PF.

[0094] 1-4: Operation of the embodiment 14 and 15 are flowcharts showing the operation of the first projector 10-1 according to embodiment 1. Hereinafter, the operation of the first projector 10-1 will be described with reference to FIGS.

[0095] In step S11, the processing device 12 functions as the projection control unit 126. The processing device 12 causes the projection device 11 to project a pattern image onto the projection surface PF. Similarly, the processing device 12 causes the second projector 10-2 to project a pattern image onto the projection surface PF.

[0096] In step S12, the processing device 12 functions as a first captured image acquisition unit 122-1A and a second captured image acquisition unit 122-1B. The processing device 12 acquires a captured image of a pattern image captured by the first imaging device 20-1. The processing device 12 also acquires a captured image of a pattern image captured by the second imaging device 20-2. Furthermore, the processing device 12 functions as a correspondence relationship acquisition unit 122-1. The processing device 12 acquires a correspondence relationship between the first camera image coordinate system and the first panel image coordinate system, a correspondence relationship between the first camera image coordinate system and the second panel image coordinate system, a correspondence relationship between the second camera image coordinate system and the first panel image coordinate system, and a correspondence relationship between the second camera image coordinate system and the second panel image coordinate system.

[0097] In step S13, the processing device 12 functions as an axis direction detection unit 122-2. The processing device 12 detects, in the camera image coordinate system, the panel horizontal central axis direction, which is the direction of the axis corresponding to the horizontal central axis in the panel image coordinate system.

[0098] In step S14, the processing device 12 functions as a plane orientation estimation unit 122-3. The processing device 12 estimates the orientation of the projection surface PF relative to the first imaging device 20-1.

[0099] In step S15, the processing device 12 functions as an on-plane transformation unit 123. The processing device 12 uses the plane parameters of the projection surface PF to transform a two-dimensional panel horizontal central axis direction vector in the camera image coordinate system into a three-dimensional panel horizontal central axis direction vector on the projection surface PF.

[0100] In step S16, the processing device 12 functions as a normal direction acquisition unit 124-1. The processing device 12 acquires the normal direction of the projection surface PF.

[0101] In step S17, the processing device 12 functions as a vertical direction acquisition unit 124-2. The processing device 12 acquires the vertical direction of the projection surface PF.

[0102] In step S18, the processing device 12 functions as a horizontal direction acquisition unit 124-3. The processing device 12 acquires the horizontal direction of the projection surface PF.

[0103] In step S19, the processing device 12 functions as a transformation matrix calculation unit 125-1. The processing device 12 calculates a transformation matrix from the first camera coordinate system, which is a three-dimensional coordinate system seen from the first imaging device 20-1, to a three-dimensional coordinate system when the projection surface PF is seen from the front.

[0104] In step S20, the processing device 12 functions as a projection area detection unit 125-2. The processing device 12 detects the projection area of ​​each projector 10 on the image captured by the first imaging device 20-1.

[0105] In step S21, the processing device 12 functions as the coordinate system conversion unit 125-3. The processing device 12 converts the coordinate values ​​of the projection area in the first camera image coordinate system into coordinate values ​​in the projection surface coordinate system.

[0106] In step S22, the processing device 12 functions as a search unit 125-4. The processing device 12 searches for a rectangle SQ with a maximum area inscribed in the entire area, which is the sum of the projection area AR1 by the first projector 10-1 and the projection area AR2 by the second projector 10-2.

[0107] In step S23, the processing device 12 functions as a coordinate value calculation unit 125-5. The processing device 12 calculates the coordinate values ​​of the four corners of a rectangle SQ1' in the first panel image coordinate system of the first projector 10-1 and a rectangle SQ2' in the second panel image coordinate system of the second projector 10-2, using the coordinate values ​​of the four corners of the corrected connected region stored by the search unit 125-4.

[0108] In step S24, the processing device 12 functions as a geometric transformation unit 125-6. The processing device 12 performs geometric transformation on the projected image using the corrected four corner coordinates of the rectangle SQ1' and the corrected four corner coordinates of the rectangle SQ2'.

[0109] In step S25, the processing device 12 functions as the projection control unit 126. The processing device 12 causes the adjusted projection image to be projected from each of the projection device 11 and the second projector 10-2 onto the projection surface PF.

[0110] 2: Variations The present disclosure is not limited to the above-described embodiments. Specific modified embodiments are exemplified below.

[0111] 2-1: Variation 1 In the above embodiment, the transformation matrix acquisition unit 122-3A calculates and acquires a projective transformation matrix from the first camera normalized coordinate system to the second camera normalized coordinate system. However, the projective transformation matrix calculated and acquired by the transformation matrix acquisition unit 122-3A is not limited to the projective transformation matrix from the first camera normalized coordinate system to the second camera normalized coordinate system.

[0112] For example, the transformation matrix acquisition unit 122-3A may calculate and acquire a projection transformation matrix from the first panel image coordinate system in the projection device 11 provided in the first projector 10-1 to the first camera normalized coordinate system. In this case, the first imaging device 20-1 is an example of the "first device." The projection device 11 provided in the first projector 10-1 is an example of the "first projection device." The first panel image coordinate system is an example of the "third coordinate system." The first camera normalized coordinate system is an example of the "first coordinate system."

[0113] Alternatively, the transformation matrix acquisition unit 122-3A may calculate and acquire a projection transformation matrix from the first camera normalized coordinate system to the first panel image coordinate system in the projection device 11 provided in the first projector 10-1. In this case, the first imaging device 20-1 is an example of the "first device." The projection device 11 provided in the first projector 10-1 is an example of the "first projection device." The first camera normalized coordinate system is an example of the "first coordinate system." The "first panel image coordinate system" is an example of the "third coordinate system."

[0114] Alternatively, the transformation matrix acquisition unit 122-3A may calculate and acquire a projection transformation matrix from the first panel image coordinate system in the projection device 11 provided in the first projector 10-1 to the second panel image coordinate system in the projection device 11 provided in the second projector 10-2. In this case, the projection device 11 provided in the first projector 10-1 is an example of the "first projection device." The projection device 11 provided in the second projector 10-2 is an example of the "second device." The first panel image coordinate system is an example of the "first coordinate system." The second panel image coordinate system is an example of the "second coordinate system."

[0115] Alternatively, transformation matrix acquisition section 122-3A may calculate and acquire a projection transformation matrix from the second panel image coordinate system in projection device 11 provided in second projector 10-2 to the first panel image coordinate system in projection device 11 provided in first projector 10-1.

[0116] Alternatively, the transformation matrix acquisition unit 122-3A may calculate and acquire a projection transformation matrix from the first camera normalized coordinate system to the second panel image coordinate system in the projection device 11 provided in the second projector 10-2. In this case, the first imaging device 20-1 is an example of a "first device." The projection device 11 provided in the second projector 10-2 is an example of a "second device." The first camera normalized coordinate system is an example of a "first coordinate system." The "second panel image coordinate system" is an example of a "second coordinate system."

[0117] Alternatively, the transformation matrix acquisition unit 122-3A may calculate and acquire a projection transformation matrix from the second panel image coordinate system in the projection device 11 of the second projector 10-2 to the first camera normalized coordinate system. In this case, the projection device 11 of the second projector 10-2 is an example of a "first device." The first imaging device 20-1 is an example of a "second device." The "second panel image coordinate system" is an example of a "first coordinate system." The first camera normalized coordinate system is an example of a "second coordinate system."

[0118] Alternatively, the transformation matrix acquisition unit 122-3A may calculate and acquire a projective transformation matrix from the first camera image coordinate system to the second camera image coordinate system. In this case, the first imaging device 20-1 is an example of a "first device". The second imaging device 20-2 is an example of a "second device". The first camera image coordinate system is an example of a "first coordinate system". The second camera image coordinate system is an example of a "second coordinate system". In this case, the transformation from the first camera image coordinate system to the first camera normalized coordinate system and the transformation from the second camera image coordinate system to the second camera normalized coordinate system are not essential operations.

[0119] 2-2: Variation 2 In the above embodiment, the processing device 12 included in the first projector 10-1 includes, as functional blocks, an acquisition unit 121, a three-dimensional shape calculation unit 122, an on-plane conversion unit 123, a direction acquisition unit 124, an adjustment unit 125, and a projection control unit 126. However, an information processing device connected to the communication network NET and separate from the first projector 10-1 may include one or more of these functional blocks. The information processing device may be any of a PC, a smartphone, or a tablet. Alternatively, these functional blocks may be distributed as an application to a terminal device connected to the communication network NET.

[0120] 2-3: Variation 3 In the above embodiment, the projection system 1 includes two projectors 10: the first projector 10-1 and the second projector 10-2. However, the projection system 1 may include any number of projectors 10.

[0121] When the projection system 1 includes only one first projector 10-1, a first direction indicated by a three-dimensional panel horizontal central axis direction vector HV1 associated with the first projector 10-1 on the projection surface PF becomes the vertical direction.

[0122] 2-4: Variation 4 In the above embodiment, the first projector 10-1 and the first imaging device 20-1 are separate from each other. However, these two devices may be realized as a single device housed in the same housing. The same applies to the second projector 10-2 and the second imaging device 20-2.

[0123] 2-5: Variation 5 In the above embodiment, the projection system 1 may use a stereo camera having two imaging devices instead of the first imaging device 20-1 and the second imaging device 20-2. Alternatively, the projection system 1 may use a TOF camera capable of three-dimensional measurement independently instead of the first imaging device 20-1 and the second imaging device 20-2.

[0124] 3: Summary of this disclosure The following is a summary of this disclosure. (Supplementary Note 1) A method for adjusting a projection image comprising: acquiring a first captured image corresponding to a first device having a first lens by capturing an image of a flat projection surface onto which a first pattern image including at least four unit images is projected from a first projection device; acquiring a second captured image corresponding to a second device having a second lens by capturing an image of the projection surface; acquiring a projective transformation matrix indicating any one of a transformation from a first coordinate system in the first device to a second coordinate system in the second device, a transformation from a third coordinate system in the first projection device to the first coordinate system in the first device, and a transformation from the first coordinate system in the first device to the third coordinate system in the first projector, based on the first captured image and the second captured image; acquiring plane parameters of the projection surface using the projective transformation matrix; and projecting a projection image adjusted based on the plane parameters onto the projection surface from the first projection device, wherein at least one of the first device and the second device is an imaging device.

[0125] The above-described method for adjusting a projected image makes it possible to obtain plane parameters of the projection surface using a projective transformation matrix, and to adjust the projected image based on the obtained plane parameters. Therefore, when a user of the projection system 1 uses an external imaging device instead of the imaging means built into each projector, the user does not need to manually calibrate parameters related to the positional relationship between the devices. As a result, it is possible to suppress the possibility of a decrease in convenience for the user.

[0126] (Supplementary Note 2) The method for adjusting a projection image described in Supplementary Note 1, wherein acquiring the projective transformation matrix includes acquiring a first group of converted coordinate values ​​by converting each coordinate value in the first coordinate system, where the at least four unit images are located in the first captured image, into coordinate values ​​in a normalized coordinate system in the first device; acquiring a second group of converted coordinate values ​​by converting each coordinate value in the second coordinate system, where the at least four unit images are located in the second captured image, into coordinate values ​​in the normalized coordinate system in the second device; and acquiring the projective transformation matrix that converts at least four coordinate values ​​included in the first group of converted coordinate values ​​into at least four coordinate values ​​included in the second group of converted coordinate values ​​that have a one-to-one correspondence with the at least four coordinate values.

[0127] By using the above-described method for adjusting a projection image, the first projector 10-1 can obtain a homography matrix between the normalized coordinate system of the first device and the normalized coordinate system of the second device. By decomposing the homography matrix, the first projector 10-1 can obtain plane parameters even in a configuration that uses an imaging device whose external parameters are unknown.

[0128] (Supplementary Note 3) The method for adjusting a projection image of Supplementary Note 1, wherein acquiring the projective transformation matrix includes acquiring a first transformed image by transforming the first captured image into an image in a first normalized coordinate system in the first device, acquiring a second transformed image by transforming the second captured image into an image in a second normalized coordinate system in the second device, acquiring a first group of transformed coordinate values ​​indicating positions in the first normalized coordinate system of the at least four unit images included in the first transformed image, acquiring a second group of transformed coordinate values ​​indicating positions in the second normalized coordinate system of the at least four unit images included in the second transformed image, and acquiring the projective transformation matrix that transforms the at least four coordinate values ​​included in the first group of transformed coordinate values ​​into at least four coordinate values ​​included in the second group of transformed coordinate values ​​that correspond one-to-one to the at least four coordinate values.

[0129] By using the above-described method for adjusting a projection image, the first projector 10-1 can obtain a homography matrix between the normalized coordinate system of the first device and the normalized coordinate system of the second device. By decomposing the homography matrix, the first projector 10-1 can obtain plane parameters even in a configuration that uses an imaging device whose external parameters are unknown.

[0130] (Supplementary Note 4) The method for adjusting a projection image described in any one of Supplementary Note 1 to Supplementary Note 3, further comprising acquiring a first direction representing a direction from the first device to the second device, and acquiring plane parameters of the projection surface comprises acquiring a first solution and a second solution of an equation using the projection transformation matrix, and acquiring the plane parameters of the projection surface indicated by the first solution when the direction from the first device to the second device indicated by the first solution coincides with the first direction and the direction from the first device to the second device indicated by the second solution does not coincide with the first direction.

[0131] By using the above-mentioned method for adjusting a projection image to obtain a first direction representing the direction from the first device to the second device, the first projector 10-1 can obtain optimal plane parameters from the above-mentioned projection transformation matrix.

[0132] (Supplementary Note 5) The method for adjusting a projection image according to Supplementary Note 4, further comprising: externally receiving an operation for designating the first direction.

[0133] By using the above-described method for adjusting a projection image, first projector 10-1 can obtain the positional relationship between the imaging devices based on an input from the user.

[0134] (Supplementary Note 6) The method for adjusting a projection image described in Supplementary Note 4, wherein the first device is a first camera disposed on a housing of a first projector and separate from the first projector, the second device is a second camera disposed on a housing of a second projector and separate from the second projector, the first projector includes the first projection device, and acquiring the first direction includes acquiring a third captured image captured by the first device of the projection surface onto which a second pattern image projected from the second projector is projected, and acquiring the first direction representing a direction from the first pattern image to the second pattern image in an captured image coordinate system that defines a position of an captured image captured by the first device based on the first captured image and the third captured image.

[0135] When the first imaging device 20-1 is attached to the first projector 10-1, it can be assumed that the positional relationship of the projected image corresponds to the positional relationship of the cameras. By using the above-described method for adjusting the projected image, the first projector 10-1 can regard the positional relationship of the projected image as the positional relationship between the cameras based on the captured image obtained by capturing the projected image, and can obtain the first direction.

[0136] (Supplementary Note 7) A method for producing a first captured image includes: a first projection device; a first device having a first lens, which captures an image of a flat projection surface onto which a first pattern image including at least four unit images is projected from the first projection device; and a second device having a second lens, which captures an image of the projection surface to produce a second captured image; obtaining the first captured image, obtaining the second captured image, and converting from a first coordinate system in the first device to a second coordinate system in the second device based on the first captured image and the second captured image, and converting from a third coordinate system in the first projection device to a third captured image. a fourth acquisition unit that acquires plane parameters of the projection surface using the projective transformation matrix; and a processing device that projects a projection image adjusted based on the plane parameters onto the projection surface from the first projection device, wherein at least one of the first device and the second device is an imaging device.

[0137] The above projection system can obtain plane parameters of the projection surface using a projective transformation matrix, and adjust the projected image based on the obtained plane parameters. Therefore, when a user uses an external imaging device instead of the imaging means built into each projector, the user does not need to manually calibrate parameters related to the positional relationship between the devices. As a result, the possibility of a decrease in convenience for the user can be suppressed.

[0138] (Supplementary Note 8) A first captured image corresponding to a first device having a first lens is obtained by capturing an image of a flat projection surface onto which a first pattern image including at least four unit images is projected from a first projection device, and a second captured image corresponding to a second device having a second lens is obtained by capturing an image of the projection surface, and based on the first captured image and the second captured image, conversion from a first coordinate system in the first device to a second coordinate system in the second device, and conversion from a third coordinate system in the first projection device to the second coordinate system in the first device are performed. and acquiring a projection transformation matrix indicating any one of a transformation to a first coordinate system and a transformation from the first coordinate system in the first device to the third coordinate system in the first projection device, acquiring plane parameters of the projection surface using the projection transformation matrix, and projecting a projection image adjusted based on the plane parameters from the first projection device onto the projection surface, wherein at least one of the first device and the second device is an imaging device.

[0139] The information processing program described above can obtain plane parameters of the projection surface using a projective transformation matrix, and adjust the projected image based on the obtained plane parameters. Therefore, when using an external imaging device instead of the imaging means built into each projector, the user of the projection system 1 does not need to manually calibrate parameters related to the positional relationship between the devices. As a result, it is possible to suppress the possibility of a decrease in convenience for the user. [Explanation of symbols]

[0140] 10...projector, 10-1...first projector, 10-2...second projector, 11...projection device, 12, 12A...processing device, 13, 13A...storage device, 14...communication device, 20...imaging device, 20-1...first imaging device, 20-2...second imaging device, 110...optical system, 121, 121A...acquisition unit, 122...3D shape calculation unit, 122-1...correspondence relationship acquisition unit, 122-1A...first captured image acquisition unit, 122-1B...second captured image acquisition unit, 122-2...axial direction detection unit, 122-3...plane orientation estimation unit, 122-3A...transformation matrix acquisition unit, 122-3B...plane parameter acquisition unit, 123...on-plane conversion unit, 124...direction acquisition unit, 124-1...normal direction acquisition unit, 124-2...vertical direction acquisition unit, 124-3... horizontal direction acquisition unit, 125... adjustment unit, 125-1... transformation matrix calculation unit, 125-2... projection area detection unit, 125-3... coordinate system conversion unit, 125-4... search unit, 125-5... coordinate value calculation unit, 125-6... geometric deformation unit, 126, 126A... projection control unit, 140... lighting device, 150... separation optical system, 151, 152, 155... mirror, 153, 154... dichroic mirror, 160... liquid crystal panel, 180... projection optical system, 181... dichroic prism, 182... optical path shift element, 183... projection lens system, AR1, AR2... projection area, HV1, HV2... panel horizontal central axis direction vector, PP1, PP2... projection image, PR1, PR1A... control program, SQ1, SQ2... rectangle

Claims

1. A first pattern image including at least four unit images is projected from a first projection device. a first captured image corresponding to a first device having a first lens by capturing an image of the projection surface of the surface; To obtain and a second captured image corresponding to a second device having a second lens by capturing an image of the projection surface; and Based on the first captured image and the second captured image, a transformation from a coordinate system to a second coordinate system in the second device and a transformation from a first coordinate system to a second coordinate system in the first projection device Transformation from the three coordinate systems to the first coordinate system in the first device and a transformation from the first coordinate system in the first projection device to the third coordinate system in the first projection device; Obtaining a projective transformation matrix that indicates one of the transformations; obtaining plane parameters of the projection surface using the projective transformation matrix; The projection image adjusted based on the plane parameters is projected from the first projection device onto the projection plane. and projecting the Including, at least one of the first device and the second device is an imaging device; A method for adjusting a projected image, comprising:

2. Obtaining the projective transformation matrix includes: The first location where the at least four unit images are located in the first captured image. a first transformer obtained by transforming each coordinate value in the coordinate system into a coordinate value in the normalized coordinate system of the first device; obtaining a group of transformed coordinate values; The second coordinate system where the at least four unit images are located in the second captured image. a second transformer obtained by transforming each coordinate value in the coordinate system into a coordinate value in the normalized coordinate system of the second device; obtaining a group of transformed coordinate values; At least four coordinate values ​​included in the first group of transformed coordinate values ​​are converted into the at least four and converting the coordinate values ​​into at least four coordinate values ​​included in the second converted coordinate value group, which correspond one-to-one to the coordinate values. obtaining the projective transformation matrix that transforms the The method for adjusting a projected image according to claim 1 , comprising:

3. Obtaining the projective transformation matrix includes: The first captured image is transformed into an image in a first normalized coordinate system in the first device. obtaining a first transformed image; The second captured image is transformed into an image in a second normalized coordinate system in the second device. obtaining a second transformed image; the first normalized coordinates of the at least four unit images included in the first transformed image; obtaining a first set of transformed coordinate values ​​indicative of a position in the system; the second normalized coordinates of the at least four unit images included in the second transformed image; obtaining a second set of transformed coordinate values ​​indicative of a position in the system; At least four coordinate values ​​included in the first group of transformed coordinate values ​​are converted into the at least four and converting the coordinate values ​​into at least four coordinate values ​​included in the second converted coordinate value group, which correspond one-to-one to the coordinate values. obtaining the projective transformation matrix that transforms the The method for adjusting a projected image according to claim 1 , comprising:

4. and obtaining a first direction representing a direction from the first device to the second device. Including, Obtaining plane parameters of the projection surface includes: obtaining a first solution and a second solution of the equation using the projective transformation matrix; The direction from the first device to the second device indicated by the first solution is , coincident with the first direction and indicated by the second solution, When the direction toward the device does not match the first direction, obtaining plane parameters of the projection surface as indicated by the first solution. The method for adjusting a projection image according to any one of claims 1 to 3.

5. further comprising receiving an operation to designate the first direction from an external device. The method for adjusting a projected image according to claim 4 .

6. The first device is disposed in a housing of the first projector. It is the first camera, separate from the the second device is disposed in a housing of the second projector; It is a second camera separate from the the first projector includes the first projection device, Obtaining the first direction includes: The projection surface onto which the second pattern image projected from the second projector is projected is acquiring a third captured image captured by the first device; The first device captures an image based on the first captured image and the third captured image. In a captured image coordinate system that defines a position in a captured image, obtaining the first direction representing a direction to a second pattern image; The method for adjusting a projected image according to claim 4 .

7. The plane parameters are three-dimensional plane parameters, 2. The method for adjusting a projected image according to claim 1, wherein:

8. The plane parameters are coefficients of an equation defining the projection surface; 10. The method for adjusting a projection image according to claim 1 or 7, wherein:

9. a first projection device; A first pattern image including at least four unit images is projected from the first projection device. a first lens for capturing an image of a flat projection surface, thereby generating a first captured image; The device, a second device including a second lens that captures an image of the projection surface to generate a second captured image; Place and acquiring the first captured image; acquiring the second captured image; Based on the first captured image and the second captured image, Transformation from one coordinate system to a second coordinate system in the second device and Transformation from a third coordinate system to the first coordinate system in the first device, and a transformation from the first coordinate system in the first projection device to the third coordinate system in the first projection device; Obtaining a projective transformation matrix that indicates one of the transformations; a fourth acquisition unit that acquires plane parameters of the projection surface using the projective transformation matrix; The projection image adjusted based on the plane parameters is projected from the first projection device to the projection surface. projecting the light onto a surface; and a processing unit that executes at least one of the first device and the second device is an imaging device; A projection system comprising:

10. A first pattern image including at least four unit images is projected from a first projection device. a first captured image corresponding to a first device having a first lens by capturing an image of the projection surface of the surface; To obtain and a second captured image corresponding to a second device having a second lens by capturing an image of the projection surface; and Based on the first captured image and the second captured image, a transformation from a coordinate system to a second coordinate system in the second device and a transformation from a first coordinate system to a second coordinate system in the first projection device Transformation from the three coordinate systems to the first coordinate system in the first device and a transformation from the first coordinate system in the first projection device to the third coordinate system in the first projection device; Obtaining a projective transformation matrix that indicates one of the transformations; obtaining plane parameters of the projection surface using the projective transformation matrix; The projection image adjusted based on the plane parameters is projected from the first projection device onto the projection plane. and projecting the on the computer, at least one of the first device and the second device is an imaging device; An information processing program characterized by:

Images