Method for adjusting projection image, projection system, and program for information processing

Abstract

To prevent, when a projection image is projected on a projection surface, rotation of the projection image on the projection surface seen from a projector.SOLUTION: A first display panel 260 has a first side orthogonal to a first axis. A method for adjusting a projection image includes: acquiring the normal direction of a projection surface PF based on calculation data; based on the calculation data, acquiring a first direction parallel to a second axis that corresponds to the first axis and is in a first projection image projected on the projection surface PF; acquiring a second direction orthogonal to the normal direction and the first direction; adjusting the shape of a second projection image including a part of a rectangular first display image having a second side orthogonal to the first direction and a third side orthogonal to the second direction so as to display the first image on the projection surface PF; and causing a first projector 20-1 to project the second projection image on the projection surface PF.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a projection image adjustment method, a projection system, and an information processing program. [Background technology]

[0002] When a projector projects a projection image onto a projection surface to display a display image, the range of the projection image on the projection surface may be distorted into a trapezoid depending on the relative positions of the projector and the projection surface. Conventionally, techniques have been used to correct such trapezoidal distortion.

[0003] For example, Patent Document 1 discloses a technology for correcting the above-mentioned trapezoidal distortion by projecting a distance detection pattern from the projector onto the projection surface, calculating the distance between the projector and the projection surface, and calculating the inclination of the projector relative to the projection surface. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2011-217403 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology disclosed in Patent Document 1 does not take into consideration the rotation of the projection image on the projection surface as seen by the projector when the projection image is projected on the projection surface. Therefore, in the technology disclosed in Patent Document 1, it is possible that the projection image after correction may be displayed as a display image in a state rotated on the projection surface. [Means for solving the problem]

[0006] A method for adjusting a projection image according to one embodiment of the present invention includes obtaining measurement data measuring a three-dimensional shape of a projection surface onto which a first projection image is projected from a first projector having a rectangular first display panel having a first side perpendicular to a first axis, calculating parameters related to the three-dimensional shape based on the obtained measurement data, obtaining a normal direction of the projection surface based on the parameters, obtaining a first direction parallel to a second axis in the first projection image corresponding to the first axis based on the parameters, obtaining a second direction perpendicular to the normal direction and the first direction, adjusting a shape of a second projection image on the projection surface including a portion of a rectangular first display image having a second side perpendicular to the first direction and a third side perpendicular to the second direction, and projecting the second projection image onto the projection surface from the first projector.

[0007] A projection system according to one embodiment of the present invention includes a first projector having a rectangular first display panel having a first side perpendicular to a first axis; a sensor that measures a three-dimensional shape of a projection surface onto which the first projector projects a first projection image; and an information processing device that performs the following operations: calculate parameters related to the three-dimensional shape based on measurement data acquired from the sensor; acquire a normal direction of the projection surface based on the parameters; acquire a first direction parallel to a second axis in the first projection image corresponding to the first axis based on the parameters; acquire a second direction perpendicular to the normal direction and the first direction; adjust a shape of a second projection image on the projection surface, the second projection image including a portion of a rectangular first display image having a second side perpendicular to the first direction and a third side perpendicular to the second direction; and project the second projection image from the first projector onto the projection surface.

[0008] An information processing program according to one embodiment of the present invention causes a computer to perform the following steps: calculate parameters related to a three-dimensional shape based on measurement data acquired from a sensor that measures the three-dimensional shape of a projection surface onto which a first projection image is projected by a first projector having a rectangular first display panel having a first side perpendicular to a first axis; obtain a normal direction of the projection surface based on the parameters; obtain a first direction parallel to a second axis in the first projection image corresponding to the first axis based on the parameters; obtain a second direction perpendicular to the normal direction and the first direction; adjust a shape of a second projection image on the projection surface, the second projection image including a portion of a rectangular first display image having a second side perpendicular to the first direction and a third side perpendicular to the second direction; and project the second projection image from the first projector onto the projection surface. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing the configuration of a projection system 1. [Diagram 2] FIG. 2 is a block diagram showing the configuration of a first projector 20-1. [Diagram 3] FIG. 2 is an explanatory diagram showing an example of an optical system 210. [Figure 4] FIG. 1 is a block diagram showing an example of the configuration of an information processing device 10. [Diagram 5] FIG. 2 is a functional block diagram showing the functions of a three-dimensional shape calculation unit 111. [Figure 6] 13 is a flowchart showing a solution selection operation by plane parameter acquisition unit 111-3B. [Figure 7] FIG. 4 is a functional block diagram showing the functions of a direction acquisition unit 113. [Figure 8] 13 is a diagram showing an example of a panel horizontal central axis direction vector HV1, a panel horizontal central axis direction vector HV2, and a vector AV that is an average of the two. FIG. [Figure 9] FIG. 4 is a functional block diagram showing functions of an adjustment unit 114. [Figure 10]1 is a diagram showing an example of a projection area AR1 by a first projector 20-1, a projection area AR2 by a second projector 20-2, and a rectangle SQ with a maximum area. [Figure 11] FIG. 4 is an explanatory diagram of the operation of a coordinate value calculation unit 114-5. [Figure 12] FIG. 4 is an explanatory diagram of the operation of a coordinate value calculation unit 114-5. [Figure 13] FIG. 4 is an explanatory diagram of the operation of a coordinate value calculation unit 114-5. [Figure 14] 4 is a flowchart showing the operation of the information processing device 10. [Figure 15] 4 is a flowchart showing the operation of the information processing device 10. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments for carrying out the present invention will be described with reference to the drawings. However, in each drawing, the dimensions and scale of each part are appropriately different from the actual ones. In addition, since the embodiments described below are preferred specific examples of the present invention, various technically preferable limitations are attached, but the scope of the present invention is not limited to these embodiments unless otherwise specified in the following description to the effect that the present invention is limited.

[0011] 1: First embodiment 1-1: Overall composition FIG. 1 is a block diagram showing a configuration of a projection system 1 according to a first embodiment. The projection system 1 includes an information processing device 10, a first projector 20-1, a second projector 20-2, a first imaging device 30-1, and a second imaging device 30-2. The information processing device 10, the first projector 20-1, and the second projector 20-2 are communicably connected to each other via a communication network NET. The first imaging device 30-1 and the first projector 20-1 are communicably connected. A captured image captured by the first imaging device 30-1 is output to the information processing device 10 via the first projector 20-1. Similarly, the second imaging device 30-2 and the second projector 20-2 are communicably connected. A captured image captured by the second imaging device 30-2 is output to the information processing device 10 via the second projector 20-2. Note that the first imaging device 30-1 may be connected to the communication network NET instead of being directly connected to the first projector 20-1. In this case, the captured image captured by the first imaging device 30-1 is output to the information processing device 10 via the communication network NET. Similarly, the second imaging device 30-2 may be connected to the communication network NET instead of being directly connected to the second projector 20-2. In this case, the captured image captured by the second imaging device 30-2 is output to the information processing device 10 via the communication network NET.

[0012] The first projector 20-1 and the second projector 20-2 project a projection image onto a projection surface such as a wall surface or a screen, thereby displaying a display image. In this embodiment, the first projector 20-1 and the second projector 20-2 execute a tiling display. Specifically, the first projector 20-1 projects a projection image PP1 onto the projection surface PF. The second projector 20-2 projects a projection image PP2 onto the projection surface PF. On the projection surface PF, the projection images PP1 and PP2 partially overlap each other. A single display image DP is displayed in the entire area, which is the sum of the area of ​​the projection image PP1 and the area of ​​the projection image PP2. A part of the display image DP is included in the projection image PP1, and a part of the display image DP is included in the projection image PP2. A part of the display image DP included in the projection image PP1 and a part of the display image DP included in the projection image PP2 are partially overlapped with each other, thereby displaying a single display image DP on the projection surface PF.

[0013] In this embodiment, it is assumed that the first projector 20-1 and the second projector 20-2 are both placed substantially horizontally.

[0014] The first imaging device 30-1 captures the projection surface PF. Similarly, the second imaging device 30-2 captures the projection surface PF. The information processing device 10 can acquire the three-dimensional shape of the projection surface PF based on the captured image of the projection surface PF captured by the first imaging device 30-1 and the captured image of the projection surface PF captured by the second imaging device 30-2. That is, it can be said that the first imaging device 30-1 and the second imaging device 30-2 measure the three-dimensional shape of the projection surface PF as one sensor 30. Note that the information processing device 10 may acquire the three-dimensional shape of the projection surface PF by using one stereo camera or one TOF (Time of Flight) camera instead of the first imaging device 30-1 and the second imaging device 30-2.

[0015] The information processing device 10 adjusts the outer shape of the projection image PP1 projected from the first projector 20-1 and the outer shape of the projection image PP2 projected from the second projector 20-2 using measurement data related to the three-dimensional shape of the projection surface PF. As a result, the display image DP is displayed without being rotated within the projection surface PF. In particular, in this embodiment, when the display image DP is rectangular, the display image DP has one side perpendicular to the vertical direction within the projection surface PF and the other side perpendicular to the horizontal direction within the projection surface PF.

[0016] 1-2: Projector configuration FIG. 2 is a block diagram showing the configuration of the first projector 20-1. The first projector 20-1 includes a projection device 21, a processing device 22, a storage device 23, and a communication device 24. The elements of the first projector 20-1 are connected to each other by a single or multiple buses for communicating information. Each element of the first projector 20-1 is composed of a single or multiple devices, and some elements of the first projector 20-1 may be omitted. Note that the second projector 20-2 is not shown in the figure because it is configured in the same way as the first projector 20-1.

[0017] The projection device 21 is a device that projects a projection image PP1 acquired from the information processing device 10 by an acquisition unit 221 described later onto a projection surface PF such as a wall or a screen. The projection device 21 projects various images under the control of the processing device 22. As described later with reference to FIG. 3, the projection device 21 includes, for example, an illumination device 240, a liquid crystal panel 260, and a projection lens system 283, and modulates light from the illumination device 240 using the liquid crystal panel 260. The projection device 21 also projects the modulated light onto the projection surface PF via the projection lens system 283.

[0018] The processing device 22 is a processor that controls the entire first projector 20-1, and is composed of, for example, a single chip or multiple chips. The processing device 22 is composed of, for example, a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, and a register. Note that some or all of the functions of the processing device 22 may be realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The processing device 22 executes various processes in parallel or sequentially.

[0019] The storage device 23 is a recording medium readable by the processing device 22, and stores a plurality of programs including the control program PR2 executed by the processing device 22. The storage device 23 may be configured with at least one of, for example, a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a Random Access Memory (RAM), etc. The storage device 23 may be called a register, a cache, a main memory, a primary storage device, or the like.

[0020] The communication device 24 is hardware as a transmitting / receiving device for communicating with other devices. The communication device 24 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 24 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 24 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include those that comply with a wired LAN (Local Area Network), IEEE1394, or USB (Universal Serial Bus). Examples of the wireless communication interface include those that comply with a wireless LAN, Bluetooth (registered trademark), etc.

[0021] FIG. 3 is an explanatory diagram showing an example of the optical system 210 provided in the projection device 21. The optical system 210 includes an illumination device 240, a separation optical system 250, three liquid crystal panels 260R, 260G, and 260B, and a projection optical system 280. Hereinafter, the liquid crystal panels 260R, 260G, and 260B may be collectively referred to as the liquid crystal panel 260. The liquid crystal panel 260 is an example of a "display panel." In particular, the liquid crystal panel 260 provided in the first projector 20-1 is an example of a "first display panel." Moreover, the liquid crystal panel 260 provided in the second projector 20-2 is an example of a "second display panel." Furthermore, the liquid crystal panel 260 as the first display panel is rectangular in shape having sides perpendicular to an axis corresponding to the vertical axis of the liquid crystal panel 260 on the projection surface PF. The "vertical axis of liquid crystal panel 260 as the first display panel" is an example of the "first axis." Moreover, the side perpendicular to the "first axis" is an example of the "first side." Similarly, liquid crystal panel 260 as the second display panel is rectangular having a side perpendicular to the axis corresponding to the vertical axis of liquid crystal panel 260 on projection surface PF. The "vertical axis of liquid crystal panel 260 as the second display panel" is an example of the "third axis." Moreover, the side perpendicular to the "third axis" is an example of the "eleventh side."

[0022] The illumination device 240 includes a white light source such as a halogen lamp.

[0023] The separation optical system 250 has three mirrors 251, 252, and 255 and dichroic mirrors 253 and 254 therein. The separation optical system 250 separates white light, which is visible light emitted from the illumination device 240, into the three primary colors of red, green, and blue. Hereinafter, "red" will be referred to as "R", "green" as "G", and "blue" as "B".

[0024] For example, white light emitted from lighting device 240 is separated into three primary color light components of R wavelength region, G wavelength region, and B wavelength region by mirrors 251, 252, and 255 and dichroic mirrors 253 and 254 arranged inside separation optical system 250. Then, the R wavelength region light is guided to liquid crystal panel 260R, the G wavelength region light is guided to liquid crystal panel 260G, and the B wavelength region light is guided to liquid crystal panel 260B.

[0025] Specifically, the dichroic mirror 254 transmits light in the R wavelength range out of the white light and reflects light in the G and B wavelength ranges. The dichroic mirror 253 transmits light in the B wavelength range out of the light in the G and B wavelength ranges reflected by the dichroic mirror 254 and reflects light in the G wavelength range.

[0026] Here, the liquid crystal panels 260R, 260G, and 260B are used as spatial light modulators. Each of the liquid crystal panels 260R, 260G, and 260B has, for example, 800 data lines, 600 scanning lines, and pixels arranged in a matrix of 800 horizontal columns and 600 vertical rows. In each pixel, the polarization state of the transmitted light, which is the outgoing light relative to the incident light, is controlled according to the gradation. Note that the numbers of the scanning lines, data lines, and pixels of the liquid crystal panels 260R, 260G, and 260B described above are merely examples, and are not limited to the above examples.

[0027] The projection optical system 280 includes a dichroic prism 281, an optical path shift element 282, and a projection lens system 283. The light modulated by the liquid crystal panels 260R, 260G, and 260B is incident on the dichroic prism 281 from three directions. In this dichroic prism 281, the light in the R wavelength range and the light in the B wavelength range are refracted at 90 degrees, while the light in the G wavelength range travels straight. As a result, images of the primary colors R, G, and B are synthesized.

[0028] The light emitted from the dichroic prism 281 passes through the light path shift element 282 and reaches the projection lens system 283. For example, the light path shift element 282 is disposed between the dichroic prism 281 and the projection lens system 283.

[0029] The projection lens system 283 enlarges and projects the light emitted from the light path shift element 282, specifically, the composite image, onto a projection surface PF such as a screen. The liquid crystal panels 260R, 260G, and 260B receive light corresponding to the primary colors R, G, and B, respectively, via the dichroic mirrors 253 and 254.

[0030] 3 is merely an example. Optical system 210 may include, for example, a DMD panel instead of liquid crystal panel 260. In this case, the DMD panel is an example of a "display panel." The DMD panel included in first projector 20-1 is an example of a "first display panel," and the DMD panel included in second projector 20-2 is an example of a "second display panel."

[0031] 2, the processing device 22 reads out and executes a control program PR2 from the storage device 23, thereby functioning as an acquisition unit 221 and a projection control unit 222. Note that the control program PR2 may be transmitted from another device, such as a server that manages the first projector 20-1, via the communication network NET.

[0032] The acquisition unit 221 acquires the projection image PP1 from the information processing device 10 via the communication device 24.

[0033] The projection control unit 222 causes the projection device 21 to project the projection image PP1 acquired by the acquisition unit 221 onto a wall or a screen.

[0034] Although not shown, the first projector 20-1 also has other functions that are included in a normal projector.

[0035] 1-3: Configuration of information processing device 4 is a block diagram showing a configuration example of the information processing device 10. The information processing device 10 is typically a PC, but is not limited to this and may be, for example, a tablet terminal or a smartphone. The information processing device 10 includes a processing device 11, a storage device 12, a display device 13, and a communication device 14. The elements of the information processing device 10 are connected to each other by a single or multiple buses for communicating information.

[0036] The processing device 11 is a processor that controls the entire information processing device 10, and is composed of, for example, one or more chips. The processing device 11 is composed of, for example, a CPU including an interface with peripheral devices, an arithmetic unit, and a register. Note that some or all of the functions of the processing device 11 may be realized by hardware such as a DSP, an ASIC, a PLD, or an FPGA. The processing device 11 executes various processes in parallel or sequentially.

[0037] The storage device 12 is a recording medium that can be read and written by the processing device 11, and stores a plurality of programs including the control program PR1 executed by the processing device 11. The storage device 12 may also store images projected by the first projector 20-1 and the second projector 20-2. The storage device 12 may also store layout information relating to the arrangement of the first projector 20-1 and the second projector 20-2. The storage device 12 may be configured with at least one of, for example, a ROM, an EPROM, an EEPROM, a RAM, and the like. The storage device 12 may also be called a register, a cache, a main memory, a primary storage device, or the like.

[0038] The display device 13 is a device that displays images and text information. The display device 13 displays various images under the control of the processing device 11. For example, various display panels such as a liquid crystal display panel and an organic EL (Electro Luminescence) display panel are suitably used as the display device 13.

[0039] The communication device 14 is hardware as a transmitting / receiving device for communicating with other devices. The communication device 14 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 14 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 14 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include those conforming to wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include those conforming to wireless LAN, Bluetooth (registered trademark), etc.

[0040] The processing device 11 reads out and executes a control program PR1 from the storage device 12, thereby functioning as a three-dimensional shape calculation unit 111, an on-plane conversion unit 112, a direction acquisition unit 113, an adjustment unit 114, and a projection control unit 115. The control program PR1 may be transmitted from another device, such as a server that manages the information processing device 10, via a communication network NET.

[0041] The three-dimensional shape calculation unit 111 calculates parameters related to the three-dimensional shape of the projection surface PF as viewed from the first imaging device 30-1. In other words, the three-dimensional shape calculation unit 111 calculates and acquires three-dimensional plane parameters of the projection surface PF for the first imaging device 30-1. Note that the "three-dimensional plane parameters of the projection surface PF" refer to coefficients a, b, and c when the projection surface PF is expressed by the formula ax+by+cz=1 in a three-dimensional coordinate system that is an xyz coordinate system on the captured image captured by the first imaging device 30-1.

[0042] 5 is a functional block diagram showing functions of the three-dimensional shape calculation unit 111. The three-dimensional shape calculation unit 111 includes a correspondence relationship acquisition unit 111-1, an axial direction detection unit 111-2, and a plane orientation estimation unit 111-3. The correspondence relationship acquisition unit 111-1 includes a first captured image acquisition unit 111-1A and a second captured image acquisition unit 111-1B. The plane orientation estimation unit 111-3 includes a transformation matrix acquisition unit 111-3A and a plane parameter acquisition unit 111-3B. The three-dimensional shape calculation unit 111 is an example of a "first calculation unit."

[0043] The correspondence relationship acquisition unit 111-1 acquires the correspondence relationship between the camera image coordinate system in the first imaging device 30-1 and the panel image coordinate system in the first projector 20-1, and the correspondence relationship between the camera image coordinate system in the first imaging device 30-1 and the panel image coordinate system in the second projector 20-2. In this specification, the camera image coordinate system in the first imaging device 30-1 is referred to as the "first camera image coordinate system". Similarly, the camera image coordinate system in the second imaging device 30-2 is referred to as the "second camera image coordinate system". In this specification, the panel image coordinate system in the first projector 20-1 is referred to as the "first panel image coordinate system". Similarly, the panel image coordinate system in the second projector 20-2 is referred to as the "second panel image coordinate system". In other words, the correspondence relationship acquisition unit 111-1 acquires the correspondence relationship between the first camera image coordinate system and the first panel image coordinate system, and the correspondence relationship between the first camera image coordinate system and the second panel image coordinate system.

[0044] More specifically, the first projector 20-1 projects a pattern image onto the projection surface PF. The pattern image is an example of the "first projection image." Examples of the pattern image include a checkered pattern, a Gaussian dot pattern, and a circle pattern. The first imaging device 30-1 captures the pattern image projected onto the projection surface PF. The first captured image acquisition unit 111-1A included in the correspondence relationship acquisition unit 111-1 acquires a captured image of the pattern image captured by the first imaging device 30-1. The correspondence relationship acquisition unit 111-1 executes pattern detection on the captured image. For example, when the pattern image is a checkered pattern, the correspondence relationship acquisition unit 111-1 acquires the coordinate values ​​of the grid on the checkered pattern. When the pattern image is a Gaussian dot pattern, the correspondence relationship acquisition unit 111-1 acquires the coordinate values ​​of the location where the luminance is maximum. When the pattern image is a circle pattern, the correspondence relationship acquisition unit 111-1 acquires the coordinate values ​​of the center of the circle. The correspondence relationship acquisition unit 111-1 acquires the correspondence relationship between these coordinate values ​​on the captured image and these coordinate values ​​on the liquid crystal panel 260 provided in the first projector 20-1. That is, the correspondence relationship acquisition unit 111-1 acquires the correspondence relationship between these coordinate values ​​in the first camera image coordinate system and these coordinate values ​​in the first panel image coordinate system.

[0045] Similarly, the second projector 20-2 projects a pattern image onto the projection surface PF. The first captured image acquisition unit 111-1A included in the correspondence relationship acquisition unit 111-1 acquires a captured image of the pattern image captured by the first imaging device 30-1. The correspondence relationship acquisition unit 111-1 executes pattern detection on the captured image. The correspondence relationship acquisition unit 111-1 acquires a correspondence relationship between coordinate values ​​in the first camera image coordinate system and coordinate values ​​in the second panel image coordinate system based on the pattern detection.

[0046] Further, the second captured image acquisition section 111-1B included in the correspondence relationship acquisition section 111-1 acquires a captured image of a pattern image captured by the second imaging device 30-2. The correspondence relationship acquisition section 111-1 executes pattern detection on the captured image. Based on the pattern detection, the correspondence relationship acquisition section 111-1 acquires a correspondence relationship between coordinate values ​​in the second camera image coordinate system and coordinate values ​​in the first panel image coordinate system, and a correspondence relationship between coordinate values ​​in the second camera image coordinate system and coordinate values ​​in the second panel image coordinate system.

[0047] The correspondence acquisition unit 111-1 acquires, for each of the first imaging device 30-1 and the second imaging device 30-2, the correspondence between the coordinate values ​​in the camera image coordinate system and the coordinate values ​​in the panel image coordinate system of the projector, among all projectors, whose imaging range includes the display image DP displayed by the projector projecting the projection image PP onto the projection surface PF.

[0048] The axis direction detector 111-2 detects, in the camera image coordinate system, the panel horizontal central axis direction, which is the direction of the axis that corresponds to the horizontal central axis in the panel image coordinate system.

[0049] Specifically, the axial direction detection unit 111-2 acquires where at least two points on the horizontal central axis, which is a vertical axis passing through the optical center, are located on the liquid crystal panel 260 of the first projector 20-1 in the camera image coordinate system. The horizontal central axis passes through the intersection of the optical axis of the projection lens system 283 and the liquid crystal panel 260. The horizontal central axis is an example of the above-mentioned "first axis". The liquid crystal panel 260 has two sides that are both parallel to the horizontal central axis and two sides that are both perpendicular to the horizontal central axis. The liquid crystal panel 260 has a "first side" that is perpendicular to the "first axis", as well as a side that forms a right angle with the "first side". The side that forms a center with the "first side" is an example of a "fourth side". The liquid crystal panel 260 has a side that forms a right angle with the "fourth side". The side that forms a right angle with the "fourth side" is an example of a "fifth side". Furthermore, the liquid crystal panel 260 has a side that forms a right angle with the "fifth side" and a right angle with the "first side". The side that forms a right angle with the "fifth side" and a right angle with the "first side" is an example of the "sixth side". The "first axis" is an axis parallel to the "fourth side" and the "sixth side". Furthermore, in the projection image PP1 shown in FIG. 1, the second axis corresponding to the first axis may be an axis passing through the midpoint of the side ST and the midpoint of the side VU. The direction parallel to the second axis is an example of the "first direction". Moreover, the side ST is an example of the "seventh side". The side TV is an example of the "eighth side". The side VU is an example of the "ninth side". The side US is an example of the "tenth side". In other words, the projection image PP1 as the "first projection image" has a seventh side, an eighth side connected to the seventh side, a ninth side connected to the eighth side, and a tenth side connected to the ninth side and the seventh side. The second axis is an axis passing through the midpoint of the seventh side and the midpoint of the ninth side of the projection image PP1 as the first projection image on the projection surface PF.

[0050] When the axis direction detection unit 111-2 acquires two points on the horizontal central axis, which is a vertical axis passing through the optical center, on the liquid crystal panel 260 provided in the first projector 20-1, the axis direction detection unit 111-2 determines a vector that extends from top to bottom on the horizontal central axis connecting these two corresponding points in the camera image coordinate system and has a length normalized to 1 as the panel horizontal central axis direction vector in the camera image coordinate system. On the other hand, when the axis direction detection unit 111-2 acquires three or more points on the horizontal central axis, which is a vertical axis passing through the optical center, on the liquid crystal panel 260 provided in the first projector 20-1, the axis direction detection unit 111-2 determines a vector that extends from top to bottom on a line obtained by linear approximation using a method such as the least squares method for a point group of these three or more points and has a length normalized to 1 as the panel horizontal central axis direction vector in the camera image coordinate system.

[0051] The plane orientation estimation unit 111-3 estimates the orientation of the projection plane PF relative to the first imaging device 30-1. As described above, the plane orientation estimation unit 111-3 includes the transformation matrix acquisition unit 111-3A and the plane parameter acquisition unit 111-3B.

[0052] The transformation matrix acquisition unit 111-3A transforms the coordinate values ​​of the corresponding points in the first camera image coordinate system used by the correspondence acquisition unit 111-1 into coordinate values ​​of the first camera normalized coordinate system, which is a normalized coordinate system in the first imaging device 30-1. Here, the "normalized coordinate system" is a coordinate system in an XY plane that passes through a point with a length of 1 in the depth direction from the optical origin on the optical axis of the first imaging device 30-1. In the normalized coordinate system, image distortion caused by the camera lens is removed. The normalized coordinate system has the optical center as the origin on the image captured by the first imaging device 30-1. In addition, the transformation matrix acquisition unit 111-3A transforms the coordinate values ​​of the corresponding points in the second camera image coordinate system used by the correspondence acquisition unit 111-1 into coordinate values ​​of the second camera normalized coordinate system, which is a normalized coordinate system in the second imaging device 30-2.

[0053] Furthermore, the transformation matrix acquisition unit 111-3A calculates and acquires a projection transformation matrix from the first camera normalized coordinate system to the second camera normalized coordinate system using the coordinate values ​​of the first camera normalized coordinate system and the coordinate values ​​of the second camera normalized coordinate system. If the coordinates of a point in the first camera normalized coordinate system are (x1, y1) and the coordinates of a point corresponding to the point in the second camera normalized coordinate system are (x2, y2), the projection transformation matrix H is expressed by the following formula (1).

number

[0054] Plane parameter acquisition section 111-3B uses the above-mentioned projective transformation matrix H to acquire the plane parameters of the projection surface PF.

[0055] By performing singular value decomposition on the projective transformation matrix H, the position and orientation of the second imaging device 30-2 in three-dimensional coordinates relative to the first imaging device 30-1 and the three-dimensional plane parameters of the projection surface PF relative to the first imaging device 30-1 are calculated. However, according to the singular value decomposition, two sets of solutions are derived as sets of the position and orientation of the second imaging device 30-2 in three-dimensional coordinates relative to the first imaging device 30-1 and the three-dimensional plane parameters of the projection surface PF relative to the first imaging device 30-1. Therefore, the plane parameter acquisition unit 111-3B selects the solution in which the position of the second imaging device 30-2 relative to the first imaging device 30-1 indicated by each of the two solutions is closer to the position included in the layout information indicating the arrangement of the first imaging device 30-1 and the second imaging device 30-2.

[0056] Here, the "layout information" indicates, for example, the vertical positional relationship or the horizontal positional relationship between the first image capturing device 30-1 and the second image capturing device 30-2. As described above, the first projector 20-1 and the second projector 20-2 are used for tiling, and therefore are installed side by side horizontally or vertically. Therefore, the plane parameter acquisition unit 111-3B can acquire the positional relationship between the first projector 20-1 and the second projector 20-2 by comparing the projection center coordinates of the first projector 20-1 and the second projector 20-2 on the captured image. In addition, when the first image capturing device 30-1 is attached to the first projector 20-1 and the second image capturing device 30-2 is attached to the second projector 20-2, the plane parameter acquisition unit 111-3B can calculate the positional relationship between the first image capturing device 30-1 and the second image capturing device 30-2 based on the positional relationship between the first projector 20-1 and the second projector 20-2.

[0057] The layout information may be layout information stored in the storage device 12 of the information processing device 10. The layout information basically indicates the positional relationship between the first imaging device 30-1 and the second imaging device 30-2. However, as described above, when the first imaging device 30-1 is attached to the first projector 20-1 and the second imaging device 30-2 is attached to the second projector 20-2, the layout information may be layout information indicating the arrangement of the first projector 20-1 and the second projector 20-2. The layout information may be information manually set by a user of the projection system 1.

[0058] 6 is a flowchart showing the solution selection operation by the plane parameter acquisition unit 111-3B. The plane parameters of the projection plane PF for the first image pickup device 30-1 included in the first solution are expressed as (a, b, c)=(a A ,b A ,c A ), and the plane parameters of the projection plane PF for the first imaging device 30-1 included in the second solution are (a, b, c)=(a B ,b B ,cB )

[0059] In step S1, the processing device 11 functions as the plane parameter acquisition unit 111-3B to determine whether or not the second imaging device 30-2 is located to the right of the first imaging device 30-1 when facing the projection surface PF in the layout information. If the determination result in step S1 is positive (step S1 / YES), that is, if the second imaging device 30-2 is located to the right of the first imaging device 30-1, the processing device 11 executes the process of step S2. If the determination result in step S1 is negative, that is, if the second imaging device 30-2 is located to the left of the first imaging device 30-1 (step S1 / NO), the processing device 11 executes the process of step S6.

[0060] In step S2, the processing device 11, functioning as the plane parameter acquisition unit 111-3B, determines whether or not the second imaging device 30-2 is located to the right of the first imaging device 30-1 toward the projection surface PF in the first solution, and whether or not the second imaging device 30-2 is located to the left of the first imaging device 30-1 toward the projection surface PF in the second solution. If the determination result in step S2 is positive (step S2 / YES), that is, if the second imaging device 30-2 is located to the right of the first imaging device 30-1 toward the projection surface PF in the first solution, and if the second imaging device 30-2 is located to the left of the first imaging device 30-1 toward the projection surface PF in the second solution, the processing device 11 executes the process of step S3. On the other hand, if the judgment result of step S2 is negative (step S2 / NO), that is, in the first solution, the second imaging device 30-2 is to the left of the second imaging device 30-2 toward the projection surface PF, or in the second solution, the second imaging device 30-2 is to the right of the first imaging device 30-1 toward the projection surface PF, the processing device 11 executes the processing of step S4.

[0061] In step S3, the processing device 11 selects the first solution by functioning as the plane parameter acquisition unit 111-3B. That is, the processing device 11 selects the first solution by determining (a, b, c)=(a A ,b A ,c A ).

[0062] In step S4, the processing device 11, functioning as the plane parameter acquisition unit 111-3B, determines whether or not the second imaging device 30-2 is located to the right of the first imaging device 30-1 toward the projection surface PF in the second solution, and whether or not the second imaging device 30-2 is located to the left of the first imaging device 30-1 toward the projection surface PF in the first solution. If the determination result in step S4 is positive (step S4 / YES), that is, if the second imaging device 30-2 is located to the right of the first imaging device 30-1 toward the projection surface PF in the second solution, and if the second imaging device 30-2 is located to the left of the first imaging device 30-1 toward the projection surface PF in the first solution, the processing device 11 executes the process of step S3. On the other hand, if the judgment result of step S4 is negative (step S4 / NO), that is, in the second solution, the second imaging device 30-2 is to the left of the first imaging device 30-1 toward the projection surface PF, or in the first solution, the second imaging device 30-2 is to the right of the first imaging device 30-1 toward the projection surface PF, the processing device 11 executes the processing of step S5.

[0063] In step S5, the processing device 11 functions as the plane parameter acquisition unit 111-3B and determines that the positional relationship between the first imaging device 30-1 and the second imaging device 30-2 is indistinguishable. In this case, the processing device 11 may stop operating.

[0064] In step S6, the processing device 11, functioning as the plane parameter acquisition unit 111-3B, judges whether or not the second imaging device 30-2 is on the left side of the first imaging device 30-1 toward the projection surface PF in the first solution, and whether or not the second imaging device 30-2 is on the right side of the first imaging device 30-1 toward the projection surface PF in the second solution. If the judgment result in step S6 is positive (step S6 / YES), that is, if the second imaging device 30-2 is on the left side of the first imaging device 30-1 toward the projection surface PF in the first solution, and if the second imaging device 30-2 is on the right side of the first imaging device 30-1 toward the projection surface PF in the second solution, the processing device 11 executes the process of step S7. On the other hand, if the judgment result of step S6 is negative (step S6 / NO), that is, in the first solution, the second imaging device 30-2 is to the right of the first imaging device 30-1 toward the projection surface PF, or in the second solution, the second imaging device 30-2 is to the left of the first imaging device 30-1 toward the projection surface PF, the processing device 11 executes the processing of step S8.

[0065] In step S7, the processing device 11 selects the second solution by functioning as the plane parameter acquisition unit 111-3B. That is, the processing device 11 selects the second solution by determining (a, b, c)=(a B ,b B ,c B ).

[0066] In step S8, the processing device 11, functioning as the plane parameter acquisition unit 111-3B, determines whether or not the second imaging device 30-2 is located to the left of the first imaging device 30-1 toward the projection surface PF in the second solution, and whether or not the second imaging device 30-2 is located to the right of the first imaging device 30-1 toward the projection surface PF in the first solution. If the determination result in step S8 is positive (step S8 / YES), that is, if the second imaging device 30-2 is located to the left of the first imaging device 30-1 toward the projection surface PF in the second solution, and if the second imaging device 30-2 is located to the right of the first imaging device 30-1 toward the projection surface PF in the first solution, the processing device 11 executes the process of step S7. On the other hand, if the judgment result of step S8 is negative (step S8 / NO), that is, in the second solution, the second imaging device 30-2 is to the right of the first imaging device 30-1 toward the projection surface PF, or in the first solution, the second imaging device 30-2 is to the left of the first imaging device 30-1 toward the projection surface PF, the processing device 11 executes the processing of step S5.

[0067] 4, the on-plane conversion unit 112 converts the two-dimensional panel horizontal central axis direction vector in the camera image coordinate system detected by the axis direction detection unit 111-2 into a three-dimensional panel horizontal central axis direction vector on the projection surface PF, using the plane parameters of the projection surface PF acquired by the plane orientation estimation unit 111-3. Specifically, the on-plane conversion unit 112 converts the two-dimensional panel horizontal central axis vector related to the first projector 20-1 in the first camera image coordinate system into a two-dimensional panel horizontal central axis direction vector related to the first projector 20-1 in the first camera normalized coordinate system, using the internal parameters of the first imaging device 30-1. This conversion process is the same as the conversion process executed by the plane orientation estimation unit 111-3. Furthermore, the on-plane conversion unit 112 converts the two-dimensional panel horizontal central axis direction vector of the first projector 20-1 into a three-dimensional panel horizontal central axis direction vector of the first projector 20-1 on the projection surface PF, using the plane parameters of the projection surface PF for the first imaging device 30-1. Specifically, in a three-dimensional coordinate system with the optical center of the first imaging device 30-1 as the origin, when there is a plane that satisfies ax+by+cz=1 and the coordinate value of a point (X, Y, Z) on the plane observed in the first camera normalized coordinate system is (x1, y1), the following formula (2) is established.

number

[0068] By using a similar technique, the on-plane conversion unit 112 converts a two-dimensional panel horizontal central axis direction vector relating to the second projector 20-2 in the first camera image coordinate system into a three-dimensional panel horizontal central axis direction vector relating to the second projector 20-2 on the projection surface PF.

[0069] The direction acquisition unit 113 calculates and acquires vectors in three mutually orthogonal directions on the projection surface PF. FIG. 7 is a functional block diagram showing the function of the direction acquisition unit 113. The direction acquisition unit 113 includes a normal direction acquisition unit 113-1, a vertical direction acquisition unit 113-2, and a horizontal direction acquisition unit 113-3. The normal direction acquisition unit 113-1 is an example of a "first acquisition unit." The vertical direction acquisition unit 113-2 is an example of a "second acquisition unit." The horizontal direction acquisition unit 113-3 is an example of a "third acquisition unit."

[0070] The normal direction acquisition unit 113-1 acquires the normal direction of the projection surface PF by using the plane parameters acquired by the plane parameter acquisition unit 111-3B. As described above, when the plane parameters of the projection surface PF are (a, b, c), the normal vector n (n x ,n y ,n z ) is calculated by the following formula (3).

number

[0071] The vertical direction acquisition unit 113-2 calculates a vector that is the average of the three-dimensional panel horizontal central axis direction vector of the first projector 20-1 on the projection surface PF and the three-dimensional panel horizontal central axis direction vector of the second projector 20-2 on the projection surface PF, which are output from the on-plane conversion unit 112. Fig. 8 is a diagram showing an example of the three-dimensional panel horizontal central axis direction vector HV1 of the first projector 20-1 on the projection surface PF, the three-dimensional panel horizontal central axis direction vector HV2 of the second projector 20-2 on the projection surface PF, and a vector AV that is the average of both.

[0072] Specifically, the vertical direction acquisition unit 113-2 calculates the average element of the elements of the three-dimensional panel horizontal central axis direction vector HV1 associated with the first projector 20-1 on the projection surface PF and the elements of the three-dimensional panel horizontal central axis direction vector HV2 associated with the second projector 20-2 on the projection surface PF. A vector AV having an average element of the elements of both panel horizontal central axis vectors HV is a vertical vector within the projection surface PF. This vertical vector within the projection surface PF is referred to as a "vertical vector" in this specification. This vertical vector is also referred to as v(v x ,v y ,v z The vertical direction acquisition unit 113-2 acquires the vertical direction in the projection plane PF based on the vertical vector v.

[0073] Here, the direction indicated by the three-dimensional panel horizontal central axis direction vector HV1 of the first projector 20-1 on the projection surface PF is an example of the above-mentioned "first direction". Also, the direction indicated by the three-dimensional panel horizontal central axis direction vector HV2 of the second projector 20-2 on the projection surface PF is an example of the "third direction". The direction indicated by the vector AV which is the average of the three-dimensional panel horizontal central axis direction vector HV1 of the first projector 20-1 on the projection surface PF and the three-dimensional panel horizontal central axis direction vector HV2 of the second projector 20-2 on the projection surface PF is an example of the "fourth direction". The fourth direction is a direction intermediate between the first direction and the third direction.

[0074] As described above, the first projector 20-1 and the second projector 20-2 are installed substantially horizontally, but the roll rotation components of both projectors 20 are not 0. The vertical direction acquisition unit 113-2 averages these to compensate for the roll rotation as much as possible. Furthermore, when the projection system 1 performs tiling using three or more projectors 20 instead of two, the variation in the roll rotation is more averaged and the roll rotation is more compensated for.

[0075] The horizontal direction acquisition unit 113-3 acquires a horizontal direction that is perpendicular to the normal direction and the vertical direction on the projection surface PF. Specifically, the horizontal direction acquisition unit 113-3 acquires the normal vector n (n x ,n y ,n z ) and the vertical vector v(v x ,v y ,v z ) and normalize the cross product with the horizontal vector h(h x ,h y ,h z ) The direction indicated by the horizontal vector h is an example of the above-mentioned "second direction."

[0076] In FIG. 4, the adjustment unit 114 adjusts the shape of the projection images PP1 and PP2 including a part of the display image DP so that a rectangular display image DP having one side perpendicular to the vertical direction and another side perpendicular to the horizontal direction is displayed on the projection surface PF. Here, the display image DP is an example of a "first display image". The display image DP as the first display image has a side perpendicular to the first direction. The side perpendicular to the first direction is an example of a "second side". The display image DP also has a side perpendicular to the second direction. Here, the "second direction" is a direction perpendicular to the normal direction of the projection surface PF and the "first direction". The side perpendicular to the second direction is an example of a "third side". The projection images PP1 and PP2 are examples of a "second projection image". FIG. 9 is a functional block diagram showing the function of the adjustment unit 114. The adjustment unit 114 includes a transformation matrix calculation unit 114-1, a projection area detection unit 114-2, a coordinate system conversion unit 114-3, a search unit 114-4, a coordinate value calculation unit 114-5, and a geometric transformation unit 114-6.

[0077] The transformation matrix calculation unit 114-1 calculates a transformation matrix from the first camera coordinate system, which is a three-dimensional coordinate system seen from the first imaging device 30-1, to a three-dimensional coordinate system when the projection surface PF is seen from the front. x ,n y ,nz ), vertical vector v(v x ,v y ,v z ), and the horizontal vector h(h x ,h y ,h z ) is defined by the following formula (4).

number

[0078] The projection area detection unit 114-2 detects the projection area of ​​each projector 20 on the image captured by the first imaging device 30-1. Specifically, the projection area detection unit 114-2 extracts the coordinate values ​​of four lattice points closest to the coordinates corresponding to the four corners of the liquid crystal panel 260 of each projector 20 from the coordinate values ​​of the corresponding point group on the image captured by the first imaging device 30-1 acquired by the correspondence acquisition unit 111-1. The area surrounded by these four lattice points almost coincides with the projection area. Note that the projection area detection unit 114-2 may calculate in advance a projection transformation matrix between the first camera image coordinate system and the panel image coordinate system of each projector 20, and acquire the coordinate values ​​of the four corner points without margins by projecting the coordinate values ​​of the four corner points of the liquid crystal panel 260 onto the first camera image coordinate system.

[0079] The coordinate system conversion unit 114-3 converts the coordinate values ​​of the projection area in the first camera image coordinate system into coordinate values ​​in the projection plane coordinate system. Specifically, the coordinate system conversion unit 114-3 converts the coordinate values ​​of the four corner points of the projection area in the first camera image coordinate system into coordinate values ​​of the four corner points of the projection area in the first camera normalized coordinate system by using the internal parameters of the first imaging device 30-1. This conversion process is the same as the conversion process executed by the plane orientation estimation unit 111-3. Furthermore, the coordinate system conversion unit 114-3 converts the coordinate values ​​of the four corner points of the projection area in the first camera normalized coordinate system into coordinate values ​​of the four corner points of the projection area in the first camera coordinate system by using the plane parameters (a, b, c). This conversion process is the same as the conversion process executed by the on-plane conversion unit 112. Furthermore, the coordinate system conversion unit 114-3 converts the coordinate values ​​of the four corner points of the projection area in the first camera coordinate system into the coordinate values ​​of the four corner points of the projection area in the projection surface coordinate system by using the conversion matrix R. Specifically, if the coordinate values ​​of the four corner points of the projection area in the first camera coordinate system are (X1, Y1, Z1), the coordinate system conversion unit 114-3 converts the coordinate values ​​of the four corner points of the projection area in the three-dimensional projection surface coordinate system (X S ,Y S ,Z S ) is calculated using the following formula (5).

number

[0080] The search unit 114-4 searches for a rectangle with the maximum area inscribed in the entire area, which is the sum of the projection area by the first projector 20-1 and the projection area by the second projector 20-2. FIG. 10 is a diagram showing an example of the projection area AR1 by the first projector 20-1, the projection area AR2 by the second projector 20-2, and the rectangle SQ with the maximum area. The search unit 114-4 may draw a plurality of rectangles SQ in the entire area and select the rectangle SQ with the maximum area among the plurality of rectangles SQ. Alternatively, the search unit 114-4 may determine the rectangle SQ with the maximum area using dynamic programming. The aspect ratio of the rectangle SQ searched for at this time may be an aspect ratio set in advance by the user. Alternatively, if there is no particular designation, the search unit 114-4 may determine the rectangle SQ with the maximum area in the entire area regardless of the aspect ratio. Searching unit 114-4 stores the coordinates of the four corners of rectangle SQ determined by the above method in the two-dimensional projection surface coordinate system as a corrected connected area in the projection surface coordinate system.

[0081] Coordinate value calculation unit 114-5 calculates coordinate values ​​of the four corners of the corrected connected area in the first panel image coordinate system of first projector 20-1 and in the second panel image coordinate system of second projector 20-2, using the coordinate values ​​of the four corners of the corrected connected area stored by search unit 114-4. Figures 11 to 13 are explanatory diagrams of the operation of coordinate value calculation unit 114-5.

[0082] First, the coordinate value calculation unit 114-5 divides the rectangle SQ as the corrected connected region shown in Fig. 10 into two rectangles SQ1 and SQ2 that match the aspect ratios of the liquid crystal panels 260 of the first projector 20-1 and the second projector 20-2 as shown in Fig. 11. At this time, the coordinate value calculation unit 114-5 makes the left side of the rectangle SQ1 match the left side of the rectangle SQ, and the right side of the rectangle SQ2 match the right side of the rectangle SQ. In addition, the coordinate value calculation unit 114-5 sets the coordinates of the four corners of each of the rectangles SQ1 and SQ2 as corrected four corner coordinates in the projection surface coordinate system.

[0083] Next, the coordinate value calculation unit 114-5 obtains the coordinate values ​​of the four corner coordinates of the projection area AR1 before correction in the first panel image coordinate system and the coordinate values ​​of the four corner coordinates of the projection area AR1' after correction in the projection surface coordinate system. At this time, the coordinate values ​​of the four corner coordinates of the projection area AR1 before correction in the first panel image coordinate system can be obtained from the panel resolution of the first projector 20-1.

[0084] Next, the coordinate value calculation unit 114-5 calculates a projective transformation matrix H1 based on the correspondence relationship between the coordinate values ​​of the four corners of the projection area AR1 before correction in the first panel image coordinate system and the coordinate values ​​of the four corners of the projection area AR1' after correction in the projection surface coordinate system. The projective transformation matrix H1 is a projective transformation matrix from the projection surface coordinate system to the first panel image coordinate system.

[0085] Finally, as shown in FIG. 12, the coordinate value calculation unit 114-5 can calculate the corrected four corner coordinates of the rectangle SQ1', which is the final output, by projecting the corrected four corner coordinates of the rectangle SQ1 in the projection surface coordinate system into the first panel image coordinate system using the projection transformation matrix H1.

[0086] Similarly, the coordinate value calculation unit 114-5 acquires the coordinate values ​​of the four corner coordinates of the projection area AR2 before correction in the second panel image coordinate system and the coordinate values ​​of the four corner coordinates of the projection area AR2' after correction in the projection surface coordinate system. At this time, the coordinate values ​​of the four corner coordinates of the projection area AR2 before correction in the second panel image coordinate system can be obtained from the panel resolution of the second projector 20-2.

[0087] Next, the coordinate value calculation unit 114-5 calculates a projective transformation matrix H2 based on the correspondence relationship between the coordinate values ​​of the four corners of the projection area AR2 before correction in the second panel image coordinate system and the coordinate values ​​of the four corners of the projection area AR2' after correction in the projection surface coordinate system. The projective transformation matrix H2 is a projective transformation matrix from the projection surface coordinate system to the second panel image coordinate system.

[0088] Finally, as shown in Figure 13, the coordinate value calculation unit 114-5 can calculate the corrected four corner coordinates of the rectangle SQ2', which is the final output, by projecting the corrected four corner coordinates of the rectangle SQ2 in the projection surface coordinate system onto the second panel image coordinate system using the projection transformation matrix H2.

[0089] In FIG. 9, geometric transformation section 114-6 performs geometric transformation on the projected image using the corrected four corner coordinates of rectangle SQ1' and the corrected four corner coordinates of rectangle SQ2' calculated by coordinate value calculation section 114-5.

[0090] In FIG. 4, the projection control unit 115 causes the first projector 20-1 and the second projector 20-2 to project the above-mentioned pattern image toward the projection surface PF. Here, the pattern image projected from the first projector 20-1 is an example of the above-mentioned "first projection image". Moreover, the pattern image projected from the second projector 20-2 is an example of the "third projection image". Moreover, the projection control unit 115 causes the first projector 20-1 and the second projector 20-2 to project the projection image adjusted by the adjustment unit 114 toward the projection surface PF. Here, the projection image projected from the first projector 20-1 is an example of the "fourth projection image". Moreover, the projection image projected from the second projector 20-2 is an example of the "fifth projection image". Specifically, the projection control unit 115 causes the first projector 20-1 to project the projection image corrected to the shape of a rectangle SQ1' shown in FIG. 12 toward the projection surface PF. Similarly, the projection control unit 115 causes the second projector 20-2 to project the projection image corrected to the shape of a rectangle SQ2' shown in FIG. 13 onto the projection surface PF. As a result, the rectangle SQ shown in FIG. 10 is displayed on the projection surface PF. The rectangle SQ is an example of a "second display image." The second display image includes a side perpendicular to the fourth direction. The side perpendicular to the fourth direction is an example of a "second side." The second display image also includes a side perpendicular to the second direction. The side perpendicular to the second direction is an example of a "third side." The fourth projection image includes a part of the second display image, and the fifth projection image includes the remaining part of the second display image.

[0091] 1-4: Operation of the embodiment 14 and 15 are flowcharts showing the operation of the information processing device 10 according to the first embodiment. Hereinafter, the operation of the information processing device 10 will be described with reference to FIGS.

[0092] In step S11, the processing device 11 functions as the projection control unit 115. The processing device 11 causes the first projector 20-1 to project a pattern image onto the projection surface PF. Similarly, the processing device 11 causes the second projector 20-2 to project a pattern image onto the projection surface PF.

[0093] In step S12, the processing device 11 functions as a first captured image acquisition unit 111-1A and a second captured image acquisition unit 111-1B. The processing device 11 acquires a captured image of a pattern image captured by the first imaging device 30-1. The processing device 11 also acquires a captured image of a pattern image captured by the second imaging device 30-2. Furthermore, the processing device 11 functions as a correspondence relationship acquisition unit 111-1. The processing device 11 acquires a correspondence relationship between the first camera image coordinate system and the first panel image coordinate system, a correspondence relationship between the first camera image coordinate system and the second panel image coordinate system, a correspondence relationship between the second camera image coordinate system and the first panel image coordinate system, and a correspondence relationship between the second camera image coordinate system and the second panel image coordinate system.

[0094] In step S13, the processing device 11 functions as an axis direction detection unit 111-2. The processing device 11 detects, in the camera image coordinate system, the panel horizontal central axis direction, which is the direction of the axis corresponding to the horizontal central axis in the panel image coordinate system.

[0095] In step S14, the processing device 11 functions as a plane orientation estimation unit 111-3. The processing device 11 estimates the orientation of the projection surface PF with respect to the first imaging device 30-1.

[0096] In step S15, the processing device 11 functions as an on-plane transformation unit 112. The processing device 11 uses the plane parameters of the projection surface PF to transform a two-dimensional panel horizontal central axis direction vector in the camera image coordinate system into a three-dimensional panel horizontal central axis direction vector on the projection surface PF.

[0097] In step S16, the processing device 11 functions as a normal direction acquisition unit 113-1. The processing device 11 acquires the normal direction of the projection surface PF.

[0098] In step S17, the processing device 11 functions as the vertical direction acquisition unit 113-2. The processing device 11 acquires the vertical direction of the projection surface PF.

[0099] In step S18, the processing device 11 functions as a horizontal direction acquisition unit 113-3. The processing device 11 acquires the horizontal direction of the projection surface PF.

[0100] In step S19, the processing device 11 functions as a transformation matrix calculation unit 114-1. The processing device 11 calculates a transformation matrix from the first camera coordinate system, which is a three-dimensional coordinate system seen from the first imaging device 30-1, to a three-dimensional coordinate system when the projection surface PF is seen from the front.

[0101] In step S20, the processing device 11 functions as a projection area detection unit 114-2. The processing device 11 detects the projection area of ​​each projector 20 on the image captured by the first imaging device 30-1.

[0102] In step S21, the processing device 11 functions as the coordinate system conversion unit 114-3. The processing device 11 converts the coordinate values ​​of the projection area in the first camera image coordinate system into coordinate values ​​in the projection surface coordinate system.

[0103] In step S22, the processing device 11 functions as a search unit 114-4. The processing device 11 searches for a rectangle SQ with a maximum area inscribed in the entire area, which is the sum of the projection area AR1 by the first projector 20-1 and the projection area AR2 by the second projector 20-2.

[0104] In step S23, the processing device 11 functions as a coordinate value calculation unit 114-5. The processing device 11 calculates the coordinate values ​​of the four corners of a rectangle SQ1' in the first panel image coordinate system of the first projector 20-1 and a rectangle SQ2' in the second panel image coordinate system of the second projector 20-2, using the coordinate values ​​of the four corners of the corrected connected region stored by the search unit 114-4.

[0105] In step S24, the processing device 11 functions as the geometric transformation unit 114-6. The processing device 11 performs geometric transformation on the projected image using the corrected four corner coordinates of the rectangle SQ1' and the corrected four corner coordinates of the rectangle SQ2'.

[0106] In step S25, the processing device 11 functions as the projection control unit 115. The processing device 11 causes the adjusted projection image to be projected from each of the first projector 20-1 and the second projector 20-2 onto the projection surface PF.

[0107] 2: Variations The present disclosure is not limited to the above-described embodiments. Specific modified embodiments are exemplified below.

[0108] 2-1: Variation 1 In the above embodiment, the projection system 1 includes two projectors: the first projector 20-1 and the second projector 20-2. However, the projection system 1 may include any number of projectors.

[0109] When the projection system 1 includes only one first projector 20-1, a first direction indicated by a three-dimensional panel horizontal central axis direction vector HV1 associated with the first projector 20-1 on the projection surface PF becomes the vertical direction.

[0110] 2-2: Variation 2 In the above embodiment, the information processing device 10, the first projector 20-1, and the first imaging device 30-1 are separate from each other. However, two or more of these devices may be realized as a single device stored in the same housing. The same applies to the information processing device 10, the second projector 20-2, and the second imaging device 30-2.

[0111] 2-3: Variation 3 In the above embodiment, the projection system 1 may use a stereo camera having two imaging devices instead of the first imaging device 30-1 and the second imaging device 30-2. Alternatively, the projection system 1 may use a TOF camera capable of three-dimensional measurement independently instead of the first imaging device 30-1 and the second imaging device 30-2.

[0112] 2-4: Variation 4 In the above embodiment, the information processing device 10 may be any one of a PC, a smartphone, and a tablet. Alternatively, the functions of the information processing device 10 may be distributed as an application to an external device (not shown in FIG. 1) via a communication network NET.

[0113] 3: Summary of this disclosure The following is a summary of this disclosure. (Supplementary Note 1) A method for adjusting a projection image, comprising: acquiring measurement data measuring a three-dimensional shape of a projection surface onto which a first projection image is projected from a first projector including a rectangular first display panel having a first side perpendicular to a first axis; calculating parameters related to the three-dimensional shape based on the acquired measurement data; acquiring a normal direction of the projection surface based on the parameters; acquiring a first direction parallel to a second axis in the first projection image corresponding to the first axis based on the parameters; acquiring a second direction perpendicular to the normal direction and the first direction; adjusting a shape of a second projection image including a portion of a rectangular first display image on the projection surface, the second side perpendicular to the first direction and a third side perpendicular to the second direction; and projecting the second projection image onto the projection surface from the first projector.

[0114] The above-described method for adjusting a display image acquires a first direction parallel to a second axis in the first projection image and a second direction perpendicular to the normal direction of the projection surface and the first direction, and adjusts the shape of the projection image based on the acquired first and second directions. Thus, when the projection image is projected onto the projection surface, rotation of the projection image within the projection surface as seen by the projector can be suppressed.

[0115] (Supplementary Note 2) A method for adjusting a projection image according to Supplementary Note 1, wherein the second projection image is projected onto the projection surface via a lens, the first display panel has a fourth side forming a right angle with the first side, a fifth side forming a right angle with the fourth side, and a sixth side forming a right angle with the fifth side and also forming a right angle with the first side, and the first axis passes through an intersection between an optical axis of the lens and the first display panel and is parallel to the fourth side and the sixth side.

[0116] By using the above-mentioned method for adjusting a projected image, the first axis can be accurately defined while taking into account the effects of rotation by obtaining a direction that passes through the intersection of the optical axis of the lens and the first display panel and is parallel to the opposing sides.

[0117] (Supplementary Note 3) The method for adjusting a projection image described in Supplementary Note 1 or 2, wherein the first projection image has a seventh side, an eighth side connected to the seventh side, a ninth side connected to the eighth side, and a tenth side connected to the ninth side and the seventh side, and the second axis is an axis passing through a midpoint of the seventh side and a midpoint of the ninth side of the first projection image on the projection surface.

[0118] By using the above-mentioned method for adjusting a projected image, it is possible to obtain the direction connecting the midpoints of opposing sides of a first projected image, thereby accurately defining a second axis in the first projected image projected onto the projection surface that corresponds to the first axis, while taking into account the effects of rotation.

[0119] (Supplementary Note 4) The method for adjusting a projection image described in any one of Supplementary Note 1 to Supplementary Note 3, further comprising: a third projection image is projected onto the projection surface from a second projector equipped with a rectangular second display panel having an eleventh side perpendicular to a third axis; and the method further comprises: obtaining a third direction toward one end of a fourth axis corresponding to the third axis in the third projection image based on the parameters; and calculating a fourth direction that is a direction intermediate between the first direction and the third direction; and when an image is projected onto the projection surface by the first projector and the second projector, the second direction is perpendicular to the normal direction and the fourth direction.

[0120] According to the above-described method for adjusting a projection image, the fourth direction can be appropriately obtained by obtaining an intermediate direction between the directions in the images projected by a plurality of projectors.

[0121] (Supplementary Note 5) The method for adjusting a projection image described in Supplementary Note 4 includes adjusting a shape of a fourth projection image including a part of a rectangular second display image having a second side perpendicular to the fourth direction and a third side perpendicular to the second direction on the projection surface, and a shape of a fifth projection image including a remaining part of the second display image, and when an image is projected onto the projection surface by the first projector and the second projector, projecting the fourth projection image from the first projector and projecting the fifth projection image from the second projector.

[0122] The above-described method for adjusting a projected image displays a rectangular second display image having a second side perpendicular to the fourth direction and a third side perpendicular to the second direction, thereby preventing the projected image from being displayed rotated even when projected using multiple projectors.

[0123] (Supplementary Note 6) A projection system comprising: a first projector having a rectangular first display panel having a first side orthogonal to a first axis; a sensor that measures a three-dimensional shape of a projection surface onto which the first projector projects a first projection image; and an information processing device that executes the following: calculating parameters related to the three-dimensional shape based on measurement data acquired from the sensor; acquiring a normal direction of the projection surface based on the parameters; acquiring a first direction parallel to a second axis in the first projection image corresponding to the first axis based on the parameters; acquiring a second direction orthogonal to the normal direction and the first direction; adjusting a shape of a second projection image including a part of a rectangular first display image on the projection surface, the second side orthogonal to the first direction and a third side orthogonal to the second direction; and projecting the second projection image from the first projector onto the projection surface.

[0124] The above projection system acquires a first direction parallel to the second axis in the first projection image and a second direction perpendicular to the normal direction of the projection surface and the first direction, and adjusts the shape of the projection image based on the acquired first and second directions, thereby suppressing rotation of the projection image within the projection surface as seen by the projector when the projection image is projected onto the projection surface.

[0125] (Additional Note 7) An information processing program that causes a computer to perform the following operations: calculate parameters related to a three-dimensional shape based on measurement data acquired from a sensor that measures the three-dimensional shape of a projection surface onto which a first projection image is projected by a first projector including a rectangular first display panel having a first side perpendicular to a first axis; acquire a normal direction of the projection surface based on the parameters; acquire a first direction parallel to a second axis in the first projection image corresponding to the first axis based on the parameters; acquire a second direction perpendicular to the normal direction and the first direction; adjust a shape of a second projection image including a portion of a rectangular first display image on the projection surface having a second side perpendicular to the first direction and a third side perpendicular to the second direction; and project the second projection image from the first projector onto the projection surface.

[0126] The above information processing program acquires a first direction parallel to a second axis in the first projection image and a second direction perpendicular to the normal direction of the projection surface and the first direction, and adjusts the shape of the projection image based on the acquired first and second directions, thereby suppressing rotation of the projection image within the projection surface as seen by the projector when the projection image is projected onto the projection surface. [Explanation of symbols]

[0127] 1...projection system, 10...information processing device, 11...processing device, 12...storage device, 13...display device, 14...communication device, 20...projector, 20-1...first projector, 20-2...second projector, 21...projection device, 22...processing device, 23...storage device, 24...communication device, 30...sensor, 30-1...first imaging device, 30-2...second imaging device, 111...three-dimensional shape calculation unit, 111-1...correspondence relationship acquisition unit, 111-1A...first captured image acquisition unit, 111-1B...second captured image acquisition unit, 111-2...axial direction detection unit, 111-3...plane orientation estimation unit, 111-3A...transformation matrix acquisition unit, 111-3B...plane parameter acquisition unit, 112...on-plane conversion unit, 113...direction acquisition unit, 113-1...normal direction acquisition unit, 113-2...vertical direction acquisition unit acquisition unit, 113-3... horizontal direction acquisition unit, 114... adjustment unit, 114-1... transformation matrix calculation unit, 114-2... projection area detection unit, 114-3... coordinate system conversion unit, 114-4... search unit, 114-5... coordinate value calculation unit, 114-6... geometric deformation unit, 115... projection control unit, 210... optical system, 221... acquisition unit, 222... projection control unit, 240... lighting device, 250... separated optical system, 251,2 52...mirror, 253, 254...dichroic mirror, 260...liquid crystal panel, 280...projection optical system, 281...dichroic prism, 282...optical path shift element, 283...projection lens system, AR1, AR2...projection area, HV1, HV2...panel horizontal central axis direction vector, PP1, PP2...projected image, PR1, PR2...control program, SQ1, SQ2...rectangle

Claims

1. acquiring measurement data obtained by measuring a three-dimensional shape of a projection surface onto which a first projection image is projected from a first projector including a rectangular first display panel having a first side perpendicular to a first axis; Calculating parameters related to the three-dimensional shape based on the acquired measurement data; Obtaining a normal direction of the projection surface based on the parameters; obtaining a first direction parallel to a second axis in the first projection image, the first direction corresponding to the first axis, based on the parameters; obtaining a second direction perpendicular to the normal direction and the first direction; adjusting a shape of a second projection image including a part of a rectangular first display image having a second side perpendicular to the first direction and a third side perpendicular to the second direction on the projection surface; projecting the second projection image from the first projector onto the projection surface; A method for adjusting a projected image, comprising:

2. the second projection image is projected onto the projection surface via a lens; the first display panel includes a fourth side that is perpendicular to the first side, a fifth side that is perpendicular to the fourth side, and a sixth side that is perpendicular to the fifth side and is also perpendicular to the first side; the first axis passes through an intersection point between an optical axis of the lens and the first display panel and is parallel to the fourth side and the sixth side; The method for adjusting a projected image according to claim 1 .

3. the first projection image includes a seventh side, an eighth side connected to the seventh side, a ninth side connected to the eighth side, and a tenth side connected to the ninth side and the seventh side, the second axis is an axis passing through a midpoint of the seventh side and a midpoint of the ninth side of the first projected image on the projection surface; The method for adjusting a projected image according to claim 1 .

4. a third projection image is projected onto the projection surface from a second projector including a rectangular second display panel having an eleventh side perpendicular to a third axis; acquiring a third direction toward one end side of a fourth axis corresponding to the third axis in the third projection image based on the parameter; calculating a fourth direction that is an intermediate direction between the first direction and the third direction; Further comprising: When an image is projected onto the projection surface by the first projector and the second projector, the second direction is orthogonal to the normal direction and the fourth direction. The method for adjusting a projection image according to any one of claims 1 to 3.

5. adjusting, on the projection surface, a shape of a fourth projection image including a part of a rectangular second display image having a second side perpendicular to the fourth direction and a third side perpendicular to the second direction, and a shape of a fifth projection image including a remaining part of the second display image; When an image is projected onto the projection surface by the first projector and the second projector, projecting the fourth projection image from the first projector; projecting the fifth projection image from the second projector; Including, The method for adjusting a projected image according to claim 4.

6. a first projector including a rectangular first display panel having a first side perpendicular to a first axis; a sensor that measures a three-dimensional shape of a projection surface onto which the first projector projects a first projection image; Calculating parameters related to the three-dimensional shape based on the measurement data acquired from the sensor; Obtaining a normal direction of the projection surface based on the parameters; obtaining a first direction parallel to a second axis in the first projection image, the first direction corresponding to the first axis, based on the parameters; obtaining a second direction perpendicular to the normal direction and the first direction; adjusting a shape of a second projection image including a part of a rectangular first display image having a second side perpendicular to the first direction and a third side perpendicular to the second direction on the projection surface; projecting the second projection image from the first projector onto the projection surface; and an information processing device that executes A projection system comprising:

7. calculating parameters relating to the three-dimensional shape based on measurement data acquired from a sensor that measures a three-dimensional shape of a projection surface onto which a first projection image is projected by a first projector including a rectangular first display panel having a first side perpendicular to a first axis; Obtaining a normal direction of the projection surface based on the parameters; obtaining a first direction parallel to a second axis in the first projection image, the first direction corresponding to the first axis, based on the parameters; obtaining a second direction perpendicular to the normal direction and the first direction; adjusting a shape of a second projection image including a part of a rectangular first display image having a second side perpendicular to the first direction and a third side perpendicular to the second direction on the projection surface; projecting the second projection image from the first projector onto the projection surface; An information processing program that enables a computer to realize the above.

Images