Projection system, projection control device, and projection control program

Abstract

To provide a projection system, a projection control device, and a projection control program that projects an image with reduced distortion compared to conventional methods onto a three-dimensional projection surface having a concave surface from a projector on the basis of image data under a plurality of shooting conditions.SOLUTION: A projection control device acquires image data (S4), acquires model parameters, sets a three-dimensional model (S6), and determines a determined virtual projector from among a plurality of virtual projectors set in advance according to the shooting conditions of the image data (S5). The projection control device generates two-dimensional image data on the basis of a projection result when the image data is projected onto a model projection surface that represents the three-dimensional projection surface as a three-dimensional model, using the determined virtual projector arranged at a predetermined position, and two-dimensional coordinate data that represents the model projection surface in two-dimensional coordinates (S11), and projects the two-dimensional image data onto the three-dimensional projection surface using a projector (S12).SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a projection system, a projection control device, and a projection control program. [Background technology]

[0002] The projection method described in Patent Document 1 assumes a projected image that would be obtained if the same correctly reproduced image were projected from the center of a dome-shaped projection surface with the concave surface facing the observer, and transforms the image to be projected so that the image projected by a projector installed in an arbitrary position matches this image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 9-149351 Summary of the Invention [Problem to be solved by the invention]

[0004] In the conventional projection method, the image to be projected needs to be modified by adjusting the waveforms of the horizontal and vertical scanning voltages of the video projector, which makes the processing complicated. Moreover, in the conventional projection method, image data is processed by a single processing method, so that distortion occurs when image data different from the assumed shooting conditions is input.

[0005] An object of the present invention is to provide a projection system, a projection control device, and a projection control program that project an image with reduced distortion compared to conventional methods from a projector onto a three-dimensional projection surface consisting of a concave surface based on image data for each of a plurality of shooting conditions. [Means for solving the problem]

[0006] A projection system according to a first aspect of the present invention is a projection system including a projector having a fisheye lens and a projection control device that controls an image projected by the projector, in which the projection control device executes an image acquisition process that acquires image data to be processed, a model setting process that acquires model parameters including the size and shape of a three-dimensional projection surface, which is a concave surface, and sets a three-dimensional model, a virtual projector determination process that determines a determined virtual projector from among a plurality of virtual projectors that have been preset in accordance with the shooting conditions of the image data, a two-dimensional coordinate data conversion process that virtually captures a model projection surface that represents the three-dimensional projection surface in the three-dimensional model using a fisheye lens arranged at a position corresponding to the installation position of the projector, and acquires two-dimensional coordinate data that represents the model projection surface in two-dimensional coordinates, a two-dimensional image data generation process that generates two-dimensional image data based on the projection result and the two-dimensional coordinate data when the image data is virtually projected onto the model projection surface using the determined virtual projector arranged at a predetermined position with respect to the model projection surface, and an image projection process that projects the two-dimensional image data onto the three-dimensional projection surface using the projector.

[0007] Usually, when an image is projected by a projector having a fisheye lens, distortion specific to the fisheye lens occurs in the projected image. Similarly, when an image is captured by a camera having a fisheye lens, distortion specific to the fisheye lens occurs in the captured image. Here, since the capture and projection are reverse conversions, when image data captured by a fisheye lens is projected by a fisheye lens having the same lens characteristics as the fisheye lens used for the capture, the distortion at the time of capture is offset by the projection, and a projected image with less distortion can be obtained without correction specific to the fisheye lens. The projection system of the first aspect uses a virtual projector corresponding to each image data of a plurality of capture conditions to simulate an ideal projection environment corresponding to the capture conditions, and converts the projection result into two-dimensional image data when the image is captured using a fisheye lens from the actual installation position of the projector. As a result, the projection control device of the projection system can generate two-dimensional image data capable of reproducing an ideal projection environment by projection from a realistic projection environment through relatively simple processing. In other words, the projection control device of the projection system can perform planar projection and three-dimensional projection corresponding to the capture conditions within the same device. Therefore, based on image data for each of a plurality of shooting conditions, the projection system can obtain a projected image with less distortion than conventional methods, which is less unnatural to the viewer and is less noticeable regardless of the installation position of the projector, even when the projection surface has an arbitrary three-dimensional shape, through simpler processing than conventional methods.

[0008] The projection control device according to the second aspect of the present invention is a projection control device that controls an image projected by a projector having a fisheye lens, and includes an image acquisition process for acquiring image data to be processed, a model setting process for acquiring model parameters including the size and shape of a three-dimensional projection surface that is a concave surface and setting a three-dimensional model, a virtual projector determination process for determining a determined virtual projector from among a plurality of virtual projectors that have been preset according to the shooting conditions of the image data, a two-dimensional coordinate data conversion process for virtually shooting a model projection surface that represents the three-dimensional projection surface in the three-dimensional model using a fisheye lens arranged at a position according to the installation position of the projector and acquiring two-dimensional coordinate data that represents the model projection surface in two-dimensional coordinates, a two-dimensional image data generation process for generating two-dimensional image data based on the projection result and the two-dimensional coordinate data when the image data is virtually projected onto the model projection surface using the determined virtual projector arranged at a predetermined position with respect to the model projection surface, and an image projection process for projecting the two-dimensional image data from the projector. The projection control device according to the second aspect has the same effects as the projection system according to the first aspect.

[0009] A projection control program according to a third aspect of the present invention is a projection control program to be executed by a projection control device that controls an image projected by a projector having a fisheye lens, and includes instructions to cause the projection control device to execute an image acquisition process to acquire image data to be processed, a model setting process to acquire model parameters including the size and shape of a three-dimensional projection surface, which is a concave surface, and set a three-dimensional model, a virtual projector determination process to determine a determined virtual projector from among a plurality of virtual projectors set in advance according to shooting conditions of the image data, a two-dimensional coordinate data conversion process to virtually photograph a model projection surface, which represents the three-dimensional projection surface in the three-dimensional model, using a fisheye lens arranged at a position according to an installation position of the projector, and acquire two-dimensional coordinate data representing the model projection surface in two-dimensional coordinates, a two-dimensional image data generation process to generate two-dimensional image data based on the projection result and the two-dimensional coordinate data when the image data is virtually projected onto the model projection surface using the determined virtual projector arranged at a predetermined position with respect to the model projection surface, and an image projection process to project the two-dimensional image data from the projector. The projection control program of the third aspect, when executed by the projection control device, provides the same effects as the projection system of the first aspect. [Brief description of the drawings]

[0010] [Figure 1] 1A is a conceptual diagram of a projection system 1 having a projector 3 suspended from the ceiling of a room R, and FIG. 1B is an explanatory diagram of the room R in which a screen E is placed. [Diagram 2] 11 is a flowchart of a projection control process executed by the projection control device 2. [Diagram 3](A) is an oblique view of a three-dimensional model U1 set by the projection control process, (B) is an oblique view of a three-dimensional model U2 set by the projection control process, (C) is an explanatory diagram of a virtual projector PR placed at a predetermined position relative to the model projection surface UF of the three-dimensional model U1 and a virtual camera CA placed at an installation position, and (D) is an explanatory diagram of a virtual projector PR placed at a predetermined position relative to the model projection surface UK of the three-dimensional model U2 and a virtual camera CA placed at an installation position. [Figure 4] (A) is an explanatory diagram of a three-dimensional projection surface F, (B) is an explanatory diagram of a reference image Q projected onto the three-dimensional projection surface F in the projection control process, and (C) is an explanatory diagram of the reference image Q projected onto the three-dimensional projection surface F when the settings of the three-dimensional model are changed in the projection control process. [Diagram 5] 13 is a flowchart of a conversion process executed in the projection control process. [Figure 6] 1A is an explanatory diagram of image data G1 captured using a standard lens, FIG. 1B is an explanatory diagram of a projection result P1 when two-dimensional image data generated based on the image data G1 of specific example 1 is projected onto a three-dimensional projection surface F, and FIG. 1C is an explanatory diagram of a projection result P2 when two-dimensional image data generated based on the image data G1 of the comparative example is projected onto a three-dimensional projection surface F. [Figure 7] 1A is an explanatory diagram of image data G2 captured using a fisheye lens, FIG. 1B is an explanatory diagram of a projection result P3 when two-dimensional image data generated based on the image data G2 of specific example 2 is projected onto a three-dimensional projection surface F, and FIG. 1C is an explanatory diagram of a projection result P4 when two-dimensional image data generated based on the image data G2 of specific example 3 is projected onto a three-dimensional projection surface K. [Figure 8] 1A is an explanatory diagram of image data G3 captured using a 360-degree camera, and FIG. 1B is an explanatory diagram of the projection result P5 when two-dimensional image data generated based on image data G of specific example 4 is projected onto a three-dimensional projection surface F. [Figure 9]9(A) is an explanatory diagram of the projection result P7 when the projection position of the projection result P6 in FIG. 9(B) is changed, (B) is an explanatory diagram of the projection result P6, and (C) is an explanatory diagram of the projection result P8 when the projection magnification of the projection result P6 is changed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following description, the X direction, Y direction, and Z direction shown in FIG. 1(A) are used. The Z direction is the vertical direction. The X direction and the Y direction are horizontal directions perpendicular to the Z direction. The X direction, the Y direction, and the Z direction are directions perpendicular to each other.

[0012] As shown in Fig. 1(A), the projection system 1 includes a projector 3 having a fisheye lens 4, and a projection control device 2 that controls an image projected by the projector 3. The projector 3 and the projection control device 2 are connected to each other via a wired or wireless connection. The projection system 1 is used in a room R of an amusement facility such as a karaoke store. The projection system 1 can be placed in any position, and for example, the projector 3 is hung from the ceiling of the room R.

[0013] The projector 3 projects image data onto a three-dimensional projection surface, which is a concave surface, in response to an image signal output from the projection control device 2. The shape of the three-dimensional projection surface may be any shape as long as it is a concave surface, and the three-dimensional projection surface of this embodiment is a three-dimensional projection surface F shown in FIG. 1(A) or a three-dimensional projection surface K shown in FIG. 1(B). As shown in FIG. 1(A), the three-dimensional projection surface F has five surfaces, including a first surface F1 facing the fisheye lens 4 of the projector 3 and four surfaces F2 to F5 continuous with the first surface F1, among the six surfaces of the cubic room R. The first surface F1 is a surface extending in the vertical direction. The second surface F2 and the third surface F3 are each perpendicular to the first surface F1 and extend in the vertical direction. The fourth surface F4 and the fifth surface F5 are each perpendicular to the first surface F1 and extend in the horizontal direction. The three-dimensional projection surface K is a concave surface of a screen E that can be installed in the cubic room R. As shown in FIG. 1B, the three-dimensional projection surface K has five surfaces including a first surface K1 facing the fisheye lens 4 of the projector 3 and four surfaces K2 to K5 continuous with the first surface K1. The first surface K1 is a surface extending in the vertical direction. The second surface K2 and the third surface K3 each intersect with the first surface K1 and extend in the vertical direction. The fourth surface K4 and the fifth surface K5 each intersect with the first surface K1 and extend in the horizontal direction. The first surface K1 and the second surface K2 are connected by a first curved surface C1, and the first surface K1 and the third surface K3 are connected by a second curved surface C2. The first surface of the three-dimensional projection surface is less unnatural in the projection result when it is a flat surface than a curved surface, but the connection part between the first surface and the second surface and the connection part between the first surface and the third surface creates an unnatural projection result and the connection parts are noticeable. Therefore, in the three-dimensional projection plane K, the connection between the first plane K1 and the second plane K2 and the connection between the first plane K1 and the third plane K3 are curved planes C1 and C2. In order to reduce the sense of incongruity when the projection result is viewed from a place far from the screen E, the distance between the second plane K2 and the third plane K3 is set to a trapezoidal shape in a plan view, which is larger as it is farther from the first plane K1. The screen E is disposed, for example, such that the surface opposite the first plane K1 faces and is close to the first plane F1 of the room R. In this embodiment, the first plane F1 is the wall surface of the room R, but the first plane F1 may be either the ceiling surface or the floor surface of the room R.

[0014] The projection control device 2 converts image data into two-dimensional image data for projection, and performs projection control of the projector 3 based on the two-dimensional image data. The projection control device 2 may be disposed at any position in the room R, or may be disposed at a location other than the room R. The projection control device 2 includes a communication unit 12, a storage unit 15, an input unit 14, and a control unit 11. The communication unit 12, the storage unit 15, and the input unit 14 are connected to the control unit 11. The storage unit 15 is configured by, for example, a hard disk drive. The storage unit 15 stores an OS, various programs, and the like. The various programs include a projection control program that causes the CPU 21 of the control unit 11 to execute a projection control process. The storage unit 15 stores image data to be projected, internal parameters and external parameters of the projector 3, internal parameters and external parameters of the virtual projector PR, internal parameters and external parameters of the virtual camera CA, various parameters, and the like. Shooting conditions are added to the image data. The shooting conditions in this embodiment include the lens used for shooting, the lens characteristics used for shooting, and the type of camera. The input unit 14 receives an operation instruction from a user of the projection system 1, and outputs a signal according to the operation instruction to the control unit 11. The input unit 14 is, for example, a game controller having a joystick and a cross key. The control unit 11 is composed of a CPU 21, a GPU 22, a memory 23, and the like as a computer. The GPU 22 is a processor provided separately from the CPU 21, and performs image processing based on an instruction from the CPU 21. The memory 23 is, for example, composed of a ROM, a RAM, and the like.

[0015] With reference to FIG. 2 to FIG. 9(C), the projection control process will be described using specific example 1 to specific example 4 of the projection control device 2. In the projection control process, a process of projecting image data designated by a user onto a three-dimensional projection surface F is executed. In each of specific examples 1 to 4, image data is taken of each frame of video data of four people having a pleasant conversation on a stage. In specific example 1, video data taken with a camera using a standard lens is projected onto the three-dimensional projection surface F. In specific example 2, video data taken with a camera using a fisheye lens is projected onto the three-dimensional projection surface F. In specific example 3, video data taken with a camera using a fisheye lens is projected onto the three-dimensional projection surface K. The three-dimensional projection surface K is a three-dimensional projection surface whose planar shape is a trapezoid with rounded corners. In specific example 4, video data taken with an omnidirectional camera is projected onto the three-dimensional projection surface F. In specific examples 1 to 4, the control unit 11 generates two-dimensional image data to be projected by the projector 3 using a virtual projector according to the shooting conditions of the video data. The virtual projector is a parameter virtually set to calculate the correspondence between image data and pixels of a three-dimensional model. The projection control device 2 of this embodiment presets three types of virtual projectors, and stores shader programs and parameters required for virtual projection using each virtual projector in the storage unit 15. The three types of virtual projectors are a standard virtual projector, a fish-eye virtual projector, and a 360-degree virtual projector. The standard virtual projector virtually projects image data along a model projection surface that represents a three-dimensional projection surface as a three-dimensional model. The fish-eye virtual projector virtually projects image data along the model projection surface using a virtual fish-eye lens. The 360-degree projector virtually projects 360 degrees onto the model projection surface so that the elevation and depression angles and azimuth angles match when viewed from the virtual projector. The projection result of the 360-degree virtual projector is virtually projected 360 degrees, but in reality, the projection possible range of the projector 3 having the fisheye lens 4 indicated by the lens characteristics acquired in S9 described later is output. The type and number of multiple virtual projectors may be changed as appropriate.As a comparative example, a case will be described in which, in specific example 1, a fish-eye virtual projector is used to generate two-dimensional image data to be projected by the projector 3.

[0016] The projection control process is started when the user inputs a start instruction. When the control unit 11 detects the start instruction, it reads out a projection control program for executing the projection control process stored in the storage unit 15 into the memory 23. The control unit 11 executes the following steps according to the instructions contained in the program read out into the memory 23. Various parameters necessary for executing the projection control process are stored in the storage unit 15. Various data obtained in the course of the projection control process are appropriately stored in the memory 23. In the projection control process, specific examples 1 to 4 are each executed at different times.

[0017] As shown in FIG. 2, the control unit 11 acquires the initial settings stored in the storage unit 15 (S1). The initial settings include the settings of the current three-dimensional model. The settings of the current three-dimensional model are expressed using model parameters including the size and shape of the three-dimensional projection surface set in the three-dimensional model. The control unit 11 judges whether or not to change the three-dimensional model (S2). In this embodiment, the cubic three-dimensional model U1 in FIG. 3 is set as the initial value of the three-dimensional model. The projection control device 2 of this embodiment can change the shape and length of each face of the current three-dimensional model, the shape of a curved surface when the connection part of two adjacent faces of any face is a curved surface, and the shape of a trapezoid when the contour shape of any face is a trapezoid. Furthermore, the control unit 11 can make changes such as rotating or translating the current three-dimensional model. Usually, when the user changes the arrangement of the projector 3 relative to the three-dimensional projection surface, the user changes the three-dimensional model. When the user wants to change the settings of the current three-dimensional model, the user operates the input unit 14 to input an instruction to perform a process to change the settings.

[0018] As shown in FIG. 3(A), a three-dimensional model U1 is set in the current three-dimensional model for the three-dimensional projection surface F of the specific examples 1, 2, and 4. As shown in FIG. 3(B), a three-dimensional model U2 is set in the current three-dimensional model for the three-dimensional projection surface K of the specific example 3. The setting of the three-dimensional model U1 is changed in the three-dimensional model U2 according to the shape of the screen E. The framework information B1 shown by the lattice pattern of the three-dimensional model U1 and the framework information B2 shown by the lattice pattern of the three-dimensional model U2 are information for mapping image data as texture when the standard virtual projector is determined as the determined virtual projector in the process of S5 described later. The standard virtual projector of this embodiment is used in the process of generating two-dimensional image data by projecting image data by the control unit 11 so as to arrange the image data along the three-dimensional projection surface. The framework information indicated by the three-dimensional model will be described using the three-dimensional projection surface F as an example. The control unit 11 sets the center of the first surface F1 as the center of the image data and sets a projection magnification of a size such that the image data protrudes from the first surface F1. The control unit 11 projects the image data of the portion extending from the first face F1 to the second face F2 and the third face F3 that are connected to the first face F1 in the horizontal direction so as to bend the image data along the projection surface. The control unit 11 projects the image data of the portion extending from the first face F1 to the fourth face F4 and the fifth face F5 that are connected to the first face F1 in the vertical direction so as to shrink the image data toward the center of the surface as shown by a grid pattern.

[0019] When an instruction to change the setting of the three-dimensional model U1 before the setting change shown in FIG. 4(A) is detected (S2: YES), the control unit 11 uses the projector 3 to project a reference image Q showing the setting status of the three-dimensional model U1 onto the three-dimensional projection surface F via the fisheye lens 4 as shown in FIG. 4(B) (S21). The reference image Q may be an image showing the current setting status of the three-dimensional model, for example, an image showing the boundary between two adjacent faces, or an image including a lattice pattern showing the framework information B1 of the three-dimensional model U1. After starting to project the reference image Q, the control unit 11 accepts a setting change instruction to change the setting of the three-dimensional model U1 (S22). The user operates the input unit 14 to select an item to change the setting, and then inputs an instruction by specifying the amount of change. When an instruction to change the setting of the three-dimensional model U1 is detected (S22: YES), the control unit 11 changes the setting of the three-dimensional model U1 according to the instruction accepted in S22, and changes and projects the reference image Q to be projected onto the three-dimensional projection surface F (S23). While viewing the reference image projected onto the three-dimensional projection surface F, the user operates the input unit 14 to adjust the parameters of the three-dimensional model so that the reference image Q is positioned along the three-dimensional projection surface F as shown in Fig. 4(C). In Fig. 4(C), the boundary of the pattern of the reference image Q coincides with the boundary of any two adjacent surfaces among the five surfaces constituting the three-dimensional projection surface. By the processes of S22 and S23, it is possible to change the positional relationship between the room R in which the projector 3 is installed and the projector 3, the fisheye lens characteristics of the fisheye lens 4 of the projector 3 acquired in S9, the predetermined position of the virtual projector acquired in S6, and the orientation of the determined virtual projector determined in S5. For the cubic three-dimensional model U1 shown in Fig. 3(A), curved surfaces C1 and C2 are set at the connection between the first surface F1 and the second surface F2 and at the connection between the first surface F1 and the third surface F3, and the distance between the second surface F2 and the third surface F3 is changed to a trapezoidal shape in a plan view in which the distance is greater on the side farther away from the first surface F1, thereby changing the three-dimensional model U1 to the three-dimensional model U2 shown in Fig. 3(B). If an instruction to change the settings of the three-dimensional model is not detected (S22: NO), or following S23, the control unit 11 determines whether an instruction to end the process of changing the settings of the three-dimensional model has been detected (S24).If an instruction to end the process of changing the settings of the three-dimensional model is not detected (S24: NO), the control unit 11 returns the process to S21. If an instruction to end the process of changing the settings of the three-dimensional model is detected (S24: YES), the control unit 11 returns the process to S2.

[0020] If an instruction to change the setting of the three-dimensional model is not detected (S2: NO), the control unit 11 judges whether a projection instruction to project image data onto a three-dimensional projection surface is detected (S3). If a projection instruction is not detected (S3: NO), the control unit 11 returns the process to S2. If a projection instruction is detected (S3: YES), the control unit 11 acquires image data to be processed (S4). The image data may be acquired from another device via the communication unit 12, or may be acquired from the storage unit 15. The control unit 11 acquires video data including image data G1 shown in FIG. 6(A) for the specific example 1, acquires video data including image data G2 shown in FIG. 7(A) for the specific examples 2 and 3, and acquires video data including image data G3 shown in FIG. 8(A) for the specific example 4.

[0021] The control unit 11 determines a determined virtual projector from among a plurality of virtual projectors set in advance according to the shooting conditions of the image data (S5). The control unit 11 determines a determined virtual projector from among a plurality of virtual projectors set in advance according to the shooting conditions added to the image data in the process of S5. The control unit 11 determines a fisheye virtual projector as the determined virtual projector when the shooting conditions include a condition in which the image was shot using a fisheye lens as in Specific Examples 2 and 3. The control unit 11 determines a standard virtual projector as the determined virtual projector when the shooting conditions include a condition in which the image was shot using a standard lens as in Specific Example 1. The control unit 11 determines a 360-degree virtual projector as the determined virtual projector when the shooting conditions include a condition in which the image was shot using a 360-degree camera (omnidirectional camera) as in Specific Example 4. Since the shader program to be applied to the image data is determined by the process of S5, in S42 described later, the control unit 11 transmits the setting of the determined virtual projector determined in S5 to the GPU.

[0022] The control unit 11 acquires model parameters including the size and shape of the three-dimensional projection surface F, which is a concave surface, and sets a three-dimensional model including the three-dimensional projection surface (S6). In the specific examples 1, 2, and 4, the control unit 11 sets a three-dimensional model U1, and in the specific example 3, the control unit 11 sets a three-dimensional model U2.

[0023] The control unit 11 acquires the arrangement of the determined virtual projector in the three-dimensional model as a predetermined position (S7). The predetermined position may be set in advance, may be set by the user, or may be set by the process of S23. The predetermined position in this embodiment is a position where the optical axis W of the lens of the determined virtual projector is perpendicular to the first surface of the model projection surface. The predetermined position is a position where the optical axis W of the lens of the determined virtual projector passes through the center of the first surface of the model projection surface. The distance between the first surface and the determined virtual projector is appropriately set according to the shape of the three-dimensional projection surface. As shown in FIG. 3(C), the predetermined positions in the specific examples 1, 2, and 4 are positions where the optical axis W of the lens L of the determined virtual projector PR is perpendicular to the first surface UF1 of the model projection surface UF and passes through the center of the first surface UF1. The model projection surface UF is a surface that represents the three-dimensional projection surface F by the three-dimensional model U1. Surfaces UF1 to UF5 of the model projection surface UF correspond to surfaces F1 to F5 of the three-dimensional projection surface F, respectively. As shown in Fig. 3(D), the predetermined position in the specific example 3 is a position where the optical axis W of the lens L of the determined virtual projector PR is perpendicular to the first surface UK1 of the model projection surface UK and passes through the center of the first surface UK1. The model projection surface UK is a surface that represents the three-dimensional projection surface K by the three-dimensional model U2. The surfaces UK1 to UK5, UC1, and UC2 of the model projection surface UK correspond to the surfaces K1 to K5, C1, and C2 of the three-dimensional projection surface K, respectively.

[0024] The control unit 11 acquires the installation position of the projector 3 having the fisheye lens 4 with respect to the three-dimensional projection surface (S8). The installation position of the projector 3 with respect to the three-dimensional projection surface is acquired in three-dimensional coordinates of the world coordinate system. The control unit 11 acquires the lens characteristics of the fisheye lens 4 (S9). The lens characteristics of the fisheye lens 4 include a parameter corresponding to the focal length of the fisheye lens 4, a parameter indicating lens distortion, and a parameter related to the optical center. The control unit 11 executes a conversion process to project the image data acquired in S4 into two-dimensional image data to be projected by the projector 3 (S11). The image data in this embodiment is video data, and the conversion process is executed for each frame included in the video data.

[0025] As shown in FIG. 5, the control unit 11 acquires image data of a current frame from the video data as image data to be processed (S41). The control unit 11 transmits various parameters to the GPU 22 for obtaining two-dimensional image data to be used for projection from the image data (S42). The control unit 11 converts a projection result obtained by virtually projecting the image data onto a three-dimensional projection surface using the determined virtual projector determined in S5 based on the parameters transmitted in S42 into two-dimensional image data obtained by virtually photographing the image data using a fisheye lens arranged at the installation position acquired in S8 (S43 to S45). Specifically, the control unit 11 virtually photographs the model projection surface using a fisheye lens N arranged at a position corresponding to the installation position based on the installation position of the projector 3 relative to the three-dimensional projection surface acquired in S8, and acquires two-dimensional coordinate data expressing the model projection surface in two-dimensional coordinates (S43). As shown in FIG. 3(C), in the specific examples 1, 2, and 4, the control unit 11 acquires two-dimensional coordinate data when the model projection plane UF is virtually photographed using the fisheye lens N of the virtual camera CA. The virtual fisheye lens N is disposed at a position corresponding to the installation position of the projector 3 with respect to the three-dimensional projection plane F. That is, the relative position of the projector 3 with respect to the three-dimensional projection plane F is the same as the relative position of the virtual camera CA with respect to the model projection plane UF. Similarly, as shown in FIG. 3(D), in the specific example 3, the control unit 11 acquires two-dimensional coordinate data when the model projection plane UK is virtually photographed using the fisheye lens N of the virtual camera CA. It is preferable that the virtual fisheye lens N has the same or similar lens characteristics as the fisheye lens 4 of the projector 3. The virtual fisheye lens N is disposed at a position corresponding to the installation position of the projector 3 with respect to the three-dimensional projection plane K. That is, the relative position of the projector 3 with respect to the three-dimensional projection plane K is the same as the relative position of the virtual camera CA with respect to the model projection plane UK.

[0026] The process of S43 is performed by using a vertex shader provided in advance in the GPU 22 (S43). The vertex shader converts information of a three-dimensional polygon indicating a three-dimensional model into a rectangular parallelepiped coordinate system. The control unit 11 usually generates a two-dimensional image from the projection result of the three-dimensional model by a central projection (perspective projection) method in the vertex shader. The control unit 11 obtains two-dimensional coordinate data suitable for the projector 3 having the fisheye lens 4 by replacing the central projection method with an equidistant projection method according to the lens characteristics of the fisheye lens. In normal shooting by perspective projection, the coordinates of a two-dimensional image are calculated by one matrix multiplication per vertex of a three-dimensional model by a perspective transformation matrix. On the other hand, in a vertex shader using equidistant projection, the coordinates of a two-dimensional image cannot be obtained by a simple matrix calculation. The control unit 11 of this embodiment performs the process of S43 by using a three-dimensional programming API (shader program). In S43, the control unit 11 may eliminate waste of information by processing such as back face culling, clipping, attribute evaluation, and rasterization. By the process of S43, the control unit 11 can determine the coordinates of each vertex on the three-dimensional projection surface of the two-dimensional coordinate data based on the vertex data for the projection result of the three-dimensional model.

[0027] The control unit 11 performs a rasterization process on the vertex data to which the process of S43 has been applied (S44). In the rasterization process, the control unit 11 combines each vertex data into polygons that represent the deformed model data of the three-dimensional model, and determines the arrangement of pixels for drawing each polygon on the two-dimensional image data.

[0028] The control unit 11 generates two-dimensional image data based on the projection result when the image data is virtually projected onto the model projection surface using the determined virtual projector arranged at a predetermined position with respect to the model projection surface and the two-dimensional coordinate data acquired in S43 (S45). The two-dimensional image data is an image signal to be output to the projector 3. The control unit 11 of this embodiment generates two-dimensional image data in which color information is set in the two-dimensional coordinate data based on the color information of the image data and the correspondence between the image data and the two-dimensional coordinate data. The control unit 11 applies pixel shader processing to each pixel to which the processing of S44 is applied. In the pixel shader processing, the correspondence between the image data and the two-dimensional coordinate data is specified using the determined virtual projector determined in S5. If the determined virtual projector is a standard virtual projector, the control unit 11 specifies the correspondence between the image data and the two-dimensional coordinate data based on the framework information of the three-dimensional model. When the determined virtual projector is a fish-eye virtual projector or a 360-degree virtual projector, the control unit 11 performs calculations to map the projection position on the three-dimensional model when image data is projected radially from the virtual projector arranged in a predetermined position and with a predetermined attitude acquired in S7. The predetermined attitude in this embodiment is an attitude in which the optical axis W is perpendicular to the first surface of the model projection surface. In this way, the control unit 11 specifies the correspondence between the image data and the two-dimensional coordinate data of the image data captured using the fish-eye lens. The fish-eye virtual projector and the 360-degree virtual projector have different calculation methods for the projection range. In the 360-degree virtual projector, the control unit 11 calculates the elevation and depression angles and azimuth angles for each point on the three-dimensional model, with the predetermined position and attitude of the virtual projector as the origin. The control unit 11 can obtain the texture coordinates of the image data captured by the 360-degree camera from the elevation and depression angles and azimuth angles. In this way, the control unit 11 identifies the correspondence between the image data and the two-dimensional coordinate data of the image data captured using the 360-degree camera.

[0029] The control unit 11 generates two-dimensional image data by assigning the color and transparency of the two-dimensional coordinate data to each pixel based on the color information of the image data and the correspondence between the image data and the two-dimensional coordinate data. That is, the control unit 11 assigns the color and transparency at the position of the texture coordinates included in the vertex data of the vertex in the texture image as the color and transparency of the pixel corresponding to the vertex of the polygon. The color and transparency of the pixel not corresponding to the vertex of the polygon is determined by interpolating the color and transparency assigned to the vertex of the polygon. The control unit 11 may apply processes such as an alpha test, a depth test, a stencil test, and blending in addition to determining the color and transparency of the pixel by a pixel shader.

[0030] The control unit 11 outputs the two-dimensional image data image signal to the projector 3, and projects the image data onto the three-dimensional projection surface using the projector 3 (S12). The projector 3 projects the image data onto the three-dimensional projection surface according to the image signal output from the projection control device 2. In the specific example 1, the projector 3 projects the two-dimensional image data of the image data G1 shown in FIG. 6(A) onto the three-dimensional projection surface F, and obtains a projection result that reproduces the projection result P1 shown in FIG. 6(B). The projection result P1 is obtained by projecting the image data G1 along the three-dimensional projection surface F. The projection result of the image data projected onto the fourth surface F4 and the fifth surface F5 is a U-shape that is open in the direction opposite to the direction from the projector 3 toward the first surface F1. The center of the image data coincides with the center of the first surface F1. In the projection result P2 of the comparative example shown in FIG. 6(C), the distortion of the fisheye lens of the determined virtual projector is included in the projection result P2, and the contour of the projection result P2 is curved. In contrast, the projection result P1 of concrete example 1 does not include distortion of the fisheye lens, and the contour of the projection result P1 is linear, resulting in a projection result with less distortion and less discomfort compared to the projection result P1.

[0031] In specific example 2, the projector 3 projects two-dimensional image data of the image data G2 onto the three-dimensional projection surface F, and obtains a projection result that reproduces the projection result P3 shown in FIG. 7(B). In specific example 3, the projector 3 projects two-dimensional image data of the image data G2 shown in FIG. 7(A) onto the three-dimensional projection surface K, and obtains a projection result that reproduces the projection result P4 shown in FIG. 7(C). The three-dimensional projection surface K of the projection result P4 has curved surfaces at the connection portion between the horizontally adjacent first surface K1 and second surface K2, and the connection portion between the first surface K1 and third surface K3, and does not include corners in the horizontal direction. However, in both of the projection results P3 and P4, regardless of the shape of the three-dimensional projection surface, the projection results are less distorted and less unnatural. In specific example 4, the projector 3 projects two-dimensional image data of the image data G3 onto the three-dimensional projection surface F, and obtains a projection result that reproduces the projection result P5 shown in FIG. 8(B). As shown in Fig. 8(B), in specific example 4, the two-dimensional image data of the image data G3 shown in Fig. 8(A) is projected onto the three-dimensional projection surface F, and a projection result that reproduces the projection result P5 shown in Fig. 8(B) is obtained. As in specific examples 1 to 4, the projection system 1 obtains a projection result with less discomfort than in the past for each of a plurality of types of image data with different shooting conditions. The control unit 11 determines whether to end the projection control process (S31). The control unit 11 determines to end the projection control process when the projection of the last frame is completed or when an instruction to end the projection control process inputted from the user via the input unit 14 is detected.

[0032] If an instruction to end the projection control process is not detected (S31: NO), the control unit 11 judges whether a position change instruction to change the projection position has been detected (S32). If the projection position is changed, the correspondence between the representative point of the three-dimensional projection surface and the representative point of the image data is changed. The representative point is, for example, a center point. If the user wants to change the projection position, the user operates the joystick of the input unit 14 to specify the desired projection position. For example, if the user wants to move the projection result to the right from the current position, the user tilts the joystick to the right. If a position change instruction is detected (S32: YES), the control unit 11 changes the setting of the projection position of the image data on the model projection surface according to the position change instruction detected in S32 (S33). In S45 executed in S11 after the process of S33, the control unit 11 generates two-dimensional image data when the image data is virtually projected on the model projection surface at the projection position specified by the position setting instruction using the determined virtual projector arranged at a predetermined position. If the determined virtual projector is a standard virtual projector, control unit 11 changes the texture pasting position of the image data acquired in S41 up, down, left, or right in response to the position change instruction. If the determined virtual projector is a fish-eye virtual projector or a 360-degree virtual projector, control unit 11 changes the direction (angle) of the virtual projector in response to the position change instruction. As shown in Fig. 9(B), for the projection result P6 of specific example 1, if an instruction to move the projection result to the right is detected in S32 (S32: YES), the projection result P7 shown in Fig. 9(A) is projected in S12.

[0033] If the position change instruction is not detected (S32: NO), or after S33, the control unit 11 judges whether a magnification change instruction to change the projection magnification is detected (S34). If the user wants to change the projection magnification, the user operates the cross key of the input unit 14 to specify a desired projection magnification. For example, if the user wants to increase the projection magnification, the user operates to press the up key of the cross key. If a magnification change instruction is detected (S34: YES), the control unit 11 changes the setting of the projection magnification of the image data (S35). In S41, which is executed in S11 after the process of S35, the control unit 11 obtains a projection result when the image data is virtually projected on the model projection plane at the projection magnification specified by the magnification change instruction, using the determined virtual projector arranged at a predetermined position. As shown in FIG. 9, if an instruction to increase the projection magnification is detected in S32 for the projection result P6 of the specific example 1 (S32: YES), the projection result P8 shown in FIG. 9(C) is projected in S12. If an instruction to change the projection magnification is not detected (S34: NO), or after S35, the control unit 11 returns the process to S11. If an instruction to end the projection control process is detected (S31: YES), the control unit 11 stores the current setting in the storage unit 15, and ends the process.

[0034] In the above embodiment, the projection system 1, the projection control device 2, the projector 3, and the fisheye lens 4 are examples of the projection system, the projection control device, the projector, and the fisheye lens of the present invention, respectively. The process of S6 is an example of the model setting process of the present invention. The process of S4 is an example of the image acquisition process of the present invention. The process of S5 is an example of the virtual projector determination process of the present invention. The process of S43 is an example of the two-dimensional coordinate data conversion process of the present invention. The process of S45 is an example of the two-dimensional image data generation process of the present invention. The process of S21 is an example of the reference projection process of the present invention. The process of S22 is an example of the reception process of the present invention. The process of S23 is an example of the correction process of the present invention. The process of S32 is an example of the position change instruction acquisition process of the present invention. The process of S34 is an example of the magnification change instruction acquisition process of the present invention.

[0035] The projection system 1 of the embodiment includes a projector 3 having a fisheye lens 4 and a projection control device 2 that controls an image projected by the projector 3. The projection control device 2 executes an image acquisition process to acquire image data to be processed (S4). The projection control device 2 executes a model setting process to acquire model parameters including the size and shape of a three-dimensional projection surface that is a concave surface, and to set a three-dimensional model (S6). The projection control device 2 executes a virtual projector determination process to determine a determined virtual projector from among a plurality of virtual projectors that have been set in advance according to the shooting conditions of the image data (S5). The projection control device 2 executes a two-dimensional coordinate data conversion process to virtually capture a model projection surface that represents the three-dimensional projection surface as a three-dimensional model, using a fisheye lens arranged at a position according to the installation position of the projector 3, and acquire two-dimensional coordinate data of the model projection surface (S43). The projection control device 2 executes a two-dimensional image data generation process to generate two-dimensional image data based on the projection result and the two-dimensional coordinate data when image data is virtually projected onto the model projection surface, using the determined virtual projector arranged at a predetermined position with respect to the model projection surface (S45). The projection control device 2 executes an image projection process in which the two-dimensional image data is projected onto a three-dimensional projection surface using the projector 3 (S12).

[0036] Usually, when an image is projected by a projector having a fisheye lens, distortion specific to the fisheye lens occurs in the projected image. Similarly, when an image is captured by a camera having a fisheye lens, distortion specific to the fisheye lens occurs in the captured image. Here, since the capture and projection are reverse conversions, when image data captured by a fisheye lens is projected by a fisheye lens having the same lens characteristics as the fisheye lens used for the capture, the distortion at the time of capture is offset by the projection. As a result, a projected image with less distortion can be obtained without correction specific to the fisheye lens. The projection control device 2 of the projection system 1 simulates an ideal projection environment according to the capture conditions by using a virtual projector according to the image data of each of a plurality of capture conditions, and converts the projection result into two-dimensional image data when the image is captured using the fisheye lens N from the actual installation position of the projector 3. As a result, the projection control device 2 of the projection system 1 can generate two-dimensional image data capable of reproducing an ideal projection environment by projection from a realistic projection environment according to the capture conditions through a relatively simple process. In other words, the projection control device 2 can perform planar projection and three-dimensional projection according to the capture conditions within the same device. Therefore, based on image data for each of a plurality of shooting conditions, the projection system 1 can obtain a projection image with less distortion than conventional methods, which is less uncomfortable for the viewer and is less likely to appear strange to the viewer regardless of the installation position of the projector 3, even if the projection surface has an arbitrary three-dimensional shape, through simpler processing than conventional methods.

[0037] The virtual projector determination process of S5 is a process for determining, as the determined virtual projector, a fisheye virtual projector that virtually projects image data along a model projection surface using a virtual fisheye lens when the shooting conditions include a condition where the image was shot using a fisheye lens. The virtual projector determination process is a process for determining, as the determined virtual projector, a standard virtual projector that virtually projects image data along a model projection surface when the shooting conditions include a condition where the image was shot using a standard lens. The projection system 1 can project each of the image data shot using a fisheye lens and the image data shot using a standard lens from the projector 3 onto a three-dimensional projection surface consisting of a concave surface with less distortion than in the past.

[0038] The model setting process of S6 is a process of acquiring model parameters including a three-dimensional projection surface having five surfaces including a first surface and four surfaces adjacent to the first surface, and setting up a three-dimensional model (S7). The projection system 1 sets up a three-dimensional model with relatively simple settings, and can obtain a projection image that gives a viewer little discomfort, regardless of the shooting conditions of the image data and the installation position of the projector 3, with a process simpler than conventionally possible.

[0039] The model setting process of S6 is a process of acquiring model parameters including a three-dimensional projection surface having five surfaces including a first surface facing the fisheye lens and four surfaces continuous with the first surface, and setting a three-dimensional model. The predetermined position is a position where the optical axis W of the lens of the determined virtual projector is perpendicular to the first surface. Since the projection system 1 sets the predetermined position as a position facing the first surface of the three-dimensional projection surface, it is easier to simulate an ideal projection environment compared to other conditions where the predetermined position is not facing the first surface, and the projection result acquisition process can be facilitated.

[0040] The model setting process of S6 is a process of acquiring model parameters including a three-dimensional projection surface having a first surface facing the fisheye lens 4 of the projector 3, and setting a three-dimensional model. The predetermined position is a position where the optical axis W of the lens L of the determined virtual projector PR passes through the center M of the first surface of the three-dimensional projection surface. The predetermined position of the projection system 1 is located on an extension of a virtual line extending from the center of the first surface of the three-dimensional projection surface in a direction perpendicular to the first surface. Therefore, the projection system 1 can easily simulate an ideal projection environment compared to other conditions, and can easily perform the process of virtually projecting image data onto the three-dimensional projection surface.

[0041] The model setting process of S6 is a process of acquiring model parameters including a three-dimensional projection surface having a first surface facing the fisheye lens 4 of the projector 3 and extending in the vertical direction, a second surface and a third surface intersecting the first surface and extending in the vertical direction, and a fourth surface and a fifth surface intersecting the first surface and extending in the horizontal direction, and setting a three-dimensional model. The first surface and the second surface may be connected by a first curved surface, and the first surface and the third surface may be connected by a second curved surface. In general, in a three-dimensional projection surface, image distortion is more likely to occur at the joints between the first surface and the second surface and the joints between the first surface and the third surface, which are corners, than in a flat surface. In the projection system 1, distortion is less likely to occur at the joints between the first surface and the second surface and the joints between the first surface and the third surface in the projection result projected onto the three-dimensional projection surface, compared to a case in which the three-dimensional projection surface does not include the first curved surface and the second curved surface.

[0042] The image data is added with shooting conditions. The virtual projector determination process is a process for determining a determined virtual projector from among a plurality of virtual projectors according to the shooting conditions added to the image data. The projection system 1 can determine a determined virtual projector based on the shooting conditions added to the image data, which can save the user of the projection system 1 the trouble of inputting the shooting conditions.

[0043] The two-dimensional image data generation process is a process for generating two-dimensional image data in which color information is set in two-dimensional coordinate data based on color information of the image data and the correspondence between the image data and two-dimensional coordinate data (S45). The projection system 1 can generate two-dimensional image data in which color information is set in two-dimensional coordinate data through a relatively simple process based on color information of the image data and the correspondence between the image data and two-dimensional coordinate data.

[0044] The projection control device 2 executes a reference projection process to project a reference image showing the setting status of the three-dimensional model onto the three-dimensional projection surface via the fisheye lens 4 (S21). After starting projection of the reference image, the projection control device 2 executes a reception process to receive a setting change instruction to change the setting of the three-dimensional model (S22). In response to the setting change instruction, the projection control device 2 executes a correction process to change the setting of the three-dimensional model and correct the reference image to be projected onto the three-dimensional projection surface (S23). The projection system 1 can change the setting of the three-dimensional model by a relatively simple process while projecting the reference image onto the actual three-dimensional projection surface. A user of the projection system 1 can check the setting status of the three-dimensional model by the reference image projected onto the three-dimensional projection surface.

[0045] The projection control device 2 executes a position change instruction acquisition process to acquire a position change instruction to change the projection position of the image data on the model projection surface (S32). The two-dimensional image data generation process when the position change instruction is acquired is a process to generate two-dimensional image data when the image data is virtually projected onto the projection position instructed by the position change instruction on the model projection surface. The projection system 1 can project the image data of the projection position whose setting has been changed onto the three-dimensional projection surface by a relatively simple process.

[0046] The projection control device 2 executes a magnification change instruction acquisition process to acquire a magnification change instruction to change the projection magnification of the image data (S34). The two-dimensional image data generation process when the magnification change instruction is acquired is a process to generate two-dimensional image data when the image data is virtually projected on the model projection plane at the projection magnification instructed by the magnification change instruction. The projection system 1 can project the image data with the projection magnification setting changed on the three-dimensional projection plane by a relatively simple process.

[0047] The projection system, projection control device, and projection control program of the present invention are not limited to the above-described embodiment, and various modifications may be made without departing from the spirit and scope of the present invention. For example, the following modifications may be made as appropriate.

[0048] The configurations of the projection system, the projection control device, and the projection control program may be changed as appropriate. The projector 3 and the projection control device 2 may be an integrated device. The projection control device may be a dedicated device or a general-purpose PC, etc.

[0049] The projection control program including instructions for executing the projection control process may be stored in the storage unit 15 or memory 23 before the control unit 11 of the projection control device 2 executes the projection control program. Therefore, the method of acquiring the projection control program, the acquisition path, and the device that stores the program may each be changed as appropriate. The projection control program may be received from another device via a cable or wireless communication and stored in the storage unit 15 or memory 23. The other device may include, for example, a PC and a server connected via a network.

[0050] Each step of the projection control process is not limited to being executed by the control unit 11, and may be executed in part or in whole by other electronic devices (e.g., ASIC). Each step of the projection control process may be distributed and processed by multiple electronic devices (e.g., multiple CPUs). The order of each step of the projection control process may be changed, steps may be omitted, or steps may be added, as necessary. The following modifications may be made as appropriate to the processing executed by the projection system 1, including the projection control processing.

[0051] The shape of the three-dimensional projection surface may be a concave surface, and may be appropriately changed to a hemispherical surface, etc. The process of changing the three-dimensional model may be appropriately changed. When the projection control device 2 includes a sensor that measures the distance and shape to the three-dimensional projection surface, the projection control device 2 may automatically set the three-dimensional model according to the detection result of the sensor. The pattern of the reference image may be appropriately changed, and may be, for example, a pattern that represents only the shape of the first surface. The projection control device 2 may not be able to execute the process of changing the three-dimensional model using the reference image. The changeable items of the three-dimensional model may be appropriately changed, and may not include, for example, a function for correcting to a trapezoidal surface, or a function for correcting the connection part of two adjacent surfaces among the multiple surfaces that constitute the three-dimensional projection surface to a curved surface. The connection part of two adjacent surfaces in the vertical direction among the multiple surfaces that constitute the three-dimensional projection surface may be a curved surface. When the size and shape of the three-dimensional projection surface are known, such as the screen E, the three-dimensional model may be set based on model parameters including the size and shape of the three-dimensional projection surface input by the user.

[0052] The preset multiple types of virtual projectors may be changed as appropriate. Any two of the standard virtual projector, fish-eye virtual projector, and 360-degree virtual projector in the above embodiment may be set as multiple types of virtual projectors. The multiple types of virtual projectors may include projectors other than the standard virtual projector, fish-eye virtual projector, and 360-degree virtual projector. The image data may not have the shooting conditions added thereto, and the control unit 11 may set the virtual projector selected by the user as the determined virtual projector. The control unit 11 may determine the determined virtual projector according to the aspect ratio of the image. The predetermined position of the determined virtual projector may not be a position where the optical axis of the lens of the determined virtual projector is perpendicular to the first surface of the model projection surface, and may not be a position where the optical axis of the lens of the determined virtual projector passes through the center of the first surface of the model projection surface. Each of S32 to S35 may be omitted as appropriate. In the two-dimensional image data generation process in S45, it is sufficient that the two-dimensional image data is generated based on the projection result and the two-dimensional coordinate data acquired in S43, and the method of generating the two-dimensional image data may be changed as appropriate.

[0053] The above-mentioned modified examples may be appropriately combined to the extent that there are no contradictions. In addition to the combinations exemplified in the claims, the applicant intends to obtain patent rights for aspects in which the invention is combined with other aspects to the extent that the invention does not deviate from the gist of the invention and does not cause contradictions. [Explanation of symbols]

[0054] 1: projection system, 2: projection control device, 3: projector, 4: fisheye lens, 11: control unit, 12: communication unit, 14: input unit, 15: storage unit, 21: CPU, 22: GPU, 23: memory

Claims

1. A projection system including a projector having a fisheye lens and a projection control device that controls an image projected by the projector, The projection control device includes: an image acquisition process for acquiring image data to be processed; a model setting process for acquiring model parameters including a size and a shape of a three-dimensional projection surface, which is a concave surface, and setting a three-dimensional model; a virtual projector determination process for determining a virtual projector from among a plurality of virtual projectors set in advance according to a shooting condition of the image data; a two-dimensional coordinate data conversion process for virtually capturing an image of a model projection plane, which represents the three-dimensional projection plane using the three-dimensional model, by using a fisheye lens disposed at a position corresponding to an installation position of the projector, and acquiring two-dimensional coordinate data, which represents the model projection plane using two-dimensional coordinates; a two-dimensional image data generation process for generating two-dimensional image data based on a projection result when the image data is virtually projected onto the model projection plane using the determined virtual projector arranged at a predetermined position with respect to the model projection plane and the two-dimensional coordinate data; an image projection process of projecting the two-dimensional image data onto the three-dimensional projection surface using the projector; 13. A projection system comprising:

2. The virtual projector determination process includes: When the photographing conditions include a condition in which the photographing was performed using a fisheye lens, a fisheye virtual projector that virtually projects the image data along the model projection surface using a virtual fisheye lens is determined as the determined virtual projector; 2. The projection system according to claim 1, wherein when the shooting conditions include a condition in which the image was shot using a standard lens, a standard virtual projector that virtually projects the image data along the model projection surface is determined as the determined virtual projector.

3. The projection system according to claim 1 or 2, characterized in that the model setting process is a process of acquiring the model parameters including the three-dimensional projection surface having five surfaces including a first surface and four surfaces contiguous to the first surface, and setting the three-dimensional model.

4. the model setting process is a process of acquiring the model parameters including the three-dimensional projection surface having five surfaces including a first surface facing the fisheye lens and four surfaces continuous with the first surface, and setting the three-dimensional model; 3. The projection system according to claim 2, wherein the predetermined position is a position where an optical axis of a lens of the determined virtual projector intersects perpendicularly with one surface of the model projection surface.

5. the model setting process is a process of acquiring the model parameters including the three-dimensional projection surface having a first surface facing the fisheye lens of the projector, and setting the three-dimensional model; 3. The projection system according to claim 2, wherein the predetermined position is a position where an optical axis of a lens of the determined virtual projector passes through a center of a first surface of the model projection plane.

6. the model setting process is a process of acquiring the model parameters including the three-dimensional projection surface having a first surface facing the fisheye lens of the projector and extending in a vertical direction, second and third surfaces intersecting with the first surface and extending in the vertical direction, and fourth and fifth surfaces intersecting with the first surface and extending in a horizontal direction, and setting the three-dimensional model; 3. The projection system according to claim 1, wherein the first surface and the second surface are connected by a first curved surface, and the first surface and the third surface are connected by a second curved surface.

7. The image data is added with the shooting conditions, 3. The projection system according to claim 1, wherein the virtual projector determination process is a process of determining the determined virtual projector from among the plurality of virtual projectors in accordance with the shooting conditions added to the image data.

8. The two-dimensional image data generation process includes:

2. The projection system according to claim 1, further comprising a process for generating the two-dimensional image data in which color information is set in the two-dimensional coordinate data based on color information of the image data and a correspondence between the image data and the two-dimensional coordinate data.

9. The projection control device includes: a reference projection process for projecting a reference image indicating a setting status of the three-dimensional model onto the three-dimensional projection surface via the fisheye lens; a reception process for receiving a setting change instruction for changing a setting of the three-dimensional model after starting projection of the reference image; a correction process of changing a setting of the three-dimensional model in response to the setting change instruction and correcting the reference image to be projected on the three-dimensional projection surface; 2. The projection system of claim 1, further comprising:

10. The projection control device includes: A position change instruction acquisition process is further performed to acquire a position change instruction for changing a projection position of the image data on the model projection plane; The projection system according to claim 1, characterized in that the two-dimensional image data generation process when the position change instruction is acquired is a process of generating the two-dimensional image data when the image data is virtually projected onto the model projection plane at the projection position indicated by the position change instruction.

11. The projection control device includes: A magnification change instruction acquisition process is further performed to acquire a magnification change instruction for changing the projection magnification of the image data; The projection system according to claim 1, characterized in that the two-dimensional image data generation process when the magnification change instruction is acquired is a process of generating the two-dimensional image data when the image data is virtually projected on the model projection plane at the projection magnification instructed by the magnification change instruction.

12. A projection control device that controls an image projected by a projector having a fisheye lens, an image acquisition process for acquiring image data to be processed; a model setting process for acquiring model parameters including a size and a shape of a three-dimensional projection surface, which is a concave surface, and setting a three-dimensional model; a virtual projector determination process for determining a virtual projector from among a plurality of virtual projectors set in advance according to a shooting condition of the image data; a two-dimensional coordinate data conversion process for virtually capturing an image of a model projection plane, which represents the three-dimensional projection plane using the three-dimensional model, by using a fisheye lens disposed at a position corresponding to an installation position of the projector, and acquiring two-dimensional coordinate data, which represents the model projection plane using two-dimensional coordinates; a two-dimensional image data generation process for generating two-dimensional image data based on a projection result when the image data is virtually projected onto the model projection plane using the determined virtual projector arranged at a predetermined position with respect to the model projection plane and the two-dimensional coordinate data; a video projection process for projecting the two-dimensional image data from the projector; A projection control device comprising:

13. A projection control program to be executed by a projection control device that controls an image projected by a projector having a fisheye lens, The projection control device includes: an image acquisition process for acquiring image data to be processed; a model setting process for acquiring model parameters including a size and a shape of a three-dimensional projection surface, which is a concave surface, and setting a three-dimensional model; a virtual projector determination process for determining a virtual projector from among a plurality of virtual projectors set in advance according to a shooting condition of the image data; a two-dimensional coordinate data conversion process for virtually capturing an image of a model projection plane, which represents the three-dimensional projection plane using the three-dimensional model, by using a fisheye lens disposed at a position corresponding to an installation position of the projector, and acquiring two-dimensional coordinate data, which represents the model projection plane using two-dimensional coordinates; a two-dimensional image data generation process for generating two-dimensional image data based on a projection result when the image data is virtually projected onto the model projection plane using the determined virtual projector arranged at a predetermined position with respect to the model projection plane and the two-dimensional coordinate data; a video projection process for projecting the two-dimensional image data from the projector; A projection control program comprising instructions for executing the above.

Images