Projection system, projection control device, and projection control program

Abstract

Provided are a projection system, a projection control device, and a projection control program for projecting a video in which distortion is reduced more than before on a three-dimensional projection surface composed of a concave surface from a projector on the basis of image data conforming to each of a plurality of imaging conditions.　This projection control device acquires image data (S4), acquires a model parameter to set a three-dimensional model (S6), and determines a determined virtual projector from among a plurality of preset virtual projectors in accordance with an imaging condition for the image data (S5). The projection control device generates two-dimensional image data on the basis of a projection result from when image data is projected, using the determined virtual projector disposed at a prescribed position, onto a model projection surface in which a three-dimensional projection surface is represented by a three-dimensional model, and two-dimensional coordinate data in which the model projection surface is represented by two-dimensional coordinates (S11), and projects the two-dimensional image data onto the three-dimensional projection surface using a projector (S12).

Description

Projection system, projection control device, and projection control program

[0001] The present invention relates to a projection system, a projection control device, and a projection control program.

[0002] The projection method described in Patent Document 1 assumes a projected image when projected onto a projection surface so that the center of the dome-shaped projection surface, with its concave surface facing the viewer, is the center of the projected image.The image to be projected is transformed so that the assumed projected image matches the image projected by a projector installed at an arbitrary position.

[0003] Japanese Patent Application Publication No. 9-149351

[0004] In conventional projection methods, the image to be projected must be modified by adjusting the waveforms of the horizontal and vertical scanning voltages of the projector, which requires complex processing. Furthermore, in conventional projection methods, image data is processed using a single processing method, so distortion occurs when image data is input under conditions that differ from the expected shooting conditions.

[0005] The object of the present invention is to provide a projection system, a projection control device, and a projection control program that project images with reduced distortion compared to conventional methods onto a three-dimensional projection surface consisting of a concave surface from a projector based on image data for each of a plurality of shooting conditions.

[0006] A projection system according to a first aspect of the present invention is a projection system comprising a projector having a fisheye lens and a projection control device that controls an image projected by the projector, wherein the projection control device executes an image acquisition process that acquires image data to be processed, a model setting process that acquires model parameters including the size and shape of a three-dimensional projection surface, which is a concave surface, and sets a three-dimensional model, a virtual projector determination process that determines a selected virtual projector from among a plurality of virtual projectors according to the shooting conditions of the image data, a two-dimensional coordinate data conversion process that virtually photographs a model projection surface, which represents the three-dimensional projection surface using the fisheye lens placed at the installation position of the projector, and acquires two-dimensional coordinate data that represents the model projection surface in two-dimensional coordinates, a two-dimensional image data generation process that generates two-dimensional image data based on the two-dimensional coordinate data and a projection result when the image data is virtually projected onto the model projection surface using the selected virtual projector placed at a predetermined position with respect to the model projection surface, and an image projection process that projects the two-dimensional image data onto the three-dimensional projection surface using the projector.

[0007] Typically, when an image is projected using a projector with a fisheye lens, distortion specific to the fisheye lens occurs in the projected image. Similarly, when an image is captured using a camera with a fisheye lens, distortion specific to the fisheye lens occurs in the captured image. Here, since capture and projection are reverse transformations, when image data captured with a fisheye lens is projected using a fisheye lens with the same lens characteristics as the fisheye lens used for capture, the distortion caused by the capture is canceled out by the projection, resulting in a projected image with less distortion without the need for correction specific to the fisheye lens. A projection system of a first aspect simulates an ideal projection environment corresponding to the capture conditions using a virtual projector corresponding to image data for each of multiple capture conditions. The projection system converts the projection results in the ideal projection environment into two-dimensional image data captured using a fisheye lens from the actual projector installation position. This allows the projection control device of the projection system to generate two-dimensional image data capable of reproducing the ideal projection environment by projecting from a realistic projection environment through relatively simple processing. In other words, the projection control device of the projection system can perform two-dimensional projection and three-dimensional projection corresponding to the capture conditions within the same device. Therefore, the projection system can obtain, based on image data for each of a plurality of shooting conditions, a projected image with less distortion than conventional methods, which is less unnatural to the viewer and has less distortion than conventional methods, regardless of the installation position of the projector, even if the projection surface has an arbitrary three-dimensional shape, through simpler processing than conventional methods.

[0008] A second aspect of the present invention provides a projection control device that includes a control unit and controls an image projected by a projector having a fisheye lens, wherein the control unit executes the following operations: an image acquisition process for acquiring image data to be processed; a model setting process for acquiring model parameters including the size and shape of a concave three-dimensional projection surface and setting a three-dimensional model; a virtual projector determination process for determining a selected virtual projector from among a plurality of virtual projectors according to the image data capture conditions; a two-dimensional coordinate data conversion process for virtually capturing an image of a model projection surface, which represents the three-dimensional projection surface using the fisheye lens located at the installation position of the projector, to acquire two-dimensional coordinate data representing the model projection surface in two-dimensional coordinates; a two-dimensional image data generation process for generating two-dimensional image data based on the two-dimensional coordinate data and a projection result when the image data is virtually projected onto the model projection surface using the selected virtual projector located at a predetermined position relative to the model projection surface; and an image projection process for projecting the two-dimensional image data from the projector. The projection control device of the second aspect achieves the same effects as the projection system of the first aspect.

[0009] A projection control program of a third aspect of the present invention is a projection control program to be executed by a projection control device that controls an image projected by a projector having a fisheye lens, and includes instructions to cause the projection control device to execute an image acquisition process that acquires image data to be processed, a model setting process that acquires model parameters including the size and shape of a three-dimensional projection surface, which is a concave surface, and sets a three-dimensional model, a virtual projector determination process that determines a determined virtual projector from among a plurality of virtual projectors according to the shooting conditions of the image data, a two-dimensional coordinate data conversion process that uses a fisheye lens placed at an installation position of the projector to virtually photograph a model projection surface that represents the three-dimensional projection surface using the three-dimensional model, and acquires two-dimensional coordinate data that represents the model projection surface in two-dimensional coordinates, a two-dimensional image data generation process that generates two-dimensional image data based on the two-dimensional coordinate data and a projection result when the image data is virtually projected onto the model projection surface using the determined virtual projector placed at a predetermined position with respect to the model projection surface, and an image projection process that projects the two-dimensional image data from the projector. The projection control program of the third aspect, when executed by the projection control device, provides the same effects as the projection system of the first aspect.

[0010] 1A is a conceptual diagram of a projection system 1 having a projector 3 suspended from the ceiling of a room R, and FIG. 1B is an explanatory diagram of the room R in which a screen E is placed. It is a flowchart of a projection control process executed by a projection control device 2. (A) is a perspective view of a three-dimensional model U1 set in the projection control process, (B) is a perspective view of a three-dimensional model U2 set in the projection control process, (C) is an explanatory diagram of a virtual projector PR placed at a predetermined position relative to a model projection surface UF of the three-dimensional model U1 and a virtual camera CA placed at an installation position, and (D) is an explanatory diagram of a virtual projector PR placed at a predetermined position relative to a model projection surface UK of the three-dimensional model U2 and a virtual camera CA placed at an installation position. (A) is an explanatory diagram of a three-dimensional projection surface F, (B) is an explanatory diagram of a reference image Q projected onto the three-dimensional projection surface F in the projection control process, and (C) is an explanatory diagram of a reference image Q projected onto the three-dimensional projection surface F when the setting of the three-dimensional model is changed in the projection control process. 1 is a flowchart of a conversion process executed in a projection control process. (A) is an explanatory diagram of image data G1 captured using a standard lens. (B) is an explanatory diagram of a projection result P1 when two-dimensional image data generated based on the image data G1 of Specific Example 1 is projected onto a three-dimensional projection surface F. (C) is an explanatory diagram of a projection result P2 when two-dimensional image data generated based on the image data G1 of the comparative example is projected onto the three-dimensional projection surface F. (A) is an explanatory diagram of image data G2 captured using a fisheye lens. (B) is an explanatory diagram of a projection result P3 when two-dimensional image data generated based on the image data G2 of Specific Example 2 is projected onto the three-dimensional projection surface F. (C) is an explanatory diagram of a projection result P4 when two-dimensional image data generated based on the image data G2 of Specific Example 3 is projected onto the three-dimensional projection surface K. (A) is an explanatory diagram of image data G3 captured using a 360-degree camera. (B) is an explanatory diagram of a projection result P5 when two-dimensional image data generated based on the image data G of Specific Example 4 is projected onto the three-dimensional projection surface F. (A) is an explanatory diagram of the projection result P7 when the projection position of the projection result P6 in Figure 9 (B) is changed, (B) is an explanatory diagram of the projection result P6, and (C) is an explanatory diagram of the projection result P8 when the projection magnification of the projection result P6 is changed.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following description, the X direction, Y direction, and Z direction shown in FIG. 1A will be used. The Z direction is the vertical direction. The X direction and Y direction are horizontal directions perpendicular to the Z direction. The X direction, Y direction, and Z direction are directions perpendicular to each other.

[0012] As shown in Fig. 1A, the projection system 1 includes a projector 3 having a fisheye lens 4 and a projection control device 2 that controls the image projected by the projector 3. The projector 3 and the projection control device 2 are connected to each other via a wired or wireless connection. The projection system 1 is used in a room R of an amusement facility such as a karaoke bar. The projector 3 is suspended from the ceiling of the room R, for example.

[0013] The projector 3 projects image data onto a concave three-dimensional projection surface in response to an image signal output from the projection control device 2. The three-dimensional projection surface may have any shape as long as it is concave. In this embodiment, the three-dimensional projection surface is the three-dimensional projection surface F shown in FIG. 1A or the three-dimensional projection surface K shown in FIG. 1B. As shown in FIG. 1A, the three-dimensional projection surface F is composed of five of the six surfaces of the cubic room R, including a first surface F1 facing the fisheye lens 4 of the projector 3 and four surfaces F2 to F5 connected to the first surface F1. The first surface F1 extends vertically. The second surface F2 and the third surface F3 are each perpendicular to the first surface F1 and extend vertically. The fourth surface F4 and the fifth surface F5 are each perpendicular to the first surface F1 and extend horizontally. The three-dimensional projection surface K is formed on the concave surface of a screen E that can be installed in the cubic room R. As shown in FIG. 1B , the three-dimensional projection surface K is composed of five surfaces, including a first surface K1 facing the fisheye lens 4 of the projector 3 and four surfaces K2 to K5 connected to the first surface K1. The first surface K1 extends vertically. The second surface K2 and the third surface K3 each intersect with the first surface K1 and extend vertically. The fourth surface K4 and the fifth surface K5 each extend horizontally and perpendicular to the first surface K1. The first surface K1 and the second surface K2 are connected by a first curved surface C1, and the first surface K1 and the third surface K3 are connected by a second curved surface C2. To reduce the sense of incongruity when viewing the projection results from a location farther from the screen E, the distance between the second surface K2 and the third surface K3 increases as the distance from the first surface K1 increases. The screen E is positioned, for example, so that the surface opposite the first surface K1 faces and is close to the first surface F1 of the room R. In this embodiment, the first surface F1 is a wall surface of the room R, but the first surface F1 may be either the ceiling surface or the floor surface of the room R.

[0014] The projection control device 2 converts image data into two-dimensional image data for projection and controls the projection of the projector 3 based on the two-dimensional image data. The projection control device 2 may be located at any position in the room R or at a location separate from the room R. The projection control device 2 includes a communication unit 12, a memory unit 15, an input unit 14, and a control unit 11. The communication unit 12, the memory unit 15, and the input unit 14 are connected to the control unit 11. The memory unit 15 is configured, for example, by a hard disk drive. An OS and various programs are stored in the memory unit 15. The various programs include a projection control program that causes the CPU 21 of the control unit 11 to execute a projection control process. The memory unit 15 stores image data to be projected, internal and external parameters of the projector 3, internal and external parameters of the virtual projector PR, internal and external parameters of the virtual camera CA, various parameters required to execute the projection control process, and the like. Shooting conditions are added to the image data. The shooting conditions in this embodiment include the lens used for shooting, the lens characteristics used for shooting, the type of camera, etc. The input unit 14 accepts operation instructions from a user of the projection system 1 and outputs a signal corresponding to the operation instruction to the control unit 11. The input unit 14 is, for example, a game controller having a joystick and a cross key. The control unit 11 is composed of a CPU 21, a GPU 22, a memory 23, etc. as a computer. The GPU 22 is a processor provided separately from the CPU 21, and performs image processing based on instructions from the CPU 21. The memory 23 is composed of, for example, a ROM, a RAM, etc.

[0015] 2 to 9C , the projection control process will be described using specific examples 1 to 4 of the projection control device 2. In the projection control process, a process is executed to project image data specified by a user onto a three-dimensional projection surface F. In each of specific examples 1 to 4, image data is taken from each frame of video data of four people chatting on a stage. In specific examples 1 to 4, video data captured by a camera using a standard lens is projected onto the three-dimensional projection surface F. In specific example 2, video data captured by a camera using a fisheye lens is projected onto the three-dimensional projection surface F. In specific example 3, video data captured by a camera using a fisheye lens is projected onto the three-dimensional projection surface K. The three-dimensional projection surface K has a trapezoidal planar shape with rounded corners. In specific example 4, video data captured by an omnidirectional camera is projected onto the three-dimensional projection surface F. In specific examples 1 to 4, the control unit 11 generates two-dimensional image data to be projected by the projector 3 using a virtual projector according to the capture conditions of the video data. The virtual projector is a virtually set parameter for calculating pixel correspondence between image data and a three-dimensional model. The projection control device 2 of this embodiment presets three types of virtual projectors, and stores shader programs and parameters required for virtual projection using each virtual projector in the storage unit 15. The three types of virtual projectors are a standard virtual projector, a fisheye virtual projector, and a 360-degree virtual projector. The standard virtual projector virtually projects image data by aligning the three-dimensional projection surface with a model projection surface represented by a three-dimensional model. The fisheye virtual projector virtually projects image data along the model projection surface using a virtual fisheye lens. The 360-degree projector virtually projects 360 degrees onto the model projection surface so that the elevation and depression angles and azimuth angles match when viewed from the virtual projector. The projection result of the 360-degree virtual projector is virtually projected 360 degrees, but in reality, the projection range of the projector 3 having the fisheye lens 4 indicated by the lens characteristics acquired in S9 described below is output. The type and number of multiple virtual projectors may be changed as appropriate.As a comparative example, a case will be described in which, in specific example 1, a fish-eye virtual projector is used to generate two-dimensional image data to be projected by the projector 3.

[0016] The projection control process is initiated when a user inputs a start instruction. When the control unit 11 detects the start instruction, it reads into the memory 23 a projection control program for executing the projection control process stored in the storage unit 15. The control unit 11 executes the following steps in accordance with the instructions contained in the program read into the memory 23. Various parameters necessary for executing the projection control process are stored in the storage unit 15. Various data obtained during the projection control process are stored in the memory 23 as appropriate. In the projection control process, specific examples 1 to 4 are each executed at different times.

[0017] As shown in FIG. 2 , the control unit 11 acquires the initial settings stored in the storage unit 15 (S1). The initial settings include the current 3D model settings. The current 3D model settings are expressed using model parameters including the size and shape of the 3D projection surface set in the 3D model. The control unit 11 determines whether to change the 3D model (S2). In this embodiment, the cubic 3D model U1 shown in FIG. 3 is set as the initial value of the 3D model. The projection control device 2 of this embodiment can change the shape and size of each face of the current 3D model, the shape of the curved surface when the connection between two adjacent faces of any face is curved, and the trapezoid shape when the contour shape of any face is trapezoidal. Furthermore, the control unit 11 can rotate and translate the current 3D model. Typically, a user changes the 3D model when the position of the projector 3 relative to the 3D projection surface is changed. When a user wants to change the current 3D model settings, the user operates the input unit 14 to input an instruction to change the settings.

[0018] As shown in FIG. 3A, a three-dimensional model U1 is set as the current three-dimensional model for the three-dimensional projection surface F in specific examples 1, 2, and 4. As shown in FIG. 3B, a three-dimensional model U2 is set as the current three-dimensional model for the three-dimensional projection surface K in specific example 3. The setting of the three-dimensional model U1 is changed for the three-dimensional model U2 to match the shape of the screen E. The framework information B1 is represented by a lattice pattern of the three-dimensional model U1. The framework information B2 is represented by a lattice pattern of the three-dimensional model U2. The framework information B1 and the framework information B2 are each information for mapping image data as a texture when a standard virtual projector is determined as the determined virtual projector in the processing of S5 described below. The standard virtual projector of this embodiment is used by the control unit 11 in the processing for generating two-dimensional image data by projecting image data so as to arrange the image data along the three-dimensional projection surface. The framework information indicated by the three-dimensional model will be described using the three-dimensional projection surface F as an example. The control unit 11 sets the center of the first surface F1 as the center of the image data and sets a projection magnification such that the image data extends beyond the first surface F1. For the image data that extends from the first surface F1 onto the second surface F2 and the third surface F3 that are connected to the first surface F1 in the horizontal direction, the control unit 11 projects the image data so as to bend it along the projection surface. For the image data that extends from the first surface F1 onto the fourth surface F4 and the fifth surface F5 that are connected to the first surface F1 in the vertical direction, the control unit 11 deforms the image data so as to shrink it toward the center of the surface, as shown by the grid pattern, and projects it.

[0019] When an instruction to change the settings of the three-dimensional model U1 before the setting change is detected (S2: YES), the control unit 11 uses the projector 3 to project a reference image Q showing the setting status of the three-dimensional model U1 onto the three-dimensional projection surface F via the fisheye lens 4, as shown in FIG. 4(B) (S21). The reference image Q may be any image showing the current setting status of the three-dimensional model. For example, it may be an image showing the boundary between two adjacent surfaces, or an image including a grid pattern representing the framework information B1 of the three-dimensional model U1. After starting projection of the reference image Q, the control unit 11 accepts a setting change instruction to change the settings of the three-dimensional model U1 (S22). The user operates the input unit 14 to select an item to change the setting and then specify the amount of change to input the instruction. When an instruction to change the settings of the three-dimensional model U1 is detected (S22: YES), the control unit 11 changes the settings of the three-dimensional model U1 according to the instruction accepted in S22, and changes and projects the reference image Q to be projected onto the three-dimensional projection surface F (S23). While viewing the reference image projected onto the three-dimensional projection surface F, the user operates the input unit 14 to adjust the parameters of the three-dimensional model so that the reference image Q is positioned along the three-dimensional projection surface F, as shown in FIG. 4C . In FIG. 4C , the boundary of the pattern of the reference image Q coincides with the boundary of any two adjacent surfaces among the five surfaces constituting the three-dimensional projection surface. The processes of S22 and S23 can change the positional relationship between the room R in which the projector 3 is installed and the projector 3, the fisheye lens characteristics of the fisheye lens 4 of the projector 3 acquired in S9, the predetermined position of the virtual projector acquired in S6, and the orientation of the determined virtual projector determined in S5. If an instruction to change the settings of the three-dimensional model is not detected (S22: NO), or after S23, the control unit 11 determines whether an instruction to end the process of changing the settings of the three-dimensional model has been detected (S24). If an instruction to end the process of changing the settings of the three-dimensional model is not detected (S24: NO), the control unit 11 returns to S21. If an instruction to end the process of changing the settings of the three-dimensional model is detected (S24: YES), the control unit 11 returns the process to S2.

[0020] If an instruction to change the settings of the three-dimensional model is not detected (S2: NO), the control unit 11 determines whether a projection instruction to project image data onto the three-dimensional projection surface has been detected (S3). If a projection instruction is not detected (S3: NO), the control unit 11 returns the process to S2. If a projection instruction is detected (S3: YES), the control unit 11 acquires image data to be processed (S4). The image data may be acquired from another device via the communication unit 12 or from the storage unit 15. For Specific Example 1, the control unit 11 acquires video data including image data G1 shown in FIG. 6A. For Specific Examples 2 and 3, the control unit 11 acquires video data including image data G2 shown in FIG. 7A. For Specific Example 4, the control unit 11 acquires video data including image data G3 shown in FIG. 8A.

[0021] The control unit 11 determines a determined virtual projector from among a plurality of virtual projectors preset according to the shooting conditions of the image data (S5). In the processing of S5, the control unit 11 determines a determined virtual projector from among a plurality of virtual projectors preset according to the shooting conditions added to the image data. The control unit 11 determines a fisheye virtual projector as the determined virtual projector when the shooting conditions include a condition in which the image was shot using a fisheye lens, as in Specific Examples 2 and 3. The control unit 11 determines a standard virtual projector as the determined virtual projector when the shooting conditions include a condition in which the image was shot using a standard lens, as in Specific Example 1. The control unit 11 determines a 360-degree virtual projector as the determined virtual projector when the shooting conditions include a condition in which the image was shot using a 360-degree camera (omnidirectional camera), as in Specific Example 4. In S42, which will be described later, the control unit 11 transmits the settings of the determined virtual projector determined in S5 to the GPU.

[0022] The control unit 11 acquires model parameters including the size and shape of the three-dimensional projection surface F, which is a concave surface, and sets a three-dimensional model including the three-dimensional projection surface (S6). In specific examples 1, 2, and 4, the control unit 11 sets a three-dimensional model U1, and in specific example 3, the control unit 11 sets a three-dimensional model U2.

[0023] The control unit 11 acquires the position of the determined virtual projector in the three-dimensional model as a predetermined position (S7). The predetermined position may be set in advance, by the user, or by the processing of S23. In this embodiment, the predetermined position is a position where the optical axis W of the lens of the determined virtual projector is perpendicular to the first plane of the model projection plane. The predetermined position is a position where the optical axis W of the lens of the determined virtual projector passes through the center of the first plane of the model projection plane. The distance from the first plane to the determined virtual projector is set appropriately depending on the shape of the three-dimensional projection plane. As shown in FIG. 3C , the predetermined position in specific examples 1, 2, and 4 is a position where the optical axis W of the lens L of the determined virtual projector PR is perpendicular to the first plane UF1 of the model projection plane UF and passes through the center of the first plane UF1. The model projection plane UF is a plane representing the three-dimensional projection plane F using the three-dimensional model U1. Planes UF1 to UF5 of the model projection plane UF correspond to planes F1 to F5 of the three-dimensional projection plane F, respectively. 3D, the predetermined position in Example 3 is a position where the optical axis W of the lens L of the determined virtual projector PR is perpendicular to the first surface UK1 of the model projection surface UK and passes through the center of the first surface UK1. The model projection surface UK is a surface that represents the three-dimensional projection surface K using the three-dimensional model U2. Surfaces UK1 to UK5 and UC1 and UC2 of the model projection surface UK correspond to surfaces K1 to K5 and C1 and C2 of the three-dimensional projection surface K, respectively.

[0024] The control unit 11 acquires the installation position of the projector 3 having the fisheye lens 4 relative to the three-dimensional projection surface (S8). The installation position of the projector 3 relative to the three-dimensional projection surface is acquired using three-dimensional coordinates in the world coordinate system. The control unit 11 acquires the lens characteristics of the fisheye lens 4 (S9). The lens characteristics of the fisheye lens 4 include a parameter corresponding to the focal length of the fisheye lens 4, a parameter indicating lens distortion, and a parameter related to the optical center. The control unit 11 executes a conversion process to project the image data acquired in S4 into two-dimensional image data to be projected by the projector 3 (S11). The image data in this embodiment is video data, and the conversion process is executed for each frame included in the video data.

[0025] As shown in FIG. 5 , the control unit 11 acquires image data of the current frame from the video data as image data to be processed (S41). The control unit 11 transmits various parameters to the GPU 22 for obtaining two-dimensional image data to be used for projection from the image data (S42). Based on the parameters transmitted in S42, the control unit 11 converts the projection result obtained when the image data is virtually projected onto the three-dimensional projection surface using the determined virtual projector determined in S5 into two-dimensional image data obtained when the image data is virtually captured using a fisheye lens placed at the installation position acquired in S8 (S43 to S45). Specifically, based on the installation position of the projector 3 relative to the three-dimensional projection surface acquired in S8, the control unit 11 virtually captures an image of the model projection surface using the fisheye lens N placed at the installation position, and acquires two-dimensional coordinate data representing the model projection surface in two-dimensional coordinates (S43). As shown in FIG. 3(C), in specific examples 1, 2, and 4, the control unit 11 acquires two-dimensional coordinate data when a model projection surface UF is virtually photographed using a fisheye lens N of a virtual camera CA. The virtual fisheye lens N is disposed at a position corresponding to the installation position of the projector 3 with respect to the three-dimensional projection surface F. In other words, the relative position of the projector 3 with respect to the three-dimensional projection surface F is the same as the relative position of the virtual camera CA with respect to the model projection surface UF. Similarly, as shown in FIG. 3(D), in specific example 3, the control unit 11 acquires two-dimensional coordinate data when a model projection surface UK is virtually photographed using a fisheye lens N of a virtual camera CA. It is preferable that the virtual fisheye lens N has the same or similar lens characteristics as the fisheye lens 4 of the projector 3. The virtual fisheye lens N is disposed at a position corresponding to the installation position of the projector 3 with respect to the three-dimensional projection surface K. In other words, the relative position of the projector 3 with respect to the three-dimensional projection surface K is the same as the relative position of the virtual camera CA with respect to the model projection surface UK.

[0026] The processing of S43 is performed using a vertex shader pre-installed in the GPU 22 (S43). The vertex shader converts three-dimensional polygon information representing the three-dimensional model into a rectangular parallelepiped coordinate system. The control unit 11 typically generates a two-dimensional image from the projection results of the three-dimensional model using a central projection method in the vertex shader. The control unit 11 then replaces the central projection method with an equidistant projection method that corresponds to the lens characteristics of the fisheye lens, thereby obtaining two-dimensional coordinate data suitable for the projector 3 having the fisheye lens 4. In typical perspective projection photography, the coordinates of the two-dimensional image are calculated by a perspective transformation matrix using a single matrix multiplication per vertex of the three-dimensional model. On the other hand, a vertex shader using equidistant projection cannot calculate the coordinates of the two-dimensional image using a simple matrix operation. In this embodiment, the control unit 11 performs the processing of S43 using a shader program. In S43, the control unit 11 may eliminate unnecessary information by performing processes such as backface culling, clipping, attribute evaluation, and rasterization. By performing the process at S43, the control unit 11 can determine the coordinates of each vertex on the three-dimensional projection surface of the two-dimensional coordinate data based on the vertex data for the projection result of the three-dimensional model.

[0027] The control unit 11 performs a rasterization process on the vertex data processed in S43 (S44). In the rasterization process, the control unit 11 combines the vertex data to form polygons that represent the deformed model data of the three-dimensional model, and determines the arrangement of pixels for drawing each polygon on the two-dimensional image data.

[0028] The control unit 11 generates two-dimensional image data based on the projection result when image data is virtually projected onto the model projection surface using the determined virtual projector arranged at a predetermined position relative to the model projection surface and the two-dimensional coordinate data acquired in S43 (S45). The two-dimensional image data is an image signal output to the projector 3. The control unit 11 of this embodiment generates two-dimensional image data in which color information is set in the two-dimensional coordinate data based on color information of the image data and the correspondence between the image data and the two-dimensional coordinate data. The control unit 11 applies pixel shader processing to each pixel to which the processing of S44 was applied. In the pixel shader processing, the correspondence between the image data and the two-dimensional coordinate data is determined using the determined virtual projector determined in S5. If the determined virtual projector is a standard virtual projector, the control unit 11 determines the correspondence between the image data and the two-dimensional coordinate data based on framework information of the three-dimensional model. If the determined virtual projector is a fisheye virtual projector or a 360-degree virtual projector, the control unit 11 performs calculations to map the projection positions on the three-dimensional model when image data is projected radially from the virtual projector placed at the predetermined position and orientation acquired in S7. In this embodiment, the predetermined orientation is an orientation in which the optical axis W is perpendicular to the first plane of the model projection surface. This allows the control unit 11 to identify the correspondence between the image data and the two-dimensional coordinate data of the image data captured using the fisheye lens. The fisheye virtual projector and the 360-degree virtual projector use different methods for calculating the projection range. With the 360-degree virtual projector, the control unit 11 calculates the elevation / depression angles and azimuth angles for each point on the three-dimensional model, with the predetermined position and orientation of the virtual projector as the origin. The control unit 11 can calculate the texture coordinates of the image data captured by the 360-degree camera from the elevation / depression angles and azimuth angles. As a result, the control unit 11 identifies the correspondence between the image data and the two-dimensional coordinate data of the image data captured using the 360-degree camera.

[0029] The control unit 11 generates two-dimensional image data by assigning the color and transparency of the two-dimensional coordinate data to each pixel based on the color information of the image data and the correspondence between the image data and the two-dimensional coordinate data. That is, the control unit 11 assigns the color and transparency of the pixel corresponding to the vertex of the polygon to the color and transparency at the position of the texture coordinate included in the vertex data of that vertex in the texture image. The color and transparency of pixels not corresponding to the vertex of the polygon are determined by interpolating the color and transparency assigned to the vertex of the polygon. In addition to determining the color and transparency of pixels using a pixel shader, the control unit 11 may also apply processes such as an alpha test, a depth test, a stencil test, and blending.

[0030] The control unit 11 outputs a two-dimensional image data image signal to the projector 3, which projects the image data onto the three-dimensional projection surface using the projector 3 (S12). The projector 3 projects the image data onto the three-dimensional projection surface according to the image signal output from the projection control device 2. In specific example 1, the projector 3 projects the two-dimensional image data of the image data G1 shown in FIG. 6A onto the three-dimensional projection surface F, resulting in a projection result that reproduces the projection result P1 shown in FIG. 6B. The projection result P1 is obtained by projecting the image data G1 along the three-dimensional projection surface F. The projection results of the image data projected onto the fourth surface F4 and the fifth surface F5 are U-shaped, with an open end facing in the direction opposite to the direction from the projector 3 toward the first surface F1. The center of the image data coincides with the center of the first surface F1. In the projection result P2 of the comparative example shown in FIG. 6C, the fisheye lens distortion of the determined virtual projector is included in the projection result P2, and the contour of the projection result P2 is curved. In contrast, the projection result P1 of Example 1 does not include distortion caused by the fisheye lens, and the contours of the projection result P1 are linear, resulting in a projection result with less distortion and less discomfort compared to the projection result P1.

[0031] In specific example 2, the projector 3 projects two-dimensional image data of image data G2 onto a three-dimensional projection surface F, resulting in a projection result P3 shown in FIG. 7B. In specific example 3, the projector 3 projects two-dimensional image data of image data G2 shown in FIG. 7A onto a three-dimensional projection surface K, resulting in a projection result P4 shown in FIG. 7C. The three-dimensional projection surface K of the projection result P4 has curved surfaces at the connection between the horizontally adjacent first and second surfaces K1 and K2, and at the connection between the first and third surfaces K1 and K3, and does not include any corners in the horizontal direction. However, both projection results P3 and P4 are projection results with little distortion and a natural appearance, regardless of the shape of the three-dimensional projection surface. In specific example 4, the projector 3 projects two-dimensional image data of image data G3 onto a three-dimensional projection surface F, resulting in a projection result P5 shown in FIG. 8B. As shown in FIG. 8B, in Example 4, the two-dimensional image data of the image data G3 shown in FIG. 8A is projected onto the three-dimensional projection surface F, resulting in a projection result that reproduces the projection result P5 shown in FIG. 8B. As in Examples 1 to 4, the projection system 1 produces projection results that are less unnatural than conventional projection results for each of multiple types of image data captured under different conditions. The control unit 11 determines whether to terminate the projection control process (S31). The control unit 11 determines to terminate the projection control process when the projection of the last frame is completed or when an instruction to terminate the projection control process input by the user via the input unit 14 is detected.

[0032] If an instruction to terminate the projection control process is not detected (S31: NO), the control unit 11 determines whether a position change instruction to change the projection position has been detected (S32). When the projection position is changed, the correspondence between the representative point on the three-dimensional projection surface and the representative point on the image data is changed. The representative point is, for example, the center point. To change the projection position, the user operates the joystick of the input unit 14 to specify the desired projection position. For example, to move the projection result to the right from its current position, the user tilts the joystick to the right. If a position change instruction is detected (S32: YES), the control unit 11 changes the setting of the projection position of the image data on the model projection surface in accordance with the position change instruction detected in S32 (S33). In S45, which is executed in S11 after processing S33, the control unit 11 generates two-dimensional image data when virtually projecting the image data on the model projection surface at the projection position specified by the position setting instruction using the determined virtual projector arranged at a predetermined position. If the determined virtual projector is a standard virtual projector, the control unit 11 changes the texture pasting position of the image data acquired in S41 up, down, left, or right in response to the position change instruction. If the determined virtual projector is a fisheye virtual projector or a 360-degree virtual projector, the control unit 11 changes the orientation (angle) of the virtual projector in response to the position change instruction. As shown in Figure 9 (B), for the projection result P6 of specific example 1, if an instruction to move the projection result to the right is detected in S32 (S32: YES), the projection result P7 shown in Figure 9 (A) is projected in S12.

[0033] If a position change instruction is not detected (S32: NO), or after S33, the control unit 11 determines whether a magnification change instruction to change the projection magnification has been detected (S34). To change the projection magnification, the user operates the cross keys on the input unit 14 to specify the desired projection magnification. For example, if the user wants to increase the projection magnification, the user presses the up key on the cross keys. If a magnification change instruction is detected (S34: YES), the control unit 11 changes the projection magnification setting of the image data (S35). In S41, which is executed in S11 after processing S35, the control unit 11 acquires a projection result obtained when the image data is virtually projected on the model projection surface at the projection magnification specified in the magnification change instruction using the determined virtual projector arranged at a predetermined position. As shown in FIG. 9, if an instruction to increase the projection magnification is detected in S32 for the projection result P6 of Example 1 (S32: YES), the projection result P8 shown in FIG. 9C is projected in S12. If an instruction to change the projection magnification is not detected (S34: NO), or after S35, the control unit 11 returns the process to S11. If an instruction to end the projection control process is detected (S31: YES), the control unit 11 stores the current setting in the storage unit 15, and then ends the process.

[0034] In the above embodiment, the projection system 1, the projection control device 2, the projector 3, and the fisheye lens 4 are examples of the projection system, the projection control device, the projector, and the fisheye lens of the present invention, respectively. The process of S6 is an example of the model setting process of the present invention. The process of S4 is an example of the image acquisition process of the present invention. The process of S5 is an example of the virtual projector determination process of the present invention. The process of S43 is an example of the 2D coordinate data conversion process of the present invention. The process of S45 is an example of the 2D image data generation process of the present invention. The process of S21 is an example of the reference projection process of the present invention. The process of S22 is an example of the reception process of the present invention. The process of S23 is an example of the correction process of the present invention. The process of S32 is an example of the position change instruction acquisition process of the present invention. The process of S34 is an example of the magnification change instruction acquisition process of the present invention.

[0035] The projection system 1 of the above embodiment includes a projector 3 having a fisheye lens 4 and a projection control device 2 that controls the image projected by the projector 3. The projection control device 2 executes an image acquisition process to acquire image data to be processed (S4). The projection control device 2 executes a model setting process to acquire model parameters including the size and shape of a concave three-dimensional projection surface and set a three-dimensional model (S6). The projection control device 2 executes a virtual projector determination process to determine a selected virtual projector from among multiple virtual projectors according to the image data capture conditions (S5). The projection control device 2 executes a two-dimensional coordinate data conversion process to virtually capture a model projection surface, which represents the three-dimensional projection surface as a three-dimensional model, using a fisheye lens placed at the installation position of the projector 3, and acquire two-dimensional coordinate data of the model projection surface (S43). The projection control device 2 executes a two-dimensional image data generation process to generate two-dimensional image data based on the projection result and the two-dimensional coordinate data when the image data is virtually projected onto the model projection surface using the selected virtual projector placed at a predetermined position relative to the model projection surface (S45). The projection control device 2 executes an image projection process in which the two-dimensional image data is projected onto a three-dimensional projection surface using the projector 3 (S12).

[0036] Typically, when an image is projected using a projector with a fisheye lens, distortion specific to the fisheye lens occurs in the projected image. Similarly, when a photograph is taken using a camera with a fisheye lens, distortion specific to the fisheye lens occurs in the photographed image. Here, since photographing and projection are reverse transformations, when image data photographed with a fisheye lens is projected using a fisheye lens with the same lens characteristics as the fisheye lens used for photographing, the distortion during photographing is canceled out by the projection. As a result, a projected image with less distortion can be obtained without the need for correction specific to fisheye lenses. The projection control device 2 of the projection system 1 simulates an ideal projection environment corresponding to the photographing conditions based on the projection results using a virtual projector corresponding to each image data of multiple photographing conditions. The projection control device 2 of the projection system 1 converts the projection results in the ideal projection environment into two-dimensional image data captured using the fisheye lens N from the actual installation position of the projector 3. As a result, the projection control device 2 of the projection system 1 can generate two-dimensional image data capable of reproducing the ideal projection environment by projecting from a realistic projection environment corresponding to the photographing conditions through relatively simple processing. In other words, the projection control device 2 can perform planar projection and three-dimensional projection according to the shooting conditions within the same device. Therefore, the projection system 1 can obtain, based on image data for each of a plurality of shooting conditions, a projected image with less distortion than conventional projections, which is less unnatural to the viewer and has less distortion than conventional projections, regardless of the installation position of the projector 3, through simpler processing than conventional projections, even if the projection surface has an arbitrary three-dimensional shape.

[0037] The virtual projector determination process of S5 is a process for determining, as the determined virtual projector, a fisheye virtual projector that virtually projects image data along a model projection surface using a virtual fisheye lens when the shooting conditions include conditions in which the image was shot using a fisheye lens.The virtual projector determination process is a process for determining, as the determined virtual projector, a standard virtual projector that virtually projects image data along a model projection surface when the shooting conditions include conditions in which the image was shot using a standard lens.The projection system 1 can project each of the image data shot using a fisheye lens and the image data shot using a standard lens from the projector 3 onto a three-dimensional projection surface consisting of a concave surface with less distortion than conventional methods.

[0038] The model setting process of S6 is a process of acquiring model parameters including a three-dimensional projection surface having five surfaces, including a first surface and four surfaces contiguous to the first surface, and setting a three-dimensional model (S7). The projection system 1 sets a three-dimensional model with relatively simple settings, and can obtain a projected image that is less uncomfortable for the viewer, regardless of the shooting conditions of the image data and the installation position of the projector 3, with a process simpler than conventional methods.

[0039] The model setting process of S6 is a process of acquiring model parameters including a three-dimensional projection surface having five surfaces, including a first surface facing the fisheye lens and four surfaces contiguous to the first surface, and setting up a three-dimensional model. The predetermined position is a position where the optical axis W of the lens of the determined virtual projector is perpendicular to the first surface. Because the projection system 1 sets the predetermined position as a position facing the first surface of the three-dimensional projection surface, it is easier to simulate an ideal projection environment than under other conditions when the predetermined position is not facing the first surface, and the projection result acquisition process can be facilitated.

[0040] The model setting process of S6 is a process of acquiring model parameters including a three-dimensional projection surface having a first surface facing the fisheye lens 4 of the projector 3 and setting a three-dimensional model. The predetermined position is a position where the optical axis W of the lens L of the determined virtual projector PR passes through the center M of the first surface of the three-dimensional projection surface. In the projection system 1, the predetermined position is located on an extension of a virtual line extending from the center of the first surface of the three-dimensional projection surface in a direction perpendicular to the first surface. Therefore, compared to other conditions, the projection system 1 makes it easier to simulate an ideal projection environment and facilitates the process of virtually projecting image data onto the three-dimensional projection surface.

[0041] The model setting process of S6 is a process of setting a three-dimensional model by acquiring model parameters including a three-dimensional projection surface facing the fisheye lens 4 of the projector 3 and having a first surface extending vertically, second and third surfaces intersecting the first surface and extending vertically, and fourth and fifth surfaces intersecting the first surface and extending horizontally. The first and second surfaces may be connected by a first curved surface, and the first and third surfaces may be connected by a second curved surface. Generally, on a three-dimensional projection surface, image distortion is more likely to occur at the corners of the first and second surfaces and the corners of the first and third surfaces than on a flat surface. The projection system 1 is less likely to cause distortion at the corners of the first and second surfaces and the corners of the first and third surfaces in the projection result projected onto the three-dimensional projection surface than when the three-dimensional projection surface does not have the first and second curved surfaces.

[0042] The image data has shooting conditions added thereto. The virtual projector determination process is a process for determining a determined virtual projector from among a plurality of virtual projectors according to the shooting conditions added to the image data. The projection system 1 can determine a determined virtual projector based on the shooting conditions added to the image data, thereby eliminating the need for a user of the projection system 1 to input the shooting conditions.

[0043] The two-dimensional image data generation process is a process of generating two-dimensional image data in which color information is set in two-dimensional coordinate data based on the color information of the image data and the correspondence between the image data and two-dimensional coordinate data (S45).The projection system 1 can generate two-dimensional image data in which color information is set in two-dimensional coordinate data through a relatively simple process based on the color information of the image data and the correspondence between the image data and two-dimensional coordinate data.

[0044] The projection control device 2 executes a reference projection process to project a reference image indicating the setting status of the three-dimensional model onto the three-dimensional projection surface via the fisheye lens 4 (S21). After starting projection of the reference image, the projection control device 2 executes a reception process to receive a setting change instruction to change the setting of the three-dimensional model (S22). In response to the setting change instruction, the projection control device 2 executes a correction process to change the setting of the three-dimensional model and correct the reference image to be projected onto the three-dimensional projection surface (S23). The projection system 1 can change the setting of the three-dimensional model through a relatively simple process while projecting the reference image onto the actual three-dimensional projection surface. A user of the projection system 1 can check the setting status of the three-dimensional model by the reference image projected onto the three-dimensional projection surface.

[0045] The projection control device 2 executes a position change instruction acquisition process to acquire a position change instruction to change the projection position of the image data on the model projection surface (S32). The two-dimensional image data generation process when the position change instruction is acquired is a process to generate two-dimensional image data when the image data is virtually projected onto the model projection surface at the projection position instructed by the position change instruction. The projection system 1 can project the image data at the projection position whose setting has been changed onto the three-dimensional projection surface with a relatively simple process.

[0046] The projection control device 2 executes a magnification change instruction acquisition process to acquire a magnification change instruction to change the projection magnification of the image data (S34). The two-dimensional image data generation process when the magnification change instruction is acquired is a process to generate two-dimensional image data when the image data is virtually projected on the model projection surface at the projection magnification instructed in the magnification change instruction. The projection system 1 can project image data with the projection magnification setting changed on the three-dimensional projection surface with a relatively simple process.

[0047] The projection system, projection control device, and projection control program of the present invention are not limited to the above-described embodiments, and various modifications may be made without departing from the spirit and scope of the present invention. For example, the following modifications may be made as appropriate.

[0048] The configurations of the projection system, projection control device, and projection control program may be changed as appropriate. The projector 3 and the projection control device 2 may be integrated into one device. The projection control device may be a dedicated device or a general-purpose PC, etc.

[0049] The projection control program, which includes instructions for executing the projection control process, may be stored in the storage unit 15 or memory 23 before the control unit 11 of the projection control device 2 executes the projection control program. Therefore, the method and route for acquiring the projection control program, and the device that stores the program may each be changed as appropriate. The projection control program may be received from another device via a cable or wireless communication and stored in the storage unit 15 or memory 23. The other device may include, for example, a PC or a server connected via a network.

[0050] Each step of the projection control process is not limited to being executed by the control unit 11, and may be executed in part or in whole by another electronic device (for example, an ASIC). One example of the other electronic device is an ASIC. Each step of the projection control process may be distributed and processed by multiple CPUs. The order of each step of the projection control process can be changed, steps can be omitted, or steps can be added as necessary. The following modifications may be made as appropriate to the processing executed by the projection system 1, including the projection control process.

[0051] The shape of the three-dimensional projection surface may be concave, and may be modified as appropriate, such as a hemispherical surface. The process of modifying the three-dimensional model may be modified as appropriate. If the projection control device 2 is equipped with a sensor that measures the distance and shape to the three-dimensional projection surface, the projection control device 2 may automatically set the three-dimensional model based on the sensor's detection results. The pattern of the reference image may be modified as appropriate, for example, a pattern that represents only the shape of the first surface. The projection control device 2 may not be able to perform the process of modifying the three-dimensional model using the reference image. The modifiable items of the three-dimensional model may be modified as appropriate. For example, the function of correcting the surface to a trapezoidal shape may not be included, and the function of correcting the connection between two adjacent surfaces among the multiple surfaces that make up the three-dimensional projection surface to a curved surface may not be included. The connection between two adjacent surfaces in the vertical direction among the multiple surfaces that make up the three-dimensional projection surface may also be curved. If the size and shape of the three-dimensional projection surface are known, such as the screen E, the three-dimensional model may be set based on model parameters including the size and shape of the three-dimensional projection surface input by the user.

[0052] The preset multiple types of virtual projectors may be changed as appropriate. Any two of the standard virtual projector, fisheye virtual projector, and 360-degree virtual projector in the above embodiment may be set as multiple types of virtual projectors. The multiple types of virtual projectors may include projectors other than the standard virtual projector, fisheye virtual projector, and 360-degree virtual projector. The image data may not have shooting conditions attached, and the control unit 11 may set a virtual projector selected by the user as the determined virtual projector. The control unit 11 may determine the determined virtual projector according to the aspect ratio of the image. The predetermined position of the determined virtual projector does not have to be a position where the optical axis of the lens of the determined virtual projector is perpendicular to the first surface of the model projection surface, or a position where the optical axis of the lens of the determined virtual projector passes through the center of the first surface of the model projection surface. Each of S32 to S35 may be omitted as appropriate. In the two-dimensional image data generation process of S45, two-dimensional image data may be generated based on the projection result and the two-dimensional coordinate data acquired in S43, and the method for generating the two-dimensional image data may be changed as appropriate.

[0053] The above-mentioned modifications may be combined as appropriate within the scope of the claims. The applicant intends to obtain patent rights for not only the combinations exemplified in the claims, but also for other combinations that do not deviate from the gist of the present invention and do not cause any contradictions.

[0054] 1: Projection system, 2: Projection control device, 3: Projector, 4: Fisheye lens, 11: Control unit, 12: Communication unit, 14: Input unit, 15: Storage unit, 21: CPU, 22: GPU, 23: Memory

Claims

1. A projection system comprising a projector having a fisheye lens and a projection control device that controls an image projected by the projector, wherein the projection control device executes the following operations: an image acquisition process that acquires image data to be processed; a model setting process that acquires model parameters including the size and shape of a three-dimensional projection surface, which is a concave surface, and sets a three-dimensional model; a virtual projector determination process that determines a determined virtual projector from among a plurality of virtual projectors according to the shooting conditions of the image data; a two-dimensional coordinate data conversion process that uses a fisheye lens arranged at the installation position of the projector to virtually photograph a model projection surface, which represents the three-dimensional projection surface in the three-dimensional model, and acquires two-dimensional coordinate data that represents the model projection surface in two-dimensional coordinates; a two-dimensional image data generation process that generates two-dimensional image data based on the projection result and the two-dimensional coordinate data when the image data is virtually projected onto the model projection surface using the determined virtual projector arranged at a predetermined position with respect to the model projection surface; and an image projection process that projects the two-dimensional image data onto the three-dimensional projection surface using the projector.

2. The projection system according to claim 1, characterized in that the virtual projector determination process is a process for determining, when the shooting conditions include a condition in which the image was shot using a fisheye lens, a fisheye virtual projector that virtually projects the image data along the model projection surface using a virtual fisheye lens as the determined virtual projector, and for determining, when the shooting conditions include a condition in which the image was shot using a standard lens, a standard virtual projector that virtually projects the image data along the model projection surface as the determined virtual projector.

3. The projection system according to claim 1 or 2, characterized in that the model setting process is a process of obtaining the model parameters including the three-dimensional projection surface having five surfaces including a first surface and four surfaces contiguous to the first surface, and setting the three-dimensional model.

4. The model setting process is a process of acquiring the model parameters including the three-dimensional projection surface having five surfaces including a first surface facing the fisheye lens and four surfaces contiguous to the first surface, and setting the three-dimensional model, and the specified position is a position where the optical axis of the lens of the determined virtual projector is perpendicular to one surface of the model projection surface, as described in claim 2.

5. The projection system described in claim 2, characterized in that the model setting process is a process of acquiring the model parameters including the three-dimensional projection surface having a first surface facing the fisheye lens of the projector and setting the three-dimensional model, and the specified position is a position where the optical axis of the lens of the determined virtual projector passes through the center of the first surface of the model projection surface.

6. The model setting process is a process of acquiring the model parameters including the three-dimensional projection surface having a first surface facing the fisheye lens of the projector and extending in the vertical direction, second and third surfaces intersecting with the first surface and extending in the vertical direction, and fourth and fifth surfaces intersecting with the first surface and extending in the horizontal direction, and setting the three-dimensional model, wherein the first surface and the second surface are connected by a first curved surface, and the first surface and the third surface are connected by a second curved surface.

7. The projection system according to claim 1 or 2, characterized in that the image data has the shooting conditions added thereto, and the virtual projector determination process is a process of determining the determined virtual projector from among the plurality of virtual projectors in accordance with the shooting conditions added to the image data.

8. The projection system according to claim 1, characterized in that the two-dimensional image data generation process is a process of generating the two-dimensional image data in which color information is set in the two-dimensional coordinate data based on color information of the image data and the correspondence between the image data and the two-dimensional coordinate data.

9. The projection system according to claim 1, further comprising: a reference projection process for projecting a reference image indicating a setting status of the three-dimensional model onto the three-dimensional projection surface via the fisheye lens; a reception process for receiving a setting change instruction for changing the setting of the three-dimensional model after starting projection of the reference image; and a correction process for changing the setting of the three-dimensional model in response to the setting change instruction, and correcting the reference image to be projected onto the three-dimensional projection surface.

10. The projection system according to claim 1, characterized in that the projection control device further executes a position change instruction acquisition process to acquire a position change instruction to change the projection position of the image data on the model projection surface, and the two-dimensional image data generation process when the position change instruction is acquired is a process to generate the two-dimensional image data when the image data is virtually projected onto the projection position instructed by the position change instruction on the model projection surface.

11. The projection system according to claim 1, characterized in that the projection control device further executes a magnification change instruction acquisition process to acquire a magnification change instruction to change the projection magnification of the image data, and the two-dimensional image data generation process when the magnification change instruction is acquired is a process to generate the two-dimensional image data when the image data is virtually projected on the model projection plane at the projection magnification instructed by the magnification change instruction.

12. A projection control device having a control unit and controlling an image projected by a projector having a fisheye lens, wherein the control unit executes the following operations: an image acquisition process for acquiring image data to be processed; a model setting process for acquiring model parameters including the size and shape of a three-dimensional projection surface, which is a concave surface, and setting a three-dimensional model; a virtual projector determination process for determining a determined virtual projector from among a plurality of virtual projectors according to the shooting conditions of the image data; a two-dimensional coordinate data conversion process for virtually shooting a model projection surface, which represents the three-dimensional projection surface in the three-dimensional model, using a fisheye lens arranged at the installation position of the projector, and acquiring two-dimensional coordinate data representing the model projection surface in two-dimensional coordinates; a two-dimensional image data generation process for generating two-dimensional image data based on the projection result and the two-dimensional coordinate data when the image data is virtually projected onto the model projection surface using the determined virtual projector arranged at a predetermined position with respect to the model projection surface; and an image projection process for projecting the two-dimensional image data from the projector.

13. A projection control program to be executed by a projection control device that controls an image projected by a projector having a fisheye lens, the projection control program including instructions to cause the projection control device to execute the following: an image acquisition process to acquire image data to be processed; a model setting process to acquire model parameters including the size and shape of a three-dimensional projection surface, which is a concave surface, and set a three-dimensional model; a virtual projector determination process to determine a determined virtual projector from among a plurality of virtual projectors according to the shooting conditions of the image data; a two-dimensional coordinate data conversion process to virtually photograph a model projection surface, which represents the three-dimensional projection surface in the three-dimensional model, using a fisheye lens arranged at the installation position of the projector, and acquire two-dimensional coordinate data that represents the model projection surface in two-dimensional coordinates; a two-dimensional image data generation process to generate two-dimensional image data based on the projection result and the two-dimensional coordinate data when the image data is virtually projected onto the model projection surface using the determined virtual projector arranged at a predetermined position with respect to the model projection surface; and an image projection process to project the two-dimensional image data from the projector.

Images