Projection system, projection control device, and projection control program

Abstract

The invention provides a projection system, a projection control device, and a projection control program, which project an image with reduced distortion from a projector to a three-dimensional projection plane formed by a concave surface on the basis of image data of each of a plurality of imaging conditions. The projection control device acquires image data (S4), acquires model parameters, sets a three-dimensional model (S6), and determines a decision virtual projector from a plurality of preset virtual projectors according to an imaging condition of the image data (S5). A projection control device generates two-dimensional image data (S11) on the basis of a projection result when image data is projected onto a model projection surface in which a three-dimensional projection surface is represented by a three-dimensional model using a decision virtual projector disposed at a predetermined position, and two-dimensional coordinate data in which the model projection surface is represented by two-dimensional coordinates. Two-dimensional image data is projected onto a three-dimensional projection surface using a projector (S12).

Description

Technical Field

[0001] This invention relates to a projection system, a projection control device, and a projection control program. Background Technology

[0002] The projection method described in Patent Document 1 assumes a projection image where the center of a dome-shaped projection surface with its concave surface facing the observer is the center of the projected image. The projected image is distorted in a way that makes the assumed projected image consistent with the projected image based on a projector placed at an arbitrary position.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 9-149351 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] Traditional projection methods require adjusting the waveforms of the projector's horizontal and vertical scanning voltages to distort the projected image, a complex process. Furthermore, because these methods process image data using a single method, distortion occurs when image data is input under conditions different from those anticipated during the shooting.

[0008] The purpose of this invention is to provide a projection system, a projection control device, and a projection control program that, based on image data from multiple shooting conditions, projects images from a projector onto a three-dimensional projection surface composed of concave surfaces, resulting in less image distortion compared to previous methods.

[0009] Technical solutions for solving the problem

[0010] The projection system of the first aspect of the present invention comprises: a projector having a fisheye lens; and a projection control device for controlling the image projected by the projector, the projection control device performing the following processes: image acquisition processing for acquiring image data as the object of processing; model setting processing for acquiring model parameters including the size and shape of a three-dimensional projection surface that is concave, and setting a three-dimensional model; virtual projector determination processing for determining a virtual projector from a plurality of virtual projectors based on the shooting conditions of the image data; two-dimensional coordinate data conversion processing for virtually shooting a model projection surface representing the three-dimensional projection surface using the fisheye lens disposed at a set position on the projector, and acquiring two-dimensional coordinate data representing the model projection surface using two-dimensional coordinates; two-dimensional image data generation processing for generating two-dimensional image data based on the projection result when the image data is virtually projected onto the model projection surface using the determined virtual projector disposed at a predetermined position relative to the model projection surface and the two-dimensional coordinate data; and image projection processing for projecting the two-dimensional image data onto the three-dimensional projection surface using the projector.

[0011] Typically, when projecting an image using a projector with a fisheye lens, the projected image exhibits distortion characteristic of fisheye lenses. Similarly, when taking a photograph using a camera with a fisheye lens, the captured image exhibits distortion characteristic of fisheye lenses. Here, since shooting and projection are inverse transformations, if image data captured by a fisheye lens is projected through a fisheye lens with the same lens characteristics as the one used for shooting, the distortion from the shooting is canceled out by the projection, resulting in a projected image with minimal distortion even without corrections specific to fisheye lenses. The projection system of the first approach uses a virtual projector corresponding to image data for multiple shooting conditions to simulate an ideal projection environment corresponding to the shooting conditions. The projection system converts the projection result under the ideal projection environment into two-dimensional image data from a fisheye lens taken at the actual projector's location. Therefore, the projection control device of the projection system can generate two-dimensional image data that can reproduce the ideal projection environment through projection from the real projection environment with relatively simple processing. That is, the projection control device of the projection system can perform planar projection and stereoscopic projection corresponding to the shooting conditions within the same device. Therefore, based on image data from multiple shooting conditions, the projection system can produce projected images that are less discordant to viewers and have reduced distortion compared to the past, even when the projection surface has an arbitrary three-dimensional shape, through simpler processing than before. This is independent of the projector's placement.

[0012] The projection control device of the second aspect of the present invention includes a control unit that controls the image projected by a projector with a fisheye lens. The control unit performs the following processes: image acquisition processing, acquiring image data as the object of processing; model setting processing, acquiring model parameters including the size and shape of a three-dimensional projection surface that is concave, and setting a three-dimensional model; virtual projector selection processing, determining a virtual projector from among multiple virtual projectors based on the shooting conditions of the image data; two-dimensional coordinate data conversion processing, virtually photographing a model projection surface representing the three-dimensional projection surface using the fisheye lens positioned at a setting position on the projector, and acquiring two-dimensional coordinate data representing the model projection surface using two-dimensional coordinates; two-dimensional image data generation processing, generating two-dimensional image data based on the projection result when the image data is virtually projected onto the model projection surface using the determined virtual projector positioned relative to the model projection surface and the two-dimensional coordinate data; and image projection processing, projecting the two-dimensional image data from the projector. The projection control device of the second aspect achieves the same effect as the projection system of the first aspect.

[0013] The third-party projection control program of the present invention is executed by a projection control device that controls the image projected by a projector with a fisheye lens. The projection control program includes instructions for the projection control device to perform the following processes: image acquisition processing, acquiring image data as the object of processing; model setting processing, acquiring model parameters including the size and shape of a three-dimensional projection surface that is concave, and setting a three-dimensional model; virtual projector determination processing, determining a virtual projector from among a plurality of virtual projectors based on the shooting conditions of the image data; two-dimensional coordinate data conversion processing, virtually shooting a model projection surface representing the three-dimensional projection surface using the fisheye lens configured at a setting position on the projector, and acquiring two-dimensional coordinate data representing the model projection surface using two-dimensional coordinates; two-dimensional image data generation processing, generating two-dimensional image data based on the projection result when the image data is virtually projected onto the model projection surface using the determined virtual projector configured relative to the model projection surface at a predetermined position and the two-dimensional coordinate data; and image projection processing, projecting the two-dimensional image data from the projector. The projection control program of the third method achieves the same effect as the projection system of the first method by being executed by the projection control device. Attached Figure Description

[0014] Figure 1 (A) is a conceptual diagram of a projection system 1 with a projector 3 suspended from the ceiling of room R. Figure 1 (B) is an explanatory diagram of room R equipped with screen E.

[0015] Figure 2 This is a flowchart of the projection control process executed by the projection control device 2.

[0016] Figure 3 (A) is a stereoscopic view of the three-dimensional model U1 set in the projection control process. Figure 3 (B) is a stereoscopic view of the three-dimensional model U2 set in the projection control process. Figure 3 (C) is an explanatory diagram of a virtual projector PR positioned at a specified location relative to the model projection plane UF of the 3D model U1, and a virtual camera CA positioned at a set location. Figure 3 (D) is an explanatory diagram of a virtual projector PR positioned at a specified location relative to the model projection plane UK of the 3D model U2, and a virtual camera CA positioned at a set location.

[0017] Figure 4 (A) is an explanatory diagram of the three-dimensional projection plane F. Figure 4 (B) is an explanatory diagram of the reference image Q projected onto the three-dimensional projection plane F during projection control processing. Figure 4 (C) is an explanatory diagram of the reference image Q projected onto the three-dimensional projection plane F when the settings of the three-dimensional model are changed in the projection control process.

[0018] Figure 5 This is a flowchart of the conversion process performed in the projection control process.

[0019] Figure 6 (A) is an explanatory diagram of image data G1 taken using a standard lens. Figure 6 (B) is an explanatory diagram of the projection result P1 when the two-dimensional image data generated based on the image data G1 of Specific Example 1 is projected onto the three-dimensional projection plane F. Figure 6 (C) is an explanatory diagram of the projection result P2 when the two-dimensional image data generated based on the image data G1 of the comparative example is projected onto the three-dimensional projection plane F.

[0020] Figure 7 (A) is an explanatory diagram of image data G2 taken using a fisheye lens. Figure 7 (B) is an explanatory diagram of the projection result P3 when the two-dimensional image data generated based on the image data G2 of specific example 2 is projected onto the three-dimensional projection plane F. Figure 7 (C) is an explanatory diagram of the projection result P4 when the two-dimensional image data generated based on the image data G2 of Specific Example 3 is projected onto the three-dimensional projection plane K.

[0021] Figure 8 (A) is an explanatory diagram of image data G3 taken using a 360-degree camera. Figure 8(B) is an explanatory diagram of the projection result P5 when the two-dimensional image data generated based on the image data G of specific example 4 is projected onto the three-dimensional projection plane F.

[0022] Figure 9 (A) has changed Figure 9 The diagram illustrating the projection result P7 under the projection position of (B) is shown. Figure 9 (B) is an explanatory diagram of the projection result P6. Figure 9 (C) is an explanatory diagram of the projection result P8 when the projection magnification of the projection result P6 is changed. Detailed Implementation

[0023] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the term "invention" will be used... Figure 1 (A) shows the X, Y, and Z directions. The Z direction is vertical. The X and Y directions are horizontal directions perpendicular to the Z direction. The X, Y, and Z directions are mutually perpendicular.

[0024] like Figure 1 As shown in (A), the projection system 1 includes a projector 3 with a fisheye lens 4 and a projection control device 2 for controlling the image projected by the projector 3. The projector 3 and the projection control device 2 are interconnected via wired or wireless means. The projection system 1 is used, for example, in a room R of an entertainment facility such as a karaoke bar. The projector 3 is, for example, suspended from the ceiling of the room R.

[0025] Projector 3 projects image data onto a concave three-dimensional projection surface based on the image signal output from projection control device 2. The shape of the three-dimensional projection surface can be arbitrary if it is concave; in this embodiment, the three-dimensional projection surface is... Figure 1 The three-dimensional projection plane F shown in (A) or Figure 1 The three-dimensional projection plane K is shown in (B). As shown in [the diagram]... Figure 1 As shown in (A), the three-dimensional projection surface F is composed of five of the six faces of the cubic room R. These five faces include a first face F1 opposite the fisheye lens 4 of the projector 3 and four faces F2 to F5 continuous with the first face F1. The first face F1 extends vertically. The second face F2 and the third face F3 are orthogonal to the first face F1 and extend vertically. The fourth face F4 and the fifth face F5 are orthogonal to the first face F1 and extend horizontally. The three-dimensional projection surface K is formed on the concave surface of the screen E that can be set in the cubic room R. Figure 1As shown in (B), the three-dimensional projection surface K consists of five surfaces: a first surface K1 opposite the fisheye lens 4 of the projector 3, and four surfaces K2 to K5 continuous with the first surface K1. The first surface K1 extends vertically. The second surface K2 and the third surface K3 intersect the first surface K1 and extend vertically. The fourth surface K4 and the fifth surface K5 are orthogonal to the first surface K1 and extend horizontally. The first surface K1 and the second surface K2 are connected by a first curved surface C1, and the first surface K1 and the third surface K3 are connected by a second curved surface C2. To reduce the sense of disharmony when viewing the projection result from a location far from the screen E, the distance between the second surface K2 and the third surface K3 increases as they move further away from the first surface K1. The screen E is configured, for example, with the surface opposite to the first surface K1 facing and close to the first surface F1 of the room R. In this embodiment, the first surface F1 is set as the wall surface of room R, but the first surface F1 can also be either the ceiling surface or the floor surface of room R.

[0026] The projection control device 2 converts image data into two-dimensional image data for projection, and performs projection control of the projector 3 based on the two-dimensional image data. The projection control device 2 can be configured at any location in room R, or in a location different from room R. The projection control device 2 includes a communication unit 12, a storage unit 15, an input unit 14, and a control unit 11. The communication unit 12, storage unit 15, and input unit 14 are connected to the control unit 11. The storage unit 15 is, for example, a hard disk drive. The storage unit 15 stores the operating system and various programs. These programs include a projection control program that causes the CPU 21 of the control unit 11 to execute projection control processing. The storage unit 15 stores image data of the object to be projected, internal and external parameters of the projector 3, internal and external parameters of the virtual projector PR, internal and external parameters of the virtual camera CA, and various parameters required for executing projection control processing. Shooting conditions are attached to the image data. In this embodiment, the shooting conditions include the lens used for shooting, the characteristics of the lens used for shooting, and the type of camera. The input unit 14 receives operation instructions from the user of the projection system 1 and outputs signals corresponding to the operation instructions to the control unit 11. The input unit 14 is, for example, a game controller with a joystick and a D-pad. The control unit 11 consists of a CPU 21, a GPU 22, and a memory 23, which function as a computer. The GPU 22 is a processor separate from the CPU 21, performing image processing based on instructions from the CPU 21. The memory 23 consists of, for example, ROM and RAM.

[0027] Reference Figures 2 to 9(C) Using specific examples 1 to 4 of the projection control device 2, the projection control process will be explained. In the projection control process, the process of projecting user-specified image data onto a three-dimensional projection surface F is performed. Specific examples 1 to 4 respectively use frames of animation data of four people talking on a stage as image data. Specific example 1 projects animation data captured by a camera using a standard lens onto the three-dimensional projection surface F. Specific example 2 projects animation data captured by a camera using a fisheye lens onto the three-dimensional projection surface F. Specific example 3 projects animation data captured by a camera using a fisheye lens onto the three-dimensional projection surface K. The three-dimensional projection surface K is a trapezoidal three-dimensional projection surface with a planar shape and rounded corners. Specific example 4 projects animation data captured by an omnidirectional camera onto the three-dimensional projection surface F. In specific examples 1 to 4, the control unit 11 uses a virtual projector corresponding to the shooting conditions of the animation data to generate two-dimensional image data for projection by the projector 3. The virtual projector uses parameters virtually set to calculate the pixel correspondence between the image data and the three-dimensional model. In this embodiment, the projection control device 2 pre-sets three types of virtual projectors and stores the shader programs and parameters required for virtual projection using each virtual projector in the storage unit 15. The three types of virtual projectors are a standard virtual projector, a fisheye virtual projector, and a 360-degree virtual projector. The standard virtual projector virtually projects image data along a model projection plane that represents a three-dimensional projection plane through a three-dimensional model. The fisheye virtual projector uses a virtual fisheye lens to virtually project image data along the model projection plane. The 360-degree projector virtually projects 360 degrees onto the model projection plane in a manner where the pitch and azimuth angles are consistent when viewed from the virtual projector. The projection result of the 360-degree virtual projector is virtually projected 360 degrees, but in reality, it is output to the projectable range of the projector 3, which has a fisheye lens 4 represented by the lens characteristics obtained in S9 described later. The types and number of virtual projectors can be appropriately changed. As a comparative example, in Specific Example 1, the case of generating two-dimensional image data for projection by the projector 3 using a fisheye virtual projector will be described.

[0028] The projection control process is initiated upon user input of a start instruction. Upon detecting the start instruction, the control unit 11 reads the projection control program stored in the storage unit 15 for executing the projection control process into the memory 23. The control unit 11 executes the following steps according to the instructions contained in the program read into the memory 23: Various parameters required for executing the projection control process are stored in the storage unit 15. Various data obtained during the projection control process are appropriately stored in the memory 23. In the projection control process, specific examples 1 to 4 are executed at different time intervals.

[0029] like Figure 2As shown, the control unit 11 acquires the initial settings stored in the storage unit 15 (S1). The initial settings include the settings of the current 3D model. The settings of the current 3D model are represented using model parameters that include the size and shape of the 3D projection plane set by the 3D model. The control unit 11 determines whether to change the 3D model (S2). In this embodiment, the initial values ​​of the 3D model are set as follows: Figure 3 The three-dimensional model U1 is a cube. The projection control device 2 of this embodiment can change the shape and size of each face of the current three-dimensional model, the shape of the curved surface when the connection between two adjacent faces of any face is set as a curved surface, and the shape of the trapezoid when the outline of any face is set as a trapezoid. Furthermore, the control unit 11 can rotate or move the current three-dimensional model horizontally. Normally, the user changes the three-dimensional model when the configuration of the projector 3 relative to the three-dimensional projection surface is changed. When the user wants to change the settings of the current three-dimensional model, they operate the input unit 14 to input an instruction to perform the change settings.

[0030] like Figure 3 As shown in (A), a three-dimensional model U1 is set in the current three-dimensional model regarding the three-dimensional projection plane F of specific examples 1, 2, and 4. Figure 3 As shown in (B), a three-dimensional model U2 is set in the current three-dimensional model of the three-dimensional projection surface K in specific example 3. For the three-dimensional model U2, the setting of the three-dimensional model U1 is changed to match the shape of the screen E. The skeleton information B1 is represented by a grid pattern of the three-dimensional model U1. The skeleton information B2 is represented by a grid pattern of the three-dimensional model U2. The skeleton information B1 and skeleton information B2 are respectively used to map the image data as a texture in the process of determining the standard virtual projector when determining the virtual projector in the process of S5 described later. The standard virtual projector of this embodiment is used in the process of generating two-dimensional image data by the control unit 11 to project the image data in a manner that arranges the image data along the three-dimensional projection surface. Taking the three-dimensional projection surface F as an example, the skeleton information represented by the three-dimensional model will be explained. The control unit 11 sets the projection magnification of the size of the image data extending from the first surface F1 as the center of the image data. For image data extending from the first surface F1 to the second surface F2 and the third surface F3, which are connected to the first surface F1 in the horizontal direction, the control unit 11 projects the image data in a manner that bends along the projection plane. For image data extending from the first surface F1 to the fourth surface F4 and the fifth surface F5, which are connected to the first surface F1 in the vertical direction, as shown in the grid pattern, the control unit 11 deforms and projects the image data in a manner that shrinks towards the center of the surface.

[0031] After detecting the Figure 4If, as shown in (A), an instruction is given to change the settings of the 3D model U1 before the setting change (S2: Yes), then... Figure 4 As shown in (B), the control unit 11 uses the projector 3 to project a reference image Q representing the setting status of the 3D model U1 onto the 3D projection plane F via the fisheye lens 4 (S21). The reference image Q can be any image representing the current setting status of the 3D model; for example, it can be an image representing the boundary between two adjacent faces, or an image containing a grid-like pattern representing the skeleton information B1 of the 3D model U1. After the projection of the reference image Q begins, the control unit 11 receives a setting change instruction to change the setting of the 3D model U1 (S22). The user operates the input unit 14, selects the item to be changed, and inputs the instruction by specifying the change amount. If a setting change instruction for the 3D model U1 is detected (S22: Yes), the control unit 11 changes the setting of the 3D model U1 according to the instruction received in S22, changes the reference image Q projected onto the 3D projection plane F, and projects it (S23). The user observes the reference image projected onto the 3D projection plane F while... Figure 4 As shown in (C), the operation input unit 14 adjusts the parameters of the three-dimensional model so that the reference image Q is positioned along the three-dimensional projection plane F. Figure 4 In step (C), the boundary of the pattern of the reference image Q coincides with the boundary of any two adjacent surfaces among the five surfaces constituting the three-dimensional projection surface. Through processes S22 and S23, the following can be changed: the positional relationship between the room R of the projector 3 and the projector 3; the fisheye lens characteristics of the fisheye lens 4 of the projector 3 obtained in S9; the specified position of the virtual projector obtained in S6; and the orientation of the virtual projector determined in S5. If no setting change instruction for the three-dimensional model is detected (S22: No), or after S23, the control unit 11 determines whether an instruction to end the setting change process for the three-dimensional model has been detected (S24). If no instruction to end the setting change process for the three-dimensional model is detected (S24: No), the control unit 11 returns the process to S21. If an instruction to end the setting change process for the three-dimensional model is detected (S24: Yes), the control unit 11 returns the process to S2.

[0032] If no instruction to change the settings of the 3D model is detected (S2: No), the control unit 11 determines whether a projection instruction to project image data onto a 3D projection surface is detected (S3). If no projection instruction is detected (S3: No), the control unit 11 returns the processing to S2. If a projection instruction is detected (S3: Yes), the control unit 11 acquires the image data that is the object of processing (S4). The image data can be acquired from other devices via the communication unit 12 or from the storage unit 15. For example, in Specific Example 1, the control unit 11 acquires the image data containing... Figure 6 The animation data of image data G1 shown in (A), for specific examples 2 and 3, is obtained by obtaining the animation data containing... Figure 7 The animation data of image data G2 shown in (A), for specific example 4, is obtained by obtaining the animation data containing... Figure 8 The image data G3 shown in (A) is the animation data.

[0033] The controller 11 determines a virtual projector from a set of pre-set virtual projectors based on the image data shooting conditions (S5). In S5, the controller 11 determines a virtual projector from a set of pre-set virtual projectors based on the shooting conditions attached to the image data. If the shooting conditions include shooting with a fisheye lens, as in Examples 2 and 3, the controller 11 determines the fisheye virtual projector as the determined virtual projector. If the shooting conditions include shooting with a standard lens, as in Example 1, the controller 11 determines the standard virtual projector as the determined virtual projector. If the shooting conditions include shooting with a 360-degree camera (omnidirectional camera), as in Example 4, the controller 11 determines the 360-degree virtual projector as the determined virtual projector. In S42, described later, the controller 11 sends the determined virtual projector settings from S5 to the GPU.

[0034] The control unit 11 obtains model parameters including the size and shape of the three-dimensional projection surface F, which is concave, and sets a three-dimensional model including the three-dimensional projection surface (S6). The control unit 11 sets a three-dimensional model U1 in specific examples 1, 2, and 4, and sets a three-dimensional model U2 in specific example 3.

[0035] Controller 11 determines the configuration of the virtual projector in the 3D model as a predetermined position (S7). This predetermined position can be preset, set by the user, or set through the processing in S23. In this embodiment, the predetermined position is the position where the optical axis W of the virtual projector's lens is orthogonal to the first surface of the model's projection surface. The predetermined position is also the position where the optical axis W of the virtual projector's lens passes through the center of the first surface of the model's projection surface. The distance between the first surface and the determined virtual projector is appropriately set according to the shape of the 3D projection surface. Figure 3 As shown in (C), the specified positions in specific examples 1, 2, and 4 determine the position where the optical axis W of the lens L of the virtual projector PR is orthogonal to the first surface UF1 of the model projection surface UF and passes through the center of the first surface UF1. The model projection surface UF is a surface of the three-dimensional projection surface F represented by the three-dimensional model U1. Surfaces UF1 to UF5 of the model projection surface UF correspond to surfaces F1 to F5 of the three-dimensional projection surface F, respectively. Figure 3As shown in (D), the specified position in Specific Example 3 is the position where the optical axis W of the lens L of the virtual projector PR is orthogonal to the first surface UK1 of the model projection surface UK and passes through the center of the first surface UK1. The model projection surface UK is represented by the three-dimensional model U2 as a surface of the three-dimensional projection surface K. The surfaces UK1 to UK5 of the model projection surface UK, as well as UC1 and UC2, correspond to the surfaces K1 to K5 and C1 and C2 of the three-dimensional projection surface K, respectively.

[0036] The control unit 11 obtains the setting position of the projector 3 with the fisheye lens 4 relative to the three-dimensional projection surface (S8). The setting position of the projector 3 relative to the three-dimensional projection surface is obtained using three-dimensional coordinates in the world coordinate system. The control unit 11 obtains the lens characteristics of the fisheye lens 4 (S9). The lens characteristics of the fisheye lens 4 include parameters corresponding to the focal length of the fisheye lens 4, parameters representing lens distortion, and parameters related to the optical center. The control unit 11 performs a conversion process that projects the image data obtained in S4 into two-dimensional image data for projection by the projector 3 (S11). In this embodiment, the image data is animation data, and the conversion process is performed according to each frame contained in the animation data.

[0037] like Figure 5 As shown, the control unit 11 acquires the image data of the current frame in the animation data as the image data of the object to be processed (S41). The control unit 11 sends various parameters to the GPU 22 for obtaining two-dimensional image data for projection from the image data (S42). Based on the parameters sent in S42, the control unit 11 converts the projection result, in the case where the image data is virtually projected onto the three-dimensional projection surface using the virtual projector determined in S5, into two-dimensional image data, in the case where the image data is virtually captured using a fisheye lens configured at the setting position obtained in S8 (S43 to S45). Specifically, based on the setting position of the projector 3 relative to the three-dimensional projection surface obtained in S8, the control unit 11 uses the fisheye lens N configured at the setting position to virtually capture the model projection surface and acquire two-dimensional coordinate data representing the model projection surface in two-dimensional coordinates (S43). Figure 3 As shown in (C), in specific examples 1, 2, and 4, the control unit 11 acquires two-dimensional coordinate data when a fisheye lens N using a virtual camera CA is used to virtually photograph the model projection surface UF. The virtual fisheye lens N is positioned at a position corresponding to the setting position of the projector 3 relative to the three-dimensional projection surface F. That is, the relative position of the projector 3 relative to the three-dimensional projection surface F is the same as the relative position of the virtual camera CA relative to the model projection surface UF. Similarly, as Figure 3As shown in (D), in specific example 3, the control unit 11 acquires two-dimensional coordinate data when a virtual fisheye lens N using a virtual camera CA virtually captures a model projection surface UK. The virtual fisheye lens N is preferably the same as or has the same lens characteristics as the fisheye lens 4 of the projector 3. The virtual fisheye lens N is positioned at a position corresponding to the setting position of the projector 3 relative to the three-dimensional projection surface K. That is, the relative position of the projector 3 relative to the three-dimensional projection surface K is the same as the relative position of the virtual camera CA relative to the model projection surface UK.

[0038] The processing in S43 is performed using a vertex shader pre-configured on GPU 22 (S43). The vertex shader converts the information of the three-dimensional polygons representing the three-dimensional model into a cuboid coordinate system. The control unit 11 typically generates a two-dimensional image in the vertex shader based on the projection result of the three-dimensional model using a central projection method. The control unit 11 obtains two-dimensional coordinate data suitable for the projector 3 equipped with the fisheye lens 4 by replacing the central projection method with an isometric projection method corresponding to the lens characteristics of the fisheye lens. In typical perspective-based shooting, the coordinates of the two-dimensional image are calculated by performing a matrix multiplication operation on each vertex of the three-dimensional model using a perspective transformation matrix. On the other hand, in a vertex shader using isometric projection, the coordinates of the two-dimensional image cannot be obtained through simple matrix operations. In this embodiment, the control unit 11 performs the processing in S43 using a shader program. In S43, the control unit 11 can also omit redundant information through processes such as backface culling, cropping, attribute evaluation, and rasterization. The control unit 11 can determine the coordinates of each vertex in the three-dimensional projection plane of the two-dimensional coordinate data based on the vertex data, according to the projection result of the three-dimensional model through the processing of S43.

[0039] The control unit 11 performs rasterization processing (S44) on the vertex data processed in S43. In the rasterization process, the control unit 11 combines each vertex data into a polygon representing the deformable model data of the three-dimensional model, and determines the arrangement of pixels used to depict each polygon on the two-dimensional image data.

[0040] The control unit 11 generates two-dimensional image data (S45) based on the projection result when the image data is virtually projected onto the model projection surface using a virtual projector positioned relative to the model projection surface at a predetermined position, and the two-dimensional coordinate data obtained in S43. The two-dimensional image data is the image signal output to the projector 3. In this embodiment, the control unit 11 generates two-dimensional image data with color information set for the two-dimensional coordinate data based on the color information of the image data and the correspondence between the image data and the two-dimensional coordinate data. The control unit 11 applies pixel shader processing to each pixel processed in S44. In the pixel shader processing, the correspondence between the image data and the two-dimensional coordinate data is determined using the virtual projector determined in S5. When the virtual projector is a standard virtual projector, the control unit 11 determines the correspondence between the image data and the two-dimensional coordinate data based on the skeleton information of the three-dimensional model. When the virtual projector is a fisheye virtual projector or a 360-degree virtual projector, the control unit 11 performs a calculation to map the projection position on the three-dimensional model when the image data is radially projected from a virtual projector positioned at a predetermined position obtained in S7 in a predetermined posture. The prescribed posture in this embodiment is one in which the optical axis W is orthogonal to the first surface of the model's projection plane. Therefore, the control unit 11 determines the correspondence between the image data captured using the fisheye lens, the image data, and the two-dimensional coordinate data. The methods for calculating the projection range of the fisheye virtual projector and the 360-degree virtual projector are different. In the 360-degree virtual projector, the control unit 11 calculates the pitch angle and azimuth angle for each point on the three-dimensional model, with the prescribed position and posture of the virtual projector as the origin. The control unit 11 can determine the texture coordinates of the image data captured by the 360-degree camera based on the pitch angle and azimuth angle. Therefore, the control unit 11 determines the correspondence between the image data captured by the 360-degree camera, the image data, and the two-dimensional coordinate data.

[0041] The control unit 11 generates two-dimensional image data by assigning color and transparency of the two-dimensional coordinate data to each pixel based on the color information of the image data and the correspondence between the image data and the two-dimensional coordinate data. That is, the control unit 11 assigns the color and transparency of the pixel corresponding to the vertex of the polygon to the color and transparency of the position of the vertex in the texture image, which is contained in the texture coordinate data of that vertex. The color and transparency of pixels not corresponding to the vertices of the polygon are determined by interpolation of the color and transparency assigned to the vertices of the polygon. In addition to determining the color and transparency of pixels through pixel shaders, the control unit 11 can also apply alpha (α) testing, depth testing, stencil testing, and blending processes.

[0042] The control unit 11 outputs a two-dimensional image data signal to the projector 3, which is then projected onto a three-dimensional projection surface (S12). The projector 3 projects image data onto the three-dimensional projection surface according to the image signal output from the projection control device 2. In specific example 1, the projector 3 will... Figure 6 The two-dimensional image data G1 shown in (A) is projected onto the three-dimensional projection plane F to obtain a reproduced image. Figure 6 The projection result P1 shown in (B) is the projection result of the image data G1 projected along the three-dimensional projection plane F. The projection result of the image data projected onto the fourth plane F4 and the fifth plane F5 is a U-shape opening in the opposite direction from the projector 3 towards the first plane F1. The center of the image data coincides with the center of the first plane F1. Figure 6 In the comparative example shown in (C), the distortion of the fisheye lens of the virtual projector is included in the projection result P2, and the outline of the projection result P2 is curved. In contrast, the projection result P1 in specific example 1 does not contain the distortion of the fisheye lens, and the outline of the projection result P1 is straight. Compared with the projection result P1, it is a projection result with less distortion and less sense of incongruity.

[0043] In specific example 2, projector 3 projects the two-dimensional image data of image data G2 onto the three-dimensional projection surface F, thus reproducing the image data. Figure 7 The projection result P3 shown in (B) is the projection result of the projector. In specific example 3, the projector 3 will... Figure 7 The two-dimensional image data G2 shown in (A) is projected onto the three-dimensional projection plane K to obtain a reproduced image. Figure 7 The projection result P4 shown in (C) is a projection result. In the three-dimensional projection surface K of projection result P4, the connection between adjacent first surfaces K1 and second surfaces K2 and the connection between first surfaces K1 and third surfaces K3 in the horizontal direction are curved surfaces, and there are no corners in the horizontal direction. However, in either projection result P3 or P4, the shape of the three-dimensional projection surface is irrelevant, resulting in projection results with less distortion and less inconsistency. In specific example 4, the projector 3 projects the two-dimensional image data of image data G3 onto the three-dimensional projection surface F, and obtains a reproduced image. Figure 8 The projection result shown in (B) is the projection result of P5. Figure 8 As shown in (B), in specific example 4, projection is onto the three-dimensional projection plane F. Figure 8 The two-dimensional image data of image data G3 shown in (A) was reproduced. Figure 8The projection result P5 shown in (B) is the projection result of the previous projection system. As in Examples 1 to 4, in projection system 1, projection results with reduced inconsistency are obtained from various image data with different shooting conditions. The control unit 11 determines whether to end the projection control process (S31). The control unit 11 determines to end the projection control process when the projection of the last frame has ended or when it detects an instruction to end the projection control process input by the user via the input unit 14.

[0044] If no instruction to end projection control processing is detected (S31: No), the control unit 11 determines whether a position change instruction to change the projection position has been detected (S32). If the projection position is changed, the correspondence between the representative point of the three-dimensional projection surface and the representative point of the image data is changed. The representative point is, for example, the center point. If the user wants to change the projection position, they operate the joystick of the input unit 14 to specify the desired projection position. For example, when the user wants to move the projection result to the right from the current position, the user tilts the joystick to the right. If a position change instruction is detected (S32: Yes), the control unit 11 changes the setting of the projection position of the image data relative to the model projection surface according to the position change instruction detected in S32 (S33). In S45, which is executed after S11 following S33, the control unit 11 uses a virtual projector configured at a specified position to generate two-dimensional image data that virtually projects the image data onto the projection position indicated by the position setting instruction on the model projection surface. When the virtual projector is determined to be a standard virtual projector, the control unit 11 changes the pasting position of the texture of the image data acquired in S41 by moving it up, down, left, and right according to the position change instruction. When the virtual projector is determined to be a fisheye virtual projector or a 360-degree virtual projector, the control unit 11 changes the direction (angle) of the virtual projector according to the position change instruction. Figure 9 As shown in (B), regarding the projection result P6 of specific example 1, if an instruction to move the projection result to the right is detected in S32 (S32: Yes), in S12, the projection... Figure 9 The projection result shown in (A) is P7.

[0045] If no position change instruction is detected (S32: No), or after S33, the control unit 11 determines whether a magnification change instruction for changing the projection magnification has been detected (S34). If the user wants to change the projection magnification, they operate the directional keys of the input unit 14 to specify the desired projection magnification. For example, if the user wants to increase the projection magnification, they press the up key on the directional keys. If a magnification change instruction is detected (S34: Yes), the control unit 11 changes the setting of the projection magnification of the image data (S35). In S41, executed after S35 in S11, the control unit 11 uses a virtual projector positioned at a predetermined location to obtain the projection result when the image data is virtually projected onto the model projection surface at the projection magnification indicated by the magnification change instruction. Figure 9 As shown, regarding the projection result P6 of specific example 1, if an indication to increase the projection magnification is detected in S32 (S32: Yes), in S12, the projection... Figure 9 The projection result P8 is shown in (C). If no indication to change the projection magnification is detected (S34: No), or after S35, the control unit 11 returns the process to S11. If an indication to end the projection control process is detected (S31: Yes), the control unit 11 stores the current settings in the storage unit 15 and ends the above process.

[0046] In the above embodiments, projection system 1, projection control device 2, projector 3, and fisheye lens 4 are examples of the projection system, projection control device, projector, and fisheye lens of the present invention, respectively. Process S6 is an example of the model setting process of the present invention. Process S4 is an example of the image acquisition process of the present invention. Process S5 is an example of the virtual projection determination process of the present invention. Process S43 is an example of the two-dimensional coordinate data conversion process of the present invention. Process S45 is an example of the two-dimensional image data generation process of the present invention. Process S21 is an example of the reference projection process of the present invention. Process S22 is an example of the acceptance process of the present invention. Process S23 is an example of the correction process of the present invention. Process S32 is an example of the position change indication acquisition process of the present invention. Process S34 is an example of the magnification change indication acquisition process of the present invention.

[0047] The projection system 1 of the above embodiment includes: a projector 3 having a fisheye lens 4, and a projection control device 2 for controlling the image projected by the projector 3. The projection control device 2 performs image acquisition processing (S4) to acquire image data that is the object of processing. The projection control device 2 acquires model parameters including the size and shape of a three-dimensional projection surface that is concave, and performs model setting processing (S6) to set a three-dimensional model. The projection control device 2 performs virtual projector determination processing (S5) to determine a virtual projector from multiple virtual projectors based on the image data acquisition conditions. The projection control device 2 uses a fisheye lens arranged at a setting position on the projector 3 to virtually capture a model projection surface that represents the three-dimensional projection surface using a three-dimensional model, and performs two-dimensional coordinate data conversion processing (S43) to acquire two-dimensional coordinate data of the model projection surface. The projection control device 2 performs two-dimensional image data generation processing (S45) to generate two-dimensional image data based on the projection result and two-dimensional coordinate data when the image data is virtually projected onto the model projection surface using a virtual projector arranged at a predetermined position relative to the model projection surface. The projection control device 2 performs image projection processing (S12) by using the projector 3 to project two-dimensional image data onto a three-dimensional projection surface.

[0048] Typically, when an image is projected using a projector with a fisheye lens, distortion characteristic of fisheye lenses is produced in the projected image. Similarly, when an image is captured using a camera with a fisheye lens, distortion characteristic of fisheye lenses is produced in the captured image. Here, since capturing and projecting are inverse transformations, if image data captured with a fisheye lens is projected using a fisheye lens with the same lens characteristics as the one used in capturing, the distortion from capturing is canceled out by the projection. Thus, even without fisheye lens-specific correction, a projected image with minimal distortion can be obtained. The projection control device 2 of projection system 1 uses a virtual projector corresponding to image data for each of the multiple capturing conditions to simulate an ideal projection environment corresponding to the capturing conditions based on the projection results. The projection control device 2 of projection system 1 converts the projection results in the ideal projection environment into two-dimensional image data captured from the actual setting position of the projector 3 using the fisheye lens N. Thus, the projection control device 2 of projection system 1 can generate two-dimensional image data that can reproduce the ideal projection environment through projection from the real projection environment corresponding to the capturing conditions through relatively simple processing. That is, the projection control device 2 can perform planar projection and stereoscopic projection corresponding to the shooting conditions within the same device. Therefore, based on the image data of multiple shooting conditions, the projection system 1 can obtain a projected image with less dissonance to the viewer and less distortion than before, regardless of the setting position of the projector 3, even when the projection surface has an arbitrary stereoscopic shape, through simpler processing than before.

[0049] The virtual projector selection process of S5 is as follows: When shooting conditions include using a fisheye lens, a fisheye virtual projector that virtually projects image data along the model projection plane using a virtual fisheye lens is selected as the chosen virtual projector. The virtual projector selection process is as follows: When shooting conditions include using a standard lens, a standard virtual projector that virtually projects image data along the model projection plane is selected as the chosen virtual projector. The projection system 1 can project image data captured using a fisheye lens and image data captured using a standard lens from the projector 3 onto a three-dimensional projection surface composed of concave surfaces in a manner with reduced distortion compared to previous methods.

[0050] The model setting process in S6 is the process of obtaining model parameters including the three-dimensional projection surface to set the three-dimensional model (S7). The three-dimensional projection surface has five surfaces, including a first surface and four surfaces continuous with the first surface. The projection system 1 sets the three-dimensional model through relatively simple settings. Regardless of the image data shooting conditions and the setting position of the projector 3, it can obtain a projected image with less disharmony for the viewer through simpler processing than before.

[0051] The model setting process of S6 involves obtaining model parameters, including those of a three-dimensional projection surface, to set up the three-dimensional model. This three-dimensional projection surface has five surfaces, including a first surface opposite the fisheye lens and four surfaces continuous with the first surface. The specified position determines the position where the optical axis W of the virtual projector lens is orthogonal to the first surface. The projection system 1 sets the specified position to be opposite the first surface in the three-dimensional projection surface. Therefore, compared with other conditions where the specified position is not opposite the first surface, it is easier to simulate an ideal projection environment and easier to obtain and process the projection results.

[0052] The model setting process of S6 involves obtaining model parameters for a three-dimensional projection surface containing a first surface opposite the fisheye lens 4 of the projector 3, and setting the three-dimensional model. The defined position is the location where the optical axis W of the lens L of the virtual projector PR passes through the center M of the first surface of the three-dimensional projection surface. In projection system 1, the defined position is located on the extension line of an imaginary line extending from the center of the first surface of the three-dimensional projection surface in a direction orthogonal to the first surface. Therefore, compared to other conditions, projection system 1 can easily simulate an ideal projection environment and can easily perform the process of virtually projecting image data onto the three-dimensional projection surface.

[0053] The model setting process of S6 involves obtaining model parameters, including those of a three-dimensional projection surface, to set the three-dimensional model. This three-dimensional projection surface has a first surface that is opposite to the fisheye lens 4 of the projector 3 and extends vertically, a second and a third surface that intersect the first surface and extend vertically, and a fourth and a fifth surface that intersect the first surface and extend horizontally. Alternatively, the first and second surfaces can be connected by a first curved surface, and the first and third surfaces can be connected by a second curved surface. Generally, in a three-dimensional projection surface, the seams between the first and second surfaces, and between the first and third surfaces, which are corners, are more prone to image distortion compared to a flat surface. Compared to a three-dimensional projection surface without a first and a second curved surface, the projection system 1 is less prone to distortion at the seams between the first and second surfaces and between the first and third surfaces in the projection result projected onto the three-dimensional projection surface.

[0054] The image data includes shooting conditions. The virtual projector selection process determines the virtual projector from among multiple virtual projectors based on the shooting conditions attached to the image data. Since projection system 1 can determine the virtual projector based on the shooting conditions attached to the image data, the user of projection system 1 is spared the trouble of inputting shooting conditions.

[0055] The two-dimensional image data generation process is based on the color information of the image data and the correspondence between the image data and the two-dimensional coordinate data, and generates two-dimensional image data with color information set for the two-dimensional coordinate data (S45). The projection system 1 can generate two-dimensional image data with color information set for the two-dimensional coordinate data through relatively simple processing based on the color information of the image data and the correspondence between the image data and the two-dimensional coordinate data.

[0056] The projection control device 2 performs a reference projection process (S21) that projects a reference image representing the setting status of the 3D model onto a 3D projection surface via the fisheye lens 4. After the projection of the reference image begins, the projection control device 2 performs a setting change instruction acceptance process (S22) that accepts changes to the settings of the 3D model. Based on the setting change instruction, the projection control device 2 changes the settings of the 3D model and performs a correction process (S23) that corrects the reference image projected onto the 3D projection surface. The projection system 1 can project a reference image onto the actual 3D projection surface while simultaneously changing the settings of the 3D model through relatively simple processing. The user of the projection system 1 can confirm the setting status of the 3D model by viewing the reference image projected onto the 3D projection surface.

[0057] The projection control device 2 performs the following position change instruction acquisition process: acquiring a position change instruction (S32) to change the projection position of the image data relative to the model projection surface. The two-dimensional image data generation process upon acquiring the position change instruction is the process of generating two-dimensional image data in the model projection surface where the image data is virtually projected onto the projection position indicated by the position change instruction. The projection system 1 can project image data with the changed projection position onto the three-dimensional projection surface through relatively simple processing.

[0058] The projection control device 2 executes a magnification change instruction acquisition process (S34) to obtain a magnification change instruction for the changed image data. The two-dimensional image data generation process, upon obtaining the magnification change instruction, generates two-dimensional image data in which the image data is virtually projected onto the model projection plane at the magnification indicated by the magnification change instruction. The projection system 1 can project image data with the set magnification change onto the three-dimensional projection plane through relatively simple processing.

[0059] The projection system, projection control device, and projection control program of the present invention are not limited to the embodiments described above, and various modifications can be made without departing from the spirit of the present invention. For example, the following modifications can also be appropriately made.

[0060] The structure of the projection system, projection control device, and projection control program can be modified appropriately. The projector 3 and the projection control device 2 can also be an integrated device. The projection control device can be a dedicated device or a general-purpose PC, etc.

[0061] The projection control program, containing instructions for performing projection control processing, only needs to be stored in the storage unit 15 or memory 23 before the control unit 11 of the projection control device 2 executes the projection control program. Therefore, the method of obtaining the projection control program, the acquisition path, and the device for storing the program can be appropriately changed. The projection control program can also be received from other devices via cable or wireless communication and stored in the storage unit 15 or memory 23. Other devices include, for example, PCs and servers connected via a network.

[0062] The steps of the projection control processing are not limited to the example where they are executed by the control unit 11; they may also be partially or entirely executed by other electronic devices (e.g., ASICs). For example, the other electronic device is an ASIC. The steps of the projection control processing may also be processed separately by multiple CPUs. The order of the steps of the projection control processing can be changed, steps can be omitted, or steps can be added as needed. The following modifications may also be appropriately applied to the processing performed by the projection system 1, including the projection control processing.

[0063] The shape of the 3D projection surface only needs to be concave, but can be appropriately changed to a hemispherical surface, etc. The processing for changing the 3D model can also be appropriately modified. If the projection control device 2 is equipped with sensors that measure the distance and shape of the 3D projection surface, the projection control device 2 can also automatically set the 3D model based on the sensor detection results. The pattern of the reference image can be appropriately modified; for example, it can be a pattern that only represents the shape of the first surface. The projection control device 2 may also not perform the processing of changing the 3D model using the reference image. Regarding the 3D model, the modifiable items can be appropriately modified; for example, the function of correcting the surface to a trapezoidal shape may be omitted, and the function of correcting the connection between two adjacent surfaces among the multiple surfaces constituting the 3D projection surface to a curved surface may also be omitted. The connection between two adjacent surfaces in the vertical direction among the multiple surfaces constituting the 3D projection surface can also be a curved surface. If the size and shape of the 3D projection surface, such as screen E, are known, the 3D model 67 can also be set based on model parameters including the size and shape of the 3D projection surface input by the user.

[0064] The preset types of virtual projectors can be appropriately changed. Any two of the standard virtual projectors, fisheye virtual projectors, and 360-degree virtual projectors described in the above embodiments can also be used as multiple types of virtual projectors. Multiple types of virtual projectors can include projectors other than standard virtual projectors, fisheye virtual projectors, and 360-degree virtual projectors. Image data can be generated without additional shooting conditions, and the control unit 11 can set the virtual projector selected by the user as the determining virtual projector. The control unit 11 can determine the determining virtual projector based on the aspect ratio of the image. The predetermined position of the determining virtual projector may not be the position where the optical axis of the determining virtual projector's lens is orthogonal to the first surface of the model projection surface, or it may not be the position where the optical axis of the determining virtual projector's lens passes through the center of the first surface of the model projection surface. S32 to S35 can also be appropriately omitted. In the two-dimensional image data generation process of S45, two-dimensional image data can be generated based on the projection result and the two-dimensional coordinate data obtained in S43; the method for generating the two-dimensional image data can be appropriately modified.

[0065] The above variations can also be appropriately combined within a non-contradictory scope. In addition to the combinations exemplified within the claimed scope, the applicant intends to also obtain patent rights for combinations that do not depart from the spirit of the invention and do not create contradictions.

[0066] Explanation of icon numbers

[0067] 1: Projection system; 2: Projection control device; 3: Projector; 4: Fisheye lens; 11: Control unit; 12: Communication unit; 14: Input unit; 15: Storage unit; 21: CPU; 22: GPU; 23: Memory.

Claims

1. A projection system comprising: a projector having a fisheye lens; and a projection control device for controlling the image projected by the projector, characterized in that, The projection control device performs the following processing: Image acquisition and processing: acquiring image data as the object to be processed; The model setting process obtains model parameters including the size and shape of the three-dimensional projection surface to set the three-dimensional model. The three-dimensional projection surface is concave. The virtual projector selection process determines the virtual projector from among multiple virtual projectors based on the shooting conditions of the image data. Two-dimensional coordinate data conversion processing: using a fisheye lens configured at a set position on the projector, virtually photographing the model projection surface that represents the three-dimensional projection surface using the three-dimensional model, and obtaining two-dimensional coordinate data that represents the model projection surface using two-dimensional coordinates; Two-dimensional image data generation processing generates two-dimensional image data based on the projection result of virtually projecting the image data onto the model projection surface using the determined virtual projector configured at a specified position relative to the model projection surface and the two-dimensional coordinate data. as well as Image projection processing involves using the projector to project the two-dimensional image data onto the three-dimensional projection surface.

2. The projection system according to claim 1, characterized in that, The virtual projector's decision-making process is as follows: When the shooting conditions include shooting with a fisheye lens, a fisheye virtual projector is determined as the determined virtual projector, which uses a virtual fisheye lens to virtually project the image data along the model projection plane. When the shooting conditions include shooting with a standard lens, a standard virtual projector is selected as the selected virtual projector, which virtually projects the image data along the model projection surface.

3. The projection system according to claim 1 or 2, characterized in that, The model setting process is a process of obtaining the model parameters containing the three-dimensional projection surface to set the three-dimensional model. The three-dimensional projection surface has five surfaces, including a first surface and four surfaces continuous with the first surface.

4. The projection system according to claim 2, characterized in that, The model setting process involves obtaining the model parameters, including the three-dimensional projection surface, to set the three-dimensional model. The three-dimensional projection surface has five surfaces, including a first surface opposite the fisheye lens and four surfaces continuous with the first surface. The specified position is the position where the optical axis of the lens of the virtual projector is orthogonal to one of the planes of the model's projection surface.

5. The projection system according to claim 2, characterized in that, The model setting process involves obtaining the model parameters, including the three-dimensional projection surface, to set the three-dimensional model. The three-dimensional projection surface has a first surface that faces the fisheye lens of the projector. The specified position is the location where the optical axis of the lens of the virtual projector passes through the center of the first surface of the model's projection surface.

6. The projection system according to claim 1 or 2, characterized in that, The model setting process involves obtaining the model parameters, including the three-dimensional projection surface, to set the three-dimensional model. The three-dimensional projection surface has a first surface that is opposite to the fisheye lens of the projector and extends vertically, a second and a third surface that intersect the first surface and extend vertically, and a fourth and a fifth surface that intersect the first surface and extend horizontally. The first surface and the second surface are connected by a first curved surface, and the first surface and the third surface are connected by a second curved surface.

7. The projection system according to claim 1 or 2, characterized in that, The image data includes the shooting conditions. The virtual projector determination process is a process of determining the virtual projector from among the plurality of virtual projectors based on the shooting conditions attached to the image data.

8. The projection system according to claim 1, characterized in that, The two-dimensional image data generation process is based on the correspondence between the image data and the two-dimensional coordinate data and the color information of the image data, to generate two-dimensional image data with color information set for the two-dimensional coordinate data.

9. The projection system according to claim 1, characterized in that, The projection control device further performs the following processing: The reference projection process projects a reference image representing the set state of the three-dimensional model onto the three-dimensional projection surface via the fisheye lens. The system accepts and processes requests to change settings of the three-dimensional model after the projection of the reference image has begun. as well as The correction process involves changing the settings of the three-dimensional model according to the setting change instruction, and correcting the reference image projected onto the three-dimensional projection surface.

10. The projection system according to claim 1, characterized in that, The projection control device also performs a position change indication acquisition process, in which it acquires a position change indication that changes the projection position of the image data relative to the model projection surface. The two-dimensional image data generation process, when the position change indication is obtained, is the process of generating two-dimensional image data in which the image data is virtually projected onto the projection position indicated by the position change indication on the model projection plane.

11. The projection system according to claim 1, characterized in that, The projection control device also performs a magnification change indication acquisition process, in which a magnification change indication for changing the projection magnification of the image data is acquired. The two-dimensional image data generation process, when the magnification change indication is obtained, is the process of generating two-dimensional image data in which the image data is virtually projected onto the model projection surface at the projection magnification indicated by the magnification change indication.

12. A projection control device comprising a control unit for controlling an image projected by a projector having a fisheye lens, characterized in that, The control unit performs the following processing: Image acquisition and processing: acquiring image data as the object to be processed; The model is set up by obtaining model parameters including the size and shape of the three-dimensional projection surface, and setting the three-dimensional model so that the three-dimensional projection surface is concave. The virtual projector selection process determines the virtual projector from among multiple virtual projectors based on the shooting conditions of the image data. Two-dimensional coordinate data conversion processing: using a fisheye lens configured at a set position on the projector, virtually photographing the model projection surface that represents the three-dimensional projection surface using the three-dimensional model, and obtaining two-dimensional coordinate data that represents the model projection surface using two-dimensional coordinates; Two-dimensional image data generation processing generates two-dimensional image data based on the projection result of virtually projecting the image data onto the model projection surface using the determined virtual projector configured at a specified position relative to the model projection surface and the two-dimensional coordinate data. Image projection processing, which projects the two-dimensional image data from the projector.

13. A projection control program, executed by a projection control device, the projection control device controlling the image projected by a projector with a fisheye lens, characterized in that, The projection control program includes instructions for the projection control device to perform the following processing: Image acquisition and processing: acquiring image data as the object to be processed; Model setting process: Obtain model parameters including the size and shape of the three-dimensional projection surface (which is concave), and set the three-dimensional model. The virtual projector selection process determines the virtual projector from among multiple virtual projectors based on the shooting conditions of the image data. Two-dimensional coordinate data conversion processing: using a fisheye lens configured at a set position on the projector, virtually photographing the model projection surface that represents the three-dimensional projection surface using the three-dimensional model, and obtaining two-dimensional coordinate data that represents the model projection surface using two-dimensional coordinates; Two-dimensional image data generation processing generates two-dimensional image data based on the projection result of virtually projecting the image data onto the model projection surface using the determined virtual projector configured at a specified position relative to the model projection surface and the two-dimensional coordinate data. as well as Image projection processing, which projects the two-dimensional image data from the projector.

Images