Data output program

Abstract

To provide a data output program that allows setting the placement of an image on a three-dimensional projection surface, including multiple planes, using simpler processing than conventional methods. [Solution] The data output program includes instructions to the data output device to perform image acquisition processing, model setting processing, video size setting processing, and output processing. The data output device acquires the source image data that will be the basis of the input image data (S44). The data output device sets a three-dimensional model (S43). When the video size is set by the first method, the data output device sets the video size to a predetermined ratio of the size of the three-dimensional projection plane. When the video size is set by the second method, the data output device sets the video size to a size specified by an absolute value, independent of the size of the three-dimensional projection plane. The data output device outputs the input image data or the setting data (S64, S66).

Description

[[Technical Field]]

[0001] The present invention relates to a data output program. [[Background Art]]

[0002] In a conventional content generation method, a system generates image content to be projected by a projector onto a projection position in real space. In a first acquisition process, the system acquires spatial information relating to a virtual space corresponding to the real space. In a second acquisition process, the system acquires display position information relating to a display position corresponding to the projection position in the virtual space. In a third acquisition process, the system acquires reference position information relating to a reference position in the virtual space. In a generation process, the system generates, as the image content, an image including an object viewed from the reference position when the object is displayed at the display position in the virtual space, based on the spatial information, the reference position information, and the display position information. [[Prior Art Document]] [[Patent Document]]

[0003] [[Patent Document 1]] Japanese Unexamined Patent Application Publication No. 2021-182374 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0004] When a projector projects an image onto a three-dimensional projection surface including a plurality of planes, a projection method of individually projecting an image onto each plane and a projection method of projecting a continuous image across the plurality of planes are conceivable. In the projection method of projecting a continuous image across the plurality of planes, it is necessary for the user of the projector to individually set how the image is arranged in accordance with the size and shape of the three-dimensional projection surface, which is cumbersome.

[0005] The object of the present invention is to provide a data output program that allows setting the arrangement of an image on a three-dimensional projection surface, which includes multiple planes, with a simpler process than conventional methods. [Means for solving the problem]

[0006] A data output program according to one aspect of the present invention is a data output program to be executed by a data output device that outputs data relating to input image data for projecting an image onto a projector. The data output program includes instructions to the data output device to execute image acquisition processing, model setting processing, image size setting processing, and output processing. The image acquisition processing is the process of acquiring original image data, which is the source of the input image data. The model setting processing is the process of acquiring model parameters, including the size and shape of a three-dimensional projection surface including a plurality of planes, and setting a three-dimensional model. The image size setting processing is the process of setting the image size, which is the size of the image shown by the original image data on the three-dimensional projection surface. When the image size is set by the first method, the image size setting processing is the process of setting the image size to a predetermined ratio of the size of the three-dimensional projection surface. When the image size is set by the second method, the image size setting processing is the process of setting the image size to a size specified by an absolute value, independent of the size of the three-dimensional projection surface. The output process uses a projection lens positioned according to the installation location of the projector to output the input image data or setting data to a model projection surface representing the three-dimensional projection surface using the three-dimensional model. The input image data is data that projects the image of the set image size onto the projector. The setting data is data used to output the input image data.

[0007] The video size setting process of the data output program sets the video size using different methods depending on whether the video size is set using the first method or the second method. Therefore, the video size setting process of the data output program contributes to setting the placement of the video on the three-dimensional projection plane compared to setting the video size using a single method. [Brief explanation of the drawing]

[0008] [Figure 1] This is a conceptual diagram of a projection system 1 having a projector 3 suspended from the ceiling of room R. [Figure 2] Figure 2(A) is a perspective view of the three-dimensional model U set in the projection control process. Figure 2(B) is an explanatory diagram of the virtual projector PR, which is positioned at a predetermined location relative to the model projection plane UF of the three-dimensional model U, and the virtual camera CA, which is positioned at the installation location. [Figure 3] This is a flowchart of the data output process performed by data output device 2. [Figure 4] Figure 4(A) is an explanatory diagram of the settings screen G1. Figure 4(B) is an explanatory diagram of the first setting field G6 displayed in section G5 of the settings screen G1 when the projection type is planar projection. Figure 4(C) is an explanatory diagram of the second setting field G12 displayed in section G5 of the settings screen G1 when the projection type is equirectangular projection. [Figure 5] Figures 5(A) to 5(D) are explanatory diagrams of the video shown in the original image data. [Figure 6] This is an explanatory diagram of the relative coordinates set on the model projection plane UF. [Figure 7] This is an explanatory diagram of the reference point set in image J2. [Figure 8] Figures 8(A) to 8(C) are explanatory diagrams of the simulation images obtained by virtually projecting image J1 onto the model projection plane UF. [Figure 9] Figure 9(A) is an explanatory diagram of the simulation image obtained by virtually projecting images J1 and J2 onto the model projection plane UF. Figures 9(B) and 9(C) are explanatory diagrams of the simulation image obtained by virtually projecting images J1 and J2 onto the model projection plane UF. [Figure 10]Figure 10(A) is an explanatory diagram of a simulation image obtained by virtually projecting images J4 and J2 onto the model projection plane UF. Figure 10(B) is an explanatory diagram of a simulation image obtained by virtually projecting onto the model projection plane UF an image obtained by superimposing an image of image J4 projected equirectangularly onto the model projection plane UF and an image of image J2 projected planarly onto the model projection plane UF. [Figure 11] This is a flowchart of the projection control process performed by data output device 2. [Figure 12] Figure 12(A) is an explanatory diagram of the three-dimensional projection plane F. Figure 12(B) is an explanatory diagram of the reference image Q projected onto the three-dimensional projection plane F during the projection control process. Figure 12(C) is an explanatory diagram of the reference image Q projected onto the three-dimensional projection plane F when the settings of the three-dimensional model U are changed during the projection control process. [Figure 13] This is a flowchart of the transformation process performed in the projection control process. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. As shown in Figure 1, the projection system 1 comprises a projector 3 having a projection lens 4 and data output devices 2 and 5. The projection system 1 has the function of outputting input image data projected by the projector 3 installed in a room R of an amusement facility, such as a karaoke bar. The projection system 1 can be placed in any position; for example, the projector 3 can be suspended from the ceiling of the room R.

[0010] The data output device 2 outputs data relating to the input image data to be projected onto the projector 3. The data relating to the input image data is either the input image data itself or setting data. The setting data is the data used to output the input image data. In this embodiment, the setting data includes the source image data that forms the basis of the input image data and the setting of the image size, which is the size of the image represented by the source image data.

[0011] The data output device 5 generates input image data using the setting data output from the data output device 2. The data output device 5 may acquire the setting data via the network 6, or may store the setting data from a storage device such as a USB memory. The data output device 5 is connected to the projector 3 to each other via a wired or wireless connection. The data output device 5 outputs the generated input image data to the projector 3, and causes the projector 3 to project the video represented by the input image data.

[0012] The projector 3 projects the input image data onto a three-dimensional projection plane F including a plurality of planes in accordance with the image signal output from the data output device 5. The projector 3 of the present embodiment includes a fisheye lens as the projection lens 4. The projector 3 is configured as a single unit capable of projecting input image data onto a three-dimensional projection plane F including a plurality of planes. The shape of the three-dimensional projection plane F may be any shape as long as it includes a plurality of planes. The three-dimensional projection plane F of the present embodiment is the three-dimensional projection plane F shown in FIG. 1.

[0013] As shown in FIG. 1, the three-dimensional projection plane F has five surfaces among the six surfaces of a cubic room R. The five surfaces include a first surface F1 facing the projection lens 4 of the projector 3, and four surfaces F2 to F5 continuous with the first surface F1.

[0014] The first surface F1 is a surface extending in the vertical direction and the horizontal direction. The second surface F2 and the third surface F3 are each orthogonal to the first surface F1, and extend in the vertical direction and the horizontal direction. The fourth surface F4 and the fifth surface F5 are each orthogonal to the first surface F1, and extend in the horizontal direction. In the present embodiment, the first surface F1 is a wall surface of the room R, but the first surface F1 may be a ceiling surface or a floor surface of the room R. The first surface F1 may be connected to another plane via a curved surface.

[0015] The data output device 2 is, for example, a general-purpose PC. The data output device 2 includes a display 24, an input unit 25, a storage device 26, a communication unit 27, and a control unit 20. The display 24, the input unit 25, the storage device 26, and the communication unit 27 are connected to the control unit 20.

[0016] The display 24 is configured to display images. The input unit 25 accepts operation instructions from a user of the projection system 1, and outputs a signal corresponding to the operation instruction to the control unit 20. The input unit 25 is, for example, a keyboard and a mouse.

[0017] The storage device 26 is, for example, a hard disk drive. An operating system, various programs, and the like are stored in the storage device 26. The various programs include a program that causes the CPU 21 of the control unit 20 to execute data output processing and projection control processing. The storage device 26 stores various parameters and the like used for outputting data relating to input image data.

[0018] Specifically, the storage device 26 stores original image data that is a source of the input image data, and internal parameters and external parameters of the projector 3. The storage device 26 stores internal parameters and external parameters of a virtual projector PR described later, and internal parameters and external parameters of a virtual camera CA.

[0019] The control unit 20 is configured of a CPU 21 as a computer, a GPU 22, a memory 23, and the like. The GPU 22 is a processor provided separately from the CPU 21, and performs image processing based on instructions from the CPU 21. The memory 23 is configured of, for example, a ROM, a RAM, and the like.

[0020] The data output device 5 is, for example, a general-purpose PC. The data output device 5 may be disposed at any position in a room R, or may be disposed at a location different from the room R. The data output device 5 includes a display 54, an input unit 55, a storage device 56, a communication unit 57, and a control unit 50. The control unit 50 is configured of a CPU 51 as a computer, a GPU 52, a memory 53, and the like. Since the data output device 5 of the present embodiment has the same configuration as the data output device 2, detailed description thereof is omitted.

[0021] Referring to Figures 2(A) to 10(B), the data output process performed by the data output device 2 will be described. In the data output process, the control unit 20 performs the process of outputting input image data or setting data based on the original image data specified by the user. The data output process may also be performed by the data output device 5.

[0022] As a specific example, we will explain the case where setting data is generated using the following images as source images: Image J1 shown in Figure 5(A), Image J2 shown in Figure 5(B), Image J3 shown in Figure 5(C), and Image J4 shown in Figure 5(D). Image J1 is a planar image of an illustration showing the interior of a horizontally oriented room. Image J2 is a planar image of an illustration showing a bear character. In Image J2, the background of the bear character is transparent. Image J3 is a planar image of a poster depicting a person and the words "Hello!". Image J4 is an omnidirectional image of a stage venue taken with an omnidirectional camera.

[0023] In a specific example, the control unit 20 outputs input image data or setting data for projection by the projector 3 using a virtual projector corresponding to the projection type. The input image data is two-dimensional image data. The projection type is set according to the type of image represented by the original image data. In the data output processing of this embodiment, the control unit 20 accepts selection of the projection type from planar projection and equirectangular projection.

[0024] Planar projection is a projection method that uses image data representing a normal two-dimensional image as the source image data and projects the image represented by the source image data so as to be placed along the plane of the three-dimensional projection surface F. Equirectangular projection is a projection method that uses image data representing an equirectangular image as the source image data. An equirectangular image is an image represented by equirectangular projection and is exemplified by images taken with a 360-degree omnidirectional camera or a 180-degree panoramic camera. In equirectangular projection, the image represented by the source image data is unfolded into a plane by central projection and projected so as to be placed along the plane of the three-dimensional projection surface F.

[0025] Image data may represent a photograph or an illustration. Image data may represent a still image or a video.

[0026] A virtual projector is a set of parameters virtually configured to calculate the correspondence between image data and pixels in a three-dimensional model U. In this embodiment, the data output device 2 pre-configures two types of virtual projectors. The data output device 2 stores the shader programs and parameters necessary for virtual projection using each virtual projector in the storage device 26.

[0027] The two types of virtual projectors are the standard virtual projector and the 360-degree virtual projector. The standard virtual projector virtually projects the original image data along a model projection surface UF, which represents the three-dimensional projection surface F using a three-dimensional model U. The 360-degree virtual projector virtually projects 360 degrees onto the model projection surface UF so that the elevation angle and azimuth angle match the reference point as viewed from the virtual projector. The simulation image of the 360-degree virtual projector is virtually projected in 360 degrees. In reality, the simulation image of the 360-degree virtual projector is output up to the projection range of projector 3. The projection range of projector 3 is determined by the lens characteristics of projection lens 4, which are obtained in S9 described later.

[0028] The data output process is initiated when the user inputs a start command. When the control unit 20 detects the start command, it reads the data output program for executing the data output process, which is stored in the storage device 26, into the memory 23. The control unit 20 executes the following steps according to the instructions contained in the program read into the memory 23. Various parameters necessary for executing the data output process are stored in the storage device 26. Various data obtained during the data output process are stored in the memory 23 as appropriate.

[0029] In the following explanation, the vertical direction of the three-dimensional model U is also referred to as the height direction. Among the directions along the model projection plane UF, the direction parallel to the horizontal direction is also referred to as the width direction. If the first plane UF1 is defined as a plane that extends in the vertical and horizontal directions, then the width direction of the first plane UF1 is the horizontal direction. The width directions of the second plane UF2 and the third plane UF3 are the front-to-back direction. In the rectangular image shown by the original image data, the direction corresponding to the height direction of the model projection plane UF is also referred to as the height direction of the image. The direction corresponding to the width direction of the model projection plane UF is also referred to as the width direction of the image.

[0030] As shown in Figure 3, the control unit 20 retrieves the initial settings stored in the storage device 26 (S40). The initial settings include the settings of the current three-dimensional model U. The settings of the current three-dimensional model U are expressed using model parameters, including the size and shape of the three-dimensional projection plane F set in the three-dimensional model U.

[0031] The control unit 20 displays the setting screen G1 shown in Figure 4(A) on the display 24 (S41). As shown in Figure 4(A), the setting screen G1 includes fields G2 to G5. Field G2 displays an image virtually projected onto the model projection plane UF according to the current settings of the three-dimensional model U.

[0032] Column G3 shows the layer settings for the video indicated by the source image data. Column G3 displays the icons corresponding to the source image data and the source image data filenames in layer order from top to bottom. Column G4 shows the opacity and volume settings for the video indicated by the currently selected source image data. Column G5 shows the settings area for configuring the size and placement of the video according to the currently selected projection type.

[0033] The control unit 20 determines whether or not to change the three-dimensional model U (S42). In this embodiment, the three-dimensional model U is set as a cube as shown in Figure 2(A) as the initial value of the three-dimensional model. The data output device 2 in this embodiment can change the shape and length of each face of the current three-dimensional model U. Furthermore, the control unit 20 can make changes to the three-dimensional model U, such as rotating or translating it.

[0034] Typically, the user modifies the three-dimensional model U, which reflects the size of a standard room R. If the user wants to change the settings of the current three-dimensional model U, they operate the input unit 25 to input instructions for the process of changing the settings of the three-dimensional model U. While viewing the three-dimensional model U displayed in column G2, the user can specify the height and width lengths of the first face F1, the width length of the second face F2, the rounding of the corners at the joints of two adjacent faces, etc. The three-dimensional model U may also be modified by loading model data. The model data may be created with a dedicated 3D model editing application, or it may be obtained by scanning the actual room shape with a 3D scanner.

[0035] As shown in Figure 2(A), the three-dimensional model U of this embodiment has framework information B, which is represented by a grid pattern. Framework information B is information for mapping image data as a texture when a standard virtual projector is determined as the determined virtual projector PR.

[0036] The framework information B shown by the three-dimensional model U will be explained using the three-dimensional projection plane F as an example. When the projection type is planar projection, the control unit 20 places the reference point of the image shown by the original image data at an anchor set on one of the first plane F1, second plane F2, or third plane F3. The anchor indicates the position of the reference point of the image in the model projection plane UF. The position of the anchor is indicated by relative coordinates with a predetermined point set on the model projection plane UF as the origin. The image size, which is the size of the image shown by the original image data, is set to a size set by the user. For parts of the image that extend beyond the planes connected horizontally, the control unit 20 projects them so as to be folded along the projection plane.

[0037] The control unit 20 projects the image data of the portion that extends from the first surface F1 to the fourth surface F4 and the fifth surface F5, which are vertically connected to the first surface F1, as follows: That is, the control unit 20 deforms the original image data so that it shrinks toward the center of the surface, as shown by the grid pattern, and then projects it.

[0038] When the control unit 20 detects an instruction to change the settings of the three-dimensional model U (S42: YES), it changes the three-dimensional model U according to the instruction (S43). The control unit 20 changes the model projection plane UF in column G2 in accordance with the change in the three-dimensional model U.

[0039] If the control unit 20 does not detect an instruction to change the settings of the three-dimensional model U (S42: NO), or if it performs the process in S44 after the process in S43, the control unit 20 determines whether or not it has detected an instruction to add source image data (S44). If the user wants to add source image data, the user operates the input unit 25 and inputs an instruction to specify source image data in field G4. If the control unit 20 detects an instruction to add source image data showing image J1 (S44: YES), it performs the process in S45.

[0040] The control unit 20 initially places image J1, which is a video based on the added source image data, on the model projection plane UF. The initial settings may be pre-configured or specified by the user. The control unit 20 updates the model projection plane UF in column G2 to include the video based on the added source image data (S45).

[0041] As an example, the initial settings for this embodiment are as follows: The projection type is planar projection. The image size is set to a relative length of 1 based on height. The placement is such that the center of the image is positioned at the center of the first surface UF1.

[0042] As shown in Figure 8(A), in S45, the control unit 20 displays the simulation image K1, in which image J1 is virtually projected onto the model projection plane UF, in column G2. In the simulation image K1, the size of image J1 is set so that its height matches the height of the model projection plane UF. In the simulation image K1, image J1 is positioned relative to the model projection plane UF so that its center coincides with the center of the first plane UF1. For the portion of the image that extends horizontally from the first plane UF1 to the second plane UF2 and the third plane UF3, the control unit 20 virtually projects it so that it is folded along the projection plane.

[0043] If no instruction to add the original image data is detected (S44: NO), or if, after S45, the control unit 20 determines whether or not an instruction to delete the original image data has been detected (S46). If the control unit 20 detects an instruction to delete the original image data (S46: YES), it performs the process in S47. The control unit 20 deletes the video based on the deleted original image data from the model projection plane UF and updates the model projection plane UF in column G2 (S47).

[0044] If the control unit 20 does not detect an instruction to delete the original image data (S46: NO), or if it determines after S47 whether the original image data is selected (S48). If the original image data is not selected (S48: NO), the control unit 20 returns to processing S42. If the original image data is selected (S48: YES), the control unit 20 determines whether planar projection is selected as the projection type (S49). The projection type may be selected by the user. The projection type may also be automatically selected by the control unit 20 by referring to data that determines the projection type assigned to the selected original image data.

[0045] In the initial settings, planar projection is selected as the projection type (S49: YES), so the control unit 20 displays the first setting field G6 shown in Figure 4(B) in field G5 (S50). As shown in Figure 4(B), the first setting field G6 includes fields G7 to G10. Field G7 displays the selected projection type. In the first setting field G6, the value of field G7 is set to "Planar," indicating planar projection. If the user changes field G7, the control unit 50 sets the projection type by the projector 3 for the selected source image data from a selection of multiple options, including equirectangular projection and planar projection.

[0046] Column G8 shows the method for setting the video size, which is the size of the video shown in the original image data. There are several types of methods for setting the video size in this embodiment, including the first method and the second method. The first method is a method of setting the video size of the video to a predetermined ratio of the size of the three-dimensional projection plane F. There are two types of the first method in this embodiment: "height-based" and "width-based".

[0047] The height-based method specifies the size of the image as a ratio to the height of the three-dimensional projection plane F. The width-based method specifies the size of the image as a ratio to the horizontal length of the first face F1 of the three-dimensional projection plane F. The first method is selected when you want to change the size of the image to match the size of the model projection plane UF.

[0048] The second method involves setting the image size to a size specified by an absolute value, independent of the size of the three-dimensional projection surface F. The second method involves setting the length of the image in the height or width direction to a length specified by the user in meters. The second method is also called the "meter-based" method. The second method is selected, for example, when you want to project an image of a character at life-size, independent of the size of the three-dimensional projection surface F.

[0049] Column G9 shows the size of the video and the amount of movement of the video, depending on the method used to set the video size. In the first setting column G6, the first method is set as the method for setting the video size, which is "height reference". When the first method is used to set the video size, column G9 shows the predetermined ratio of the video size to the reference, i.e., the relative scale. In column G9, the relative scale in the width direction is entered in the "width" column. The relative scale in the height direction is entered in the "height" column. The amount of movement in the width direction is entered in the "positionX" column as a relative scale. The amount of movement in the height direction is entered in the "positionY" column as a relative scale.

[0050] As shown in column G9, when the relative scale in the width and height directions is set to 1 based on height, the control unit 20 sets the image size as follows: That is, the control unit 20 sets the height and width directions of the image at a magnification that makes the height of the image the same as the height of the model projection plane UF.

[0051] When "width reference" is set as the method for setting the image size, and a predetermined ratio of the width and height is set to 1, the control unit 20 sets the image size as follows: That is, the control unit 20 sets the height and width of the image by a factor such that the width of the image is the same as the width of the first surface UF1 of the model projection surface UF.

[0052] When "Metric" is selected as the method for setting the video size, the absolute scale in meters for the width direction is entered in the "width" field of column G9. In other words, column G9 shows the size of the selected video in meters. The absolute scale in meters for the height direction is entered in the "height" field. The amount of movement in the width direction is entered in the "positionX" field in the absolute scale in meters. The amount of movement in the height direction is entered in the "positionY" field in the absolute scale in meters.

[0053] When "metric standard" is set as the method for setting the image size, the control unit 20 obtains the image size when field G9 is changed. If the absolute scale in the height direction is set to 1 meter and the absolute scale in the width direction is set to 0 meters, the control unit 20 sets the image size as follows: That is, the control unit 20 sets the height and width directions of the image at a magnification that makes the height direction length of the image 1 meter, regardless of the size of the model projection plane UF. An absolute scale of 0 meters in the width direction does not indicate an instruction to set the absolute scale to 0 meters, but rather an instruction to set the width direction length at the same magnification as the height direction magnification.

[0054] Column G10 shows the settings for anchors and pivots. In this embodiment, as shown in Figure 6, XY relative coordinates are set on the model projection plane UF. The control unit 20 sets the center of the first plane UF1 in the height and width directions to the origin Q0 of the relative coordinates.

[0055] The X-coordinate indicates the position in the width direction on the model projection plane UF. The Y-coordinate indicates the position in the height direction on the model projection plane UF. In column G10, the X-coordinate of the anchor is entered in the field labeled "anchorX". The Y-coordinate of the anchor is entered in the field labeled "anchorY".

[0056] The control unit 20 sets nine coordinates relative to each plane of the model projection plane UF. When the extension direction of the first plane UF1 is defined as the left-right direction and the up-down direction, the control unit 20 sets points Q0 to Q8 for the first plane UF1.

[0057] The control unit 20 sets the relative coordinates of point Q2 at the upper right corner of the first face UF1 to (1,1). The control unit 20 sets the relative coordinates of point Q4 at the lower right corner of the first face UF1 to (1,-1). The control unit 20 sets the relative coordinates of point Q6 at the lower left corner of the first face UF1 to (-1,-1). The control unit 20 sets the relative coordinates of point Q8 at the upper left corner of the first face UF1 to (-1,1).

[0058] The control unit 20 sets the relative coordinates of the midpoint Q1 between point Q8 and point Q2 to (0,1). The control unit 20 sets the relative coordinates of the midpoint Q3 between point Q2 and point Q4 to (1,0). The control unit 20 sets the relative coordinates of the midpoint Q5 between point Q4 and point Q6 to (0,-1). The control unit 20 sets the relative coordinates of the midpoint Q7 between point Q6 and point Q8 to (-1,0).

[0059] Similar to the first face UF1, the control unit 20 sets points Q8(-1,1), Q7(-1,0), Q6(-1,-1), Q19(-2,1), Q17(-2,0), Q15(-2,1), Q16(-3,1), Q18(-3,0), and Q20(-3,-1) for the second face UF2. The control unit 20 sets points Q2(1,1), Q3(1,0), Q4(1,-1), Q13(2,-1), Q11(2,0), Q9(2,1), Q10(3,0), Q12(3,0), and Q14(3,-1) for the third face UF3. Column G10 indicates that the anchor is point Q0, indicated by (0,0).

[0060] The pivot indicates the position of the reference point of the video shown in the original image data. In this embodiment, the control unit 20 sets nine points for a single video. For example, the control unit 20 sets points P0 to P8 for image J2 shown in Figure 7. Point P0(0,0) is the center point of image J2.

[0061] If we define the height direction of image J2 as the up-down direction and the width direction as the left-right direction, then point P2(1,1) is the upper right corner of image J2. Point P4(1,-1) is the lower right corner of image J2. Point P6(-1,-1) is the lower left corner of image J2. Point P8(-1,1) is the upper left corner of image J2.

[0062] Point P1(0,1) is the midpoint between point P8 and point P2. Point P3(1,0) is the midpoint between point P2 and point P4. Point P5(0,-1) is the midpoint between point P4 and point P6. Point P7(-1,0) is the midpoint between point P6 and point P8. Column G10 indicates that the reference point of the image, i.e., the pivot point, is point P0, indicated by (0,0).

[0063] The control unit 20 determines whether or not an instruction to change the method of setting the video size in column G8 has been detected (S51). If the user wants to change the method of setting the video size, they change the settings indicated in columns G8 and G9. If the control unit 20 detects an instruction to select width-based as the method of setting the video size for image J1 and to change the relative scaling in the width and height directions to 1 (S51: YES), it performs the process in S52.

[0064] The control unit 20 identifies a method for setting the video size of the video shown by the selected source image data. The control unit 20 sets the video's height and width lengths at a magnification that makes the width of image J1 equal to the width of the first surface UF1 of the model projection plane UF. As shown in Figure 8(C), the control unit 20 updates the model projection plane UF to reflect the change in video size (S52).

[0065] If no instruction to change the method of setting the video size is detected (S51: NO), or after processing S52, the control unit 20 performs the processing in S53. The control unit 20 determines whether or not an instruction to change the relative coordinates of the anchor shown in column G10 has been detected (S53). If the control unit 20 has detected an instruction to change the relative coordinates of the anchor (S53: YES), it performs the processing in S54. The control unit 20 sets the video layout so that the video reference point is placed at the changed relative coordinates of the anchor. The control unit 20 updates the model projection plane UF shown in column G2 to reflect the change in the anchor (S54).

[0066] If the control unit 20 has not detected an instruction to change the relative coordinates of the anchor (S53: NO), or after processing S54, the control unit 20 performs the processing in S55. The control unit 20 determines whether or not it has detected an instruction to change the relative coordinates of the pivot shown in column G10, that is, the reference point of the selected image (S55). If the control unit 20 has detected an instruction to change the relative coordinates of the pivot (S55: YES), it performs the processing in S56. The control unit 20 sets the image placement so that the reference point of the changed image is placed at the relative coordinates of the anchor. The control unit 20 updates the model projection plane UF shown in column G2 to reflect the change in the reference point (S56).

[0067] Figure 8(B) shows the simulation image K2 for image J1, where the anchor and pivot are changed from the initial conditions shown in Figure 8(A). The anchor is set at point Q8(-1,1) on the model projection plane UF, and the pivot is set at point P8(-1,1) on image J1.

[0068] If the control unit 20 has not detected an instruction to change the relative coordinates of the pivot (S55: NO), or if it has just performed the process in S56, it performs the process in S62. The control unit 20 determines whether or not an instruction to change the layer indicated in column G3 has been detected (S62). If multiple icons are shown in column G3, the user can input an instruction to change the order of the image layers indicated by the icons by selecting an icon and moving it up or down. If the control unit 20 detects an instruction to change the layer (S62: YES), it changes the setting of the selected image layer and updates the model projection plane UF indicated in column G2 to reflect the change in the layer (S63).

[0069] If the control unit 20 has not detected an instruction to change the layer (S62: NO), or if it has detected an instruction to store setting data after the processing in S63 (S64), it determines whether it has detected an instruction to store setting data. If the control unit 20 has detected an instruction to store setting data (S64: YES), it stores the setting data in the storage device 26 (S65). The setting data includes the original image data and the setting of the video size of the video indicated by the original image data. In this embodiment, the setting data includes, in addition to the original image data and the setting of the video size of the video indicated by the original image data, the projection type of the video, the arrangement of the video, and the layer information. If multiple original image data are added by the processing in S44, the control unit 20 sets the original image data as setting data for each original image data, associating the original image data with the video size of the video indicated by the original image data, the projection type of the video, the arrangement of the video, and the layer information.

[0070] If the control unit 20 has not detected an instruction to store setting data (S64: NO), or if it has performed the processing in S65, it performs the processing in S66. The control unit 20 determines whether or not it has detected an output instruction to output data related to the input image data (S66). The data related to the input image data is either the input image data or setting data. The data related to the input image data may be selected by the user or may be set in advance.

[0071] When the control unit 20 detects an output instruction (S66: YES), it outputs data relating to the input image data to the specified device (S67). The specified device may be, for example, the data output device 5, or a cloud server connected to the network 6.

[0072] If the control unit 20 has not detected an output instruction (S66: NO), or if it has just finished processing S67, it proceeds to process S68. The control unit 20 determines whether or not it has detected an end instruction to terminate the data output process (S68). If the user wishes to terminate the data output process, they input an end instruction to the data output device 2 by operating the input unit 25 to close the setting screen G1. If the control unit 20 has not detected an end instruction (S68: NO), it returns to processing S42. If the control unit 20 has detected an end instruction (S68: YES), it terminates the data output process.

[0073] The control unit 20 displays the simulation image K4 shown in Figure 9(A) in column G2 in the following case: that is, when the original image data of image J2 is added in the processing of S44 (S44:YES), and the video size and arrangement of image J2 are set as follows: In S51, the meter standard is selected as the method for setting the video size, and the length in the width direction is set to 0 meters and the length in the height direction is set to 1 meter. In S53, the position of the reference point of the video, i.e., the anchor, is set to point Q4(1,-1) on the model projection plane UF. In S55, the reference point of the video, i.e., the pivot, is set to point P4(1,-1) on image J2.

[0074] In the simulation image K4, the height of image J2 is set to 1 meter, regardless of the size of the model projection plane UF. The width of image J2 is set to the same magnification as the magnification used to set the height of image J2. In other words, when the width, which is different from the reference height, is set to 0, the control unit 20 sets the image size so that the ratio of the height and width of image J2 remains constant. Image J2 is placed on the first plane UF1.

[0075] In the following case, the control unit 20 displays the simulation image K5 shown in Figure 9(B) in column G2. That is, in the process of S44, the original image data of image J3 is added (S44:YES), and the video size and placement of image J3 are set as follows: In S51, the meter standard is selected as the method for setting the video size, and the length in the width direction is set to 0 meters and the length in the height direction is set to 1 meter. In S53, the anchor is set to point Q3(1,0) on the model projection plane UF. In S55, the pivot is set to point P7(-1,0) on image J2.

[0076] In simulation image K5, the height of image J3 is set to 1 meter, regardless of the size of the model projection plane UF. The width of image J2 is set to the same magnification as the magnification used to set the height of image J2. Image J3 is placed on the third plane UF3.

[0077] If, in S42, the size of the model projection plane UF is set to 1.5 times the size of the simulation image K5 shown in Figure 9(B) (S42: YES), the control unit 20 displays the simulation image K6 shown in Figure 9(C) in column G2 (S45). In simulation image K6, the height of image J3 is set to 1 meter, regardless of the size of the model projection plane UF. Let's compare simulation image K5 and simulation image K6. In simulation image K5, the size of image J3 is relatively larger than in simulation image K6. On the other hand, in simulation image K6, the height of image J1 is set to the same length as the height of the model projection plane UF, regardless of the size of the model projection plane UF.

[0078] Under the conditions that the simulation image K4 shown in Figure 9(A) is displayed, if the following process is executed, the control unit 20 displays the simulation image K7 shown in Figure 10(A) in column G2 (S63). In S42, the three-dimensional model U is changed. In S46, the original image data showing image J1 is deleted. In S44, the original image data showing image J4 is added. In S62, the layer of image J4 is changed so that image J4 is placed behind image J2. In the simulation image K7, the center of image J4 is positioned to coincide with the center of the first surface UF1 of the model projection plane UF.

[0079] If planar projection is not selected as the projection type (S49: NO), the control unit 20 determines whether equirectangular projection is selected as the projection type (S57). If equirectangular projection is not selected as the projection type (S57: NO), the control unit 20 performs the process in S58. The control unit 20 performs other processing according to the projection type (S58) and proceeds to S62.

[0080] If equirectangular projection is selected as the projection type for image J4 (S57: YES), the control unit 20 performs the process in S59. The control unit 20 displays the second setting field G12 shown in Figure 4(C) on the display 24 (S59). As shown in Figure 4(C), the second setting field G12 includes field G7 and field G13, which are the same as the first setting field G6. The second setting field G12 does not include fields G8 to G10. Field G7 is set to "equirectangular," which indicates equirectangular projection.

[0081] Column G13 indicates the display range of the image shown by the original image data. The display range of the image is specified by the elevation angle, azimuth angle, rotation, field of view (fovX) in the X direction, and field of view (fovY) in the Y direction relative to a reference. The references for the elevation angle, azimuth angle, and rotation can be set as appropriate. For example, the references for the elevation angle, azimuth angle, and rotation may be the position where the center of the first face F1 coincides with the center of the 360-degree image. The field of view in the X direction and the field of view in the Y direction are the ranges that can be viewed without rotation. The control unit 20 projects the simulation image K8 from Figure 10(B) onto column G2 of the setting screen G1. Simulation image K8 also places image J4 on the fourth face UF4 and the fifth face UF5. On the other hand, image J2 is placed only on the first face UF1 with an image size specified by an absolute value.

[0082] The control unit 20 determines whether or not it has detected an instruction to set the display range of the image in column G13 (S60). If the control unit 20 has detected an instruction to change the display range (S60: YES), it performs the process in S61. The control unit 20 sets the arrangement of the image indicated by the original image data based on the elevation angle, azimuth angle, rotation, field of view fovX in the X direction, and field of view fovY in the Y direction of column G13. The control unit 20 updates the model projection plane UF of column G3 in accordance with the change in the display range (S61). If the control unit 20 has not detected an instruction to set the display range (S60: NO), or after the process in S61, it performs the process in S62.

[0083] When the azimuth angle is set to 68° under the condition that the simulation image K8 in Figure 10(B) is projected (S60: YES), the control unit 20 performs the process in S61. The control unit 20 displays the simulation image K9 in Figure 10(C) in column G2 (S61). As shown in Figure 10(C), the azimuth angle of image J4 has been changed in simulation image K9 compared to simulation image K8. On the other hand, the arrangement of image J2 has not been changed.

[0084] As described above, the creator confirms that the simulation image displayed in column G2 is the intended image and outputs the setting data used for outputting the input image data projected in each of the multiple karaoke establishments (S66: YES, S67). Based on the setting data acquired from the data output device 2, the data output device 5 of the karaoke establishment generates input image data that matches the shape of the three-dimensional projection surface F onto which the input image data is projected, and projects it onto the projector 3. Users of the data output device 5 of the karaoke establishment can project the image onto the three-dimensional projection surface F in the arrangement intended by the creator simply by setting the three-dimensional projection surface F.

[0085] Referring to Figures 11 to 13, the projection control process performed by the data output device 5 will be described. The projection control process is started when the user of the data output device 5 inputs a start command. When the control unit 50 detects the start command, it reads the program for executing the projection control process stored in the storage device 56 into the memory 53. The control unit 50 executes the following steps according to the instructions contained in the program read into the memory 53. Various parameters necessary for executing the projection control process are stored in the storage device 56. Various data obtained during the projection control process are stored in the memory 53 as appropriate. The projection control process may also be performed by the data output device 2.

[0086] As shown in Figure 11, the control unit 50 retrieves the initial settings stored in the storage device 56 (S1). The initial settings include the settings of the current three-dimensional model U. The settings of the current three-dimensional model U are expressed using model parameters, including the size and shape of the three-dimensional projection plane F. The control unit 50 determines whether or not to change the three-dimensional model (S2).

[0087] In this embodiment, the three-dimensional model U, a cube as shown in Figure 2, is set as the initial value for the three-dimensional model. The data output device 5 in this embodiment can set the shape and length of each face of the current three-dimensional model U. The control unit 50 can make changes to the current three-dimensional model U, such as rotating or translating it. Typically, the user changes the three-dimensional model U when changing the position of the projector 3 relative to the three-dimensional projection plane F shown in Figure 12(A). If the user wants to change the settings of the current three-dimensional model U, they operate the input unit 55 to input an instruction to perform the process of changing the settings. In this embodiment, the actual height direction length of the first face F1, that is, the length from the floor to the ceiling of the room R, is obtained as the size reference for the image.

[0088] When the control unit 50 detects an instruction to change the settings of the three-dimensional model U (S2: YES), it performs the process in S21. As shown in Figure 12(B), the control unit 50 projects the reference image Q onto the three-dimensional projection surface F via the projection lens 4 of the projector 3 (S21). The reference image Q shows the settings of the three-dimensional model U. The reference image Q can be any image that shows the current settings of the three-dimensional model U, for example, an image that shows the boundary between two adjacent surfaces. The reference image Q can also be an image that includes a grid pattern representing the framework information B of the three-dimensional model U.

[0089] The control unit 50 receives a setting change instruction to change the settings of the three-dimensional model U after the projection of the reference image Q has started (S22). The user inputs the instruction by operating the input unit 55 to select the item to change the setting of and then specifying the amount of change. If the control unit 50 detects a setting change instruction for the three-dimensional model U (S22: YES), it performs the process in S23. The control unit 50 changes the settings of the three-dimensional model U according to the instruction received in S22, and changes the reference image Q to be projected onto the three-dimensional projection plane F and projects it (S23).

[0090] The user adjusts the parameters of the three-dimensional model U by operating the input unit 55 while viewing the reference image Q projected onto the three-dimensional projection plane F. As shown in Figure 12(C), the user operates the input unit 55 to position the reference image Q along the three-dimensional projection plane F. In Figure 12(C), the boundary of the pattern of the reference image Q coincides with the boundary of any two adjacent faces among the five faces that make up the three-dimensional projection plane F.

[0091] The control unit 50 can change the following items through the processing in S22 and S23. The control unit 50 can change the positional relationship between the room R in which the projector 3 is installed and the projector 3. The control unit 50 can change the lens characteristics of the projection lens 4 of the projector 3 acquired in S8 and the predetermined position of the virtual projector acquired in S6. The control unit 50 can change the orientation of the determined virtual projector PR determined in S5. The control unit 50 acquires the widthwise length of each face of the three-dimensional projection surface F based on the length from the floor to the ceiling of the room R and the settings of the three-dimensional model U.

[0092] If the control unit 50 does not detect an instruction to change the settings of the three-dimensional model U (S22: NO), or if it performs the process of S24 after S23, the control unit 40 determines whether it has detected an instruction to terminate the process of changing the settings of the three-dimensional model U (S24). If no instruction to terminate the process of changing the settings of the three-dimensional model U is detected (S24: NO), the control unit 50 returns to the process of S21. If an instruction to terminate the process of changing the settings of the three-dimensional model U is detected (S24: YES), the control unit 50 returns to the process of S2.

[0093] If the control unit 50 does not detect an instruction to change the settings of the three-dimensional model U (S2: NO), it performs the process in S3. The control unit 50 determines whether or not it has detected a projection instruction to project the image data onto the three-dimensional projection plane F (S3). If the control unit 50 does not detect a projection instruction (S3: NO), it returns to processing S2.

[0094] When the control unit 50 detects a projection instruction (S3: YES), it retrieves the setting data stored in the storage device 56 (S4). The setting data may be obtained from an external device such as a cloud server connected via the network 6 through the communication unit 57, or from the data output device 2. In S4, the control unit 50 retrieves the original image data, the video size setting, the video arrangement, and the projection type included in the setting data.

[0095] The control unit 50 acquires model parameters including the size and shape of a three-dimensional projection plane F that includes multiple planes, and sets up a three-dimensional model U that includes the three-dimensional projection plane F (S5). The control unit 50 acquires the placement of the determined virtual projector PR in the three-dimensional model U as a predetermined position (S6). The predetermined position may be set in advance, set by the user, or set by the process in S23.

[0096] As shown in Figure 2(B), the predetermined position in this embodiment is the position where the optical axis W of the lens L of the determination virtual projector PR is perpendicular to the first surface UF1 of the model projection plane UF. The predetermined position is the position where the optical axis W of the lens L of the determination virtual projector PR passes through the center M of the first surface UF1 of the model projection plane UF. The distance between the first surface UF1 and the determination virtual projector PR is set appropriately according to the shape of the three-dimensional projection plane F. The model projection plane UF is a plane in which the three-dimensional projection plane F is represented by the three-dimensional model U. The surfaces UF1 to UF5 of the model projection plane UF correspond to the surfaces F1 to F5 of the three-dimensional projection plane F, respectively.

[0097] The control unit 50 acquires the installation position of the projector 3 having the projection lens 4 with respect to the three-dimensional projection plane F (S7). The installation position of the projector 3 with respect to the three-dimensional projection plane F is acquired, for example, in three-dimensional coordinates in the world coordinate system. The control unit 50 acquires the lens characteristics of the projection lens 4 (S8). The projection lens 4 of the present invention is a fisheye lens. The lens characteristics of the projection lens 4 include parameters corresponding to the focal length of the projection lens 4, parameters indicating lens distortion, and parameters relating to the optical center.

[0098] The control unit 50 acquires the Eth original image data included in the setting data (S9). The initial value of variable E is 1. Variable E is a variable for acquiring the original image data in a predetermined order. The predetermined order can be set as appropriate, for example, in the order in which the layers are in the back. The control unit 50 determines the designated virtual projector PR from among a plurality of pre-set virtual projectors according to the type of original image data acquired in S9 (S10). The type of original image data is identified based on the projection type associated with the original image data. If the first and second image data included in the setting data are acquired in order in S10, the control unit 50 sets the first type, which is the projection type of the first image indicated by the first image data, by repeatedly executing S10. The control unit 50 sets the second type, which is the projection type of the second image indicated by the second image data.

[0099] The control unit 50 determines a designated virtual projector PR from among a plurality of pre-configured virtual projectors according to the projection type associated with the original image data. The shader program to be applied to the original image data is determined by the processing in S10. Therefore, in S82, which will be described later, the control unit 50 sends the setting of the designated virtual projector PR determined in S10 to the GPU.

[0100] The control unit 50 performs a conversion process (S11) to project the original image data included in the setting data acquired in S4 into two-dimensional image data for projection by the projector 3. If the original image data is video data, the conversion process is performed for each frame included in the video data.

[0101] As shown in Figure 13, the control unit 50 sets the video size indicated by the original image data and the arrangement on the model projection plane UF based on the setting data output by the data output device 2 (S81). When the control unit 50 obtains the first image data and the second image data as the original image data from the setting data, it performs the following processing: The control unit 50 sets the first video size, which is the size of the first video indicated by the first image data on the three-dimensional projection plane F. The control unit 50 sets the second video size, which is the size of the second video indicated by the second image data on the three-dimensional projection plane F. The control unit 50 also sets the first arrangement of the first video and the second arrangement of the second video on the model projection plane UF, similar to the video size.

[0102] The control unit 50 transmits various parameters to the GPU 22 for obtaining two-dimensional image data to be used for projection from the Eth original image data (S82). Based on the parameters transmitted in S42, the control unit 50 converts a predetermined simulation image into two-dimensional image data for a predetermined case (S83 to S85). The predetermined simulation image is an image obtained by virtually projecting the Eth original image data onto the three-dimensional projection surface F using the determined virtual projector PR determined in S10. The predetermined case B is when the image is virtually captured using the shooting lens N placed at the installation position acquired in S8.

[0103] Specifically, the control unit 50 virtually photographs the model projection surface UF using a photographic lens N positioned at a predetermined location. The predetermined location is a position corresponding to the installation position, set based on the installation position of the projector 3 relative to the three-dimensional projection surface F acquired in S8. The control unit 50 acquires two-dimensional coordinate data representing the model projection surface UF in two-dimensional coordinates (S83). Preferably, the virtual photographic lens N has the same or similar lens characteristics as the projection lens 4 of the projector 3.

[0104] The control unit 50 performs the processing in S83 using a vertex shader pre-prepared on the GPU 52 (S83). The vertex shader converts the information of the three-dimensional polygon representing the three-dimensional model U into a cuboid coordinate system. Normally, the control unit 50 generates a two-dimensional image from the simulated image of the three-dimensional model U using a central projection method, that is, a perspective projection method, in the vertex shader. The control unit 50 replaces the central projection method with an equidistant projection method that corresponds to the lens characteristics of the projection lens 4. As a result, the control unit 50 obtains two-dimensional coordinate data suitable for the projector 3 having the projection lens 4.

[0105] In conventional perspective projection imaging, the coordinates of a two-dimensional image are calculated using a perspective transformation matrix, with one matrix multiplication for each vertex of the three-dimensional model U. On the other hand, in vertex shaders using equidistant projection, the coordinates of a two-dimensional image cannot be determined by simple matrix operations. The control unit 50 of this embodiment performs the processing in S83 using a three-dimensional programming API, such as a shader program. In S83, the control unit 50 may remove unnecessary information by performing processes such as back-facing culling, clipping, attribute evaluation, and rasterization. Through the processing in S83, the control unit 50 determines the coordinates of each vertex on the three-dimensional projection plane F of the two-dimensional coordinate data for the simulated image of the three-dimensional model U, based on the vertex data.

[0106] The control unit 50 performs rasterization on the vertex data to which the processing in S83 has been applied (S84). In the rasterization process, the control unit 50 combines each vertex data to form polygons that represent the deformation model data of the three-dimensional model U. The control unit 50 determines the arrangement of pixels for drawing each polygon on the two-dimensional image data.

[0107] The control unit 50 generates two-dimensional image data (S85) based on the simulation image for a predetermined case and the two-dimensional coordinate data acquired in S83. The predetermined case is when the image data is virtually projected onto the model projection surface UF using a determined virtual projector PR positioned at a predetermined location relative to the model projection surface UF. The two-dimensional image data is an image signal output to the projector 3. Based on the color information of the image data and the correspondence between the image data and the two-dimensional coordinate data, the control unit 50 generates two-dimensional image data with the color information set in the two-dimensional coordinate data.

[0108] The control unit 50 applies pixel shader processing to each pixel to which the processing in S84 has been applied. In the pixel shader processing, the correspondence between image data and two-dimensional coordinate data is identified using the determined virtual projector PR determined in S10. If the determined virtual projector PR is a standard virtual projector, the control unit 50 identifies the correspondence between image data and two-dimensional coordinate data based on the framework information B of the three-dimensional model U.

[0109] If the determined virtual projector PR is a 360-degree virtual projector, the control unit 50 performs a calculation to map the projection position on the three-dimensional model U in a predetermined case. The predetermined case is when image data is projected radially from a virtual projector positioned in a predetermined orientation at a predetermined position acquired in S6. The predetermined orientation in this embodiment is the orientation in which the optical axis W is orthogonal to the first surface UF1 of the model projection plane UF. Based on this, the control unit 50 identifies the correspondence between the image data and the two-dimensional coordinate data of the image data captured using the imaging lens N.

[0110] In a 360-degree virtual projector, the control unit 50 calculates the elevation angle and azimuth angle for each point on the three-dimensional model U, with the virtual projector PR's predetermined position and orientation as the origin. From the elevation angle and azimuth angle, the control unit 50 can determine the texture coordinates of the image data captured by the 360-degree camera. This allows the control unit 50 to identify the correspondence between the original image data and the two-dimensional coordinate data of the omnidirectional image.

[0111] The control unit 50 assigns the color and transparency of the two-dimensional coordinate data to each pixel based on the color information of the image data and the correspondence between the image data and the two-dimensional coordinate data. This allows the control unit 50 to generate two-dimensional image data. Specifically, the control unit 50 assigns the color and transparency of a predetermined position to the pixel corresponding to the vertex of a polygon. The predetermined position is indicated by the texture coordinates included in the vertex data of that vertex in the texture image.

[0112] The control unit 50 determines the color and transparency of pixels that do not correspond to polygon vertices by interpolating the color and transparency assigned to the polygon vertices. In addition to determining the color and transparency of pixels using a pixel shader, the control unit 50 may also apply processes such as alpha testing, depth testing, stencil testing, and blending. After S85, the control unit 50 terminates the conversion process and returns the process to projection control processing.

[0113] Following the processing in S11, the control unit 50 determines whether the source image data acquired in S9 is the last source image data acquired (S12). The control unit 50 determines that the source image data acquired in S9 is the last source image data acquired if there is no next source image data to acquire. If there is a next source image data to acquire (S12: NO), the control unit 50 increments the variable E by 1 and returns to processing S9. If the source image data acquired in S9 is the last source image data acquired (S12: YES), the control unit 50 performs the processing in S13. The control unit 50 synthesizes the two-dimensional image data so that the images shown by the two-dimensional image data generated in S85 are superimposed according to the layers to generate the input image data (S13).

[0114] The control unit 50 outputs input image data to the projector 3, causing the projector 3 to project the image indicated by the input image data onto the three-dimensional projection surface F (S14). The control unit 50 uses the projection lens 4, positioned according to the installation position of the projector 3, to output input image data that projects an image of a set size onto the model projection surface UF, which represents the three-dimensional projection surface F as a three-dimensional model U, onto the projector 3. The control unit 50 outputs input image data according to the projection type. The control unit 50 outputs input image data that projects an image of a set size onto the projector 3, which is positioned at a predetermined location on a predetermined surface of the model projection surface UF.

[0115] As shown in the simulation image K1 in Figure 8(A), the control unit 50 outputs the following input image data for image J1, for which the image size has been set by the first method. That is, the control unit 50 outputs input image data that projects onto the projector 3 an image with an image size such that the vertical length of the image projected onto the model projection surface UF is a predetermined ratio of the vertical length of the three-dimensional model U.

[0116] As shown in image J2 of simulation image K4 in Figure 9(A), when the control unit 50 sets the image size using the second method, it outputs the following input image data. Specifically, the control unit 50 outputs input image data that projects an image onto the projector 3 in which the reference point of the image size is placed at a predetermined position indicated by relative coordinates set on the model projection plane. More specifically, when the image size is set using the second method, the control unit 50 outputs input image data that projects an image onto the projector 3 in which an image size is placed at a predetermined position indicated by relative coordinates of the model projection plane UF. When the setting method is set using the second method, the control unit 50 outputs input image data that projects an image onto the projector 3 in which an image size is set by an absolute value obtained from column G9, placed at a predetermined position in the relative coordinates of the model projection plane UF.

[0117] In S4, if the setting data for the simulation image K4 shown in Figure 9(A) is acquired, the control unit 50 outputs the following input image data to the projector 3. That is, the control unit 50 outputs input image data to the projector 3 that superimposes an image on the projector 3, which is an image J1 that is the first image of the first image size and an image J2 that is the second image of the second image size.

[0118] In S4, if the setting data for the simulation image K9 shown in Figure 10(C) is acquired, the control unit 50 outputs the following input image data to the projector 3. That is, the control unit 50 outputs input image data or setting data to project an image on the projector 3 that is a superimposed image of image J4 and image J2. Image J4 is an example of the first image projected by equirectangular projection, which is the first type. Image J2 is an example of the second image projected by planar projection, which is the second type and is different from the first type. When the projection type is equirectangular projection, as in image J4, the control unit 50 outputs input image data to project an image onto the projector 3 that is positioned at a location specified by the azimuth angle and elevation angle relative to the reference. The projector 3 projects the input image data onto the three-dimensional projection surface F according to the image signal output from the data output device 5.

[0119] The control unit 50 determines whether to terminate the projection control process (S15). The control unit 50 determines to terminate the projection control process when it detects an instruction to terminate the projection control process input from the user via the input unit 55.

[0120] If no instruction to terminate the projection control process is detected (S15: NO), the control unit 50 returns to processing S15. If an instruction to terminate the projection control process is detected (S15: YES), the control unit 50 stores the current settings in the storage device 56 and terminates the process. In the above projection control process, the user of the data output device 5 can project the input image data onto the projector 3 without having to configure the original image data, based on the setting data output from the data output device 2.

[0121] In the above embodiment, data output devices 2 and 5 are examples of data output devices of the present invention. Projector 3 is an example of a projector of the present invention. Projection lens 4 is an example of a projection lens of the present invention. Three-dimensional projection surface F is an example of a three-dimensional projection surface of the present invention. Origin Q0 is an example of an origin of the present invention. Three-dimensional model U is an example of a three-dimensional model of the present invention. Model projection surface UF is an example of a model projection surface of the present invention. Processing S44 and S4 is an example of image acquisition processing of the present invention. Processing S42 and S2 is an example of model setting processing of the present invention. Processing S52 and S81 is an example of video size setting processing of the present invention. Processing S64, S66, and S14 is an example of output processing of the present invention. Processing S51 and S4 is an example of coordinate acquisition processing of the present invention. Processing S49, S55, and S10 is an example of projection type setting processing of the present invention.

[0122] The data output program in the above embodiment is a program to be executed by data output devices 2 and 5. Data output devices 2 and 5 output data relating to input image data for projecting an image onto projector 3. The data output program includes instructions to data output devices 2 and 5 to execute image acquisition processing, model setting processing, image size setting processing, and output processing. In the image acquisition processing, data output devices 2 and 5 acquire the original image data that will be the source of the input image data (S44, S4). In the model setting processing, data output devices 2 and 5 acquire model parameters and set the three-dimensional model U. The model parameters include the size and shape of the three-dimensional projection surface F which includes multiple planes. In the image size setting processing, data output devices 2 and 5 set the image size (S52, S81). The image size is the size of the image shown by the original image data on the three-dimensional projection surface F. In the image size setting processing, if the image size is set by the first method, data output devices 2 and 5 set the image size to a predetermined ratio of the size of the three-dimensional projection surface F. In the video size setting process, data output devices 2 and 5 set the video size to a size specified by an absolute value, independent of the size of the three-dimensional projection surface F, when the video size is set by the second method. In the output process, data output devices 2 and 5 output input image data or setting data using projection lenses 4 positioned according to the installation position of the projector 3 (S64, S66, S14). The input image data is data that projects an image of the set video size onto the model projection surface UF, which represents the three-dimensional projection surface F with a three-dimensional model U, onto the projector 3. The setting data is data used for outputting the input image data.

[0123] The video size setting process of the data output program sets the video size using different methods depending on whether the video size is set using the first method or the second method. Therefore, the video size setting process of the data output program contributes to setting the placement of the video on the three-dimensional projection plane F compared to setting the video size using a single method.

[0124] The multiple planes of the three-dimensional projection plane F include a first plane F1 that extends in the vertical direction. The image size setting process, when setting the image size using the first method, includes setting the image size such that the vertical length of the image projected onto the model projection plane UF is a predetermined ratio of the vertical length of the three-dimensional model U. The image size setting process of the data output program contributes to setting images with a vertical length set based on the vertical length of the model projection plane UF more easily than before.

[0125] The output processing includes the process of outputting input image data or setting data that causes an image of a certain size, positioned at a predetermined location on a predetermined surface of the model projection surface UF, to be projected onto the projector 3. The output processing of the data output program contributes to projecting an image that reduces the effect of the positional relationship of the image to the model projection surface UF, depending on the shape and size of the model projection surface UF.

[0126] The output processing includes the process of outputting input image data or setting data when the image size is set by the second method. The input image data is data that projects an image of a predetermined size, placed at a predetermined position, onto the projector 3. The predetermined position is a position set on the model projection plane UF, indicated by relative coordinates with a predetermined point as the origin Q0. The output processing of the data output program contributes to projecting an image that reduces the effect of the positional relationship of the image to the three-dimensional model U, depending on the shape and size of the model projection plane UF.

[0127] The output processing includes outputting input image data or setting data to project an image onto projector 3, in which the reference point of the image size is placed at a predetermined position, when the image size is set by the second method. The predetermined position is indicated by relative coordinates set on the model projection plane UF. The output processing of the data output program contributes to projecting an image that reduces the effect of the positional relationship of the image to the three-dimensional model U, depending on the shape and size of the model projection plane UF.

[0128] The multiple planes of the three-dimensional projection plane F include a first plane F1 that extends in the vertical and horizontal directions. The data output program includes instructions to the data output devices 2 and 5 to execute a coordinate acquisition process to acquire relative coordinates (S51, S4). The relative coordinates are indicated by XY coordinates, where the horizontal direction is the X-axis and the vertical direction is the Y-axis. The output process includes outputting input image data or setting data to project an image of the image size placed at a predetermined position onto the projector 3 when the image size is set by the second method. The predetermined position is indicated by the relative coordinates acquired in the coordinate acquisition process of the model projection plane UF. The coordinate acquisition process of the data output program contributes to simplifying the positional relationship of the image to the three-dimensional model U according to the shape and size of the model projection plane UF compared to conventional methods.

[0129] The video size setting process, when setting the video size using the second method, includes obtaining the absolute value of the video size and setting the video size to the obtained absolute value. The video size setting process in the data output program contributes to making video size adjustment easier than before.

[0130] The video size setting process includes setting the size of the first video when the first and second image data to be processed are acquired during the image acquisition process. The first video size is the size of the first video on the three-dimensional projection plane F indicated by the first image data. The video size setting process also includes setting the size of the second video, which is the size of the second video on the three-dimensional projection plane F indicated by the second image data. The output process includes outputting input image data or setting data to project an image onto the model projection plane UF, which is the superimposed image of the first video (size of the first video) and the second video (size of the second video) onto the projector 3. The video size setting process in the data output program contributes to simplifying the setting of the superimposed image of the first and second videos on the model projection plane UF compared to conventional methods.

[0131] The configuration data includes the original image data and video size settings. The output process includes outputting the configuration data to a storage device. The output process of the data output program contributes to facilitating the process of reconfiguring settings or outputting input image data with other data output devices.

[0132] The configuration data includes the original image data and the video size settings. The image acquisition process includes the process of acquiring the original image data from the configuration data output by other data output devices 2 and 5. The video size setting process includes the process of setting the video size based on the configuration data output by other data output devices 2 and 5. The output process includes the process of outputting the input image data to the projector 3 and projecting the video onto the projector 3. The video size setting process of the data output program contributes to projecting an image that reduces the effect of the shift in the positional relationship of the image with respect to the model projection surface UF, even when the shape of the model projection surface UF changes from the model projection surface UF set by other data output devices.

[0133] The data output program includes instructions for data output devices 2 and 5 to execute projection type setting processing. Projection type setting processing is the process of setting the projection type by projector 3 from a plurality of options, including equirectangular projection and planar projection (S49, S55, S10). Image size setting processing is executed when the projection type is planar projection. Output processing includes the process of outputting input image data according to the projection type. When projector 3 projects a continuous image onto a three-dimensional projection surface F, there is a need to project a three-dimensional image viewed from a certain point onto the three-dimensional projection surface F. When projector 3 projects a continuous image onto a three-dimensional projection surface F, there is also a need to project a two-dimensional image onto a plane. Existing technologies address only one of these needs. In contrast, the data output program contributes to outputting input image data of the projection type set from a plurality of options, including equirectangular projection and planar projection.

[0134] The output processing includes the output of input image data or setting data to project a predetermined image onto the projector 3 when the projection type is equirectangular projection. The predetermined image is an image positioned at a location specified by the azimuth angle and elevation angle from a predetermined position. The output processing of the data output program contributes to specifying the placement of the image by the azimuth angle and elevation angle from a predetermined position when the projection type is equirectangular projection.

[0135] The projection type setting process includes setting the first type, which is the projection type of the first image indicated by the first image data, when the first and second image data are acquired in the image acquisition process. The projection type setting process also includes setting the second type, which is the projection type of the second image indicated by the second image data. The output process includes outputting input image data or setting data to project onto the model projection surface UF onto projector 3 an image in which the first image projected with the first type and the second image of the second type are superimposed. The data output program contributes to outputting input image data to project onto projector 3 an image in which the image projected with equirectangular projection and the image projected with planar projection are superimposed.

[0136] The projection lens 4 is a fisheye lens. The input image data is data that projects an image onto the model projection surface UF using a single projector 3. The data output program contributes to outputting the input image data that projects an image onto the three-dimensional projection surface F using a single projector 3.

[0137] The printer and data output program of the present invention are not limited to the embodiments described above, and various modifications may be made without departing from the spirit of the invention. For example, the following modifications may be made as appropriate. The present invention can be implemented in various forms. The present invention may be implemented, for example, in the form of a control method for a data output device, a non-temporary computer-readable medium storing a data output program, and a projection system.

[0138] The data output program, which includes instructions for executing data output processing, only needs to be stored in the storage device 26 or memory 23 by the time the control unit 20 of the data output device 2 executes the data output program. The projection control program, which includes instructions for executing projection control processing, only needs to be stored in the storage device 56 or memory 53 by the time the control unit 50 of the data output device 5 executes the projection control program. Therefore, the method of acquiring the data output program and the projection control program, the acquisition path, and the equipment for storing the programs may be changed as appropriate. The data output program and the projection control program may be combined into a single program.

[0139] The data output program may be received from other devices via cable or wireless communication and stored in storage device 26 or memory 23. The projection control program may be received from other devices via cable or wireless communication and stored in storage device 56 or memory 53. Other devices include, for example, a PC and a server connected via a network.

[0140] Each step of the data output processing is not limited to being performed by the control unit 20, but may be partially or entirely performed by other electronic devices, such as an ASIC. Each step of the projection control processing is not limited to being performed by the control unit 50, but may be partially or entirely performed by other electronic devices. Each step of the data output processing and projection control processing may be distributed among multiple electronic devices, such as multiple CPUs. Each step of the data output processing and projection control processing can be reordered, omitted, or added as needed. The following modifications may be made as appropriate to the processing performed in the projection system 1, including the data output processing and projection control processing.

[0141] The shape of the three-dimensional projection plane F may include multiple planes or curved surfaces. The process for changing the three-dimensional model U may be modified as appropriate. If the data output device 2 is equipped with a sensor that measures the distance and shape to the three-dimensional projection plane F, the data output device 2 may automatically set the three-dimensional model U according to the sensor's detection results.

[0142] The projection type may be changed as appropriate. Several types of virtual projectors may be changed as appropriate depending on the projection type. Multiple types of virtual projectors may include a fisheye virtual projector. Multiple types of virtual projectors do not necessarily have to include a 360-degree virtual projector. The original image data does not need to have shooting conditions attached, and the control unit 20 may set the virtual projector selected by the user as the determined virtual projector PR.

[0143] The predetermined position of the determined virtual projector PR does not have to be a position where the optical axis W of the lens L of the determined virtual projector PR is perpendicular to the first surface UF1 of the model projection plane UF. The predetermined position does not have to be a position where the optical axis W of the lens L of the determined virtual projector PR passes through the center M of the first surface UF1 of the model projection plane UF. In the two-dimensional image data generation process of S85, two-dimensional image data should be generated based on the simulation image and the two-dimensional coordinate data acquired in S83. The method of generating the two-dimensional image data may be changed as appropriate.

[0144] The video size setting process allows the video size to be set based on either width or height when the video size is set using the first method. The predetermined ratio may be a fixed value or a variable value. The arrangement of the video may or may not be specified by the user. The absolute scale in the width direction being 0 meters may be obtained as an instruction to set the absolute scale to 0 meters. The setting screen may have an input field for an instruction to set the length in the width direction with the same magnification as the magnification in the height direction, separate from the input fields for the absolute scale in the height direction or width direction.

[0145] In output processing, when setting the image size using the second method, the placement of the image relative to the model projection plane UF may be set to a predetermined position on the model projection plane UF, indicated by absolute coordinates with the origin Q0 set at a predetermined position. The method for setting the relative coordinates on the model projection plane UF may be changed as appropriate. The origin of the relative coordinates may be a point other than point Q0. The relative coordinates set on the model projection plane UF may be indicated by XY coordinates, with the horizontal direction being the Y axis and the vertical direction being the X axis. The relative coordinates set on the model projection plane UF may also be indicated by XYZ coordinates, with the horizontal direction being the X and Y axes and the vertical direction being the Z axis.

[0146] The method for setting relative coordinates and reference points in the video may be changed as appropriate. The control units 20 and 50 may use any point in the video specified by the user as the reference point.

[0147] The method for setting the absolute value of the video size when setting the video size using the second method may be changed as appropriate. The control units 20 and 50 may set the absolute value based on the data attached to the original image data.

[0148] The control units 20 and 50 may be unable to acquire multiple source image data during the image acquisition process. The setting data only needs to include the source image data and the video size settings, and does not need to include data indicating the arrangement of the video or data indicating the projection type. The storage device to which the setting data is output may be a separate device from the data output devices 2 and 5, or it may be a device provided by the data output devices 2 and 5.

[0149] Data output devices 2 and 5 may or may not be able to acquire setting data from other devices. Data output devices 2 and 5 may not be able to set the video size based on setting data output by other data output devices 2 and 5. Data output devices 2 and 5 may not be able to perform projection type setting processing. Data output devices 2 and 5 may project equirectangular images as planar projections or equirectangular projections. Data output devices 2 and 5 may accept the placement of equirectangular images in relative or absolute coordinates. Data output devices 2 and 5 may output input image data that projects an image placed at a specific position onto projector 3.

[0150] The data output devices 2 and 5 may use the same projection type for the first and second image data when the first and second image data are acquired during the image acquisition process. The projection lens 4 does not have to be a fisheye lens. In this case, the input image data may be, for example, data used to project an image onto the model projection surface UF using multiple projectors 3.

[0151] The above modifications may be combined as appropriate, provided they are not contradictory. In addition to the combinations exemplified in the claims, the applicant intends to obtain patent rights for embodiments that combine each other in a manner that does not depart from the spirit of the present invention and is not contradictory. [Explanation of symbols]

[0152] 2, 5: Data output device, 3: Projector, 4: Projection lens, 20, 50: Control unit, 27, 57: Communication unit, 25, 55: Input unit, 26, 56: Storage device, 21, 51: CPU, 22, 52: GPU, 23, 53: Memory

Claims

1. A data output program to be executed by a data output device that outputs data related to input image data for projecting an image onto a projector, The aforementioned data output device, Image acquisition process to obtain the original image data that will serve as the basis for the input image data, A model setting process that obtains model parameters including the size and shape of a three-dimensional projection plane containing multiple planes, and sets up a three-dimensional model. A video size setting process that sets the video size, which is the size of the video shown in the original image data on the three-dimensional projection plane, When setting the image size using the first method, the image size is set to a predetermined ratio of the size of the three-dimensional projection surface. When setting the image size using the second method, the image size is set to a size specified by an absolute value, independent of the size of the three-dimensional projection surface, and the image size setting process is performed. Output processing that outputs input image data, or setting data used for outputting the input image data, which is used to project the image of the set image size onto the model projection surface, which is represented by the three-dimensional model, using a projection lens positioned according to the installation position of the projector, on the three-dimensional projection surface. A data output program characterized by including instructions to execute.

2. The aforementioned plurality of planes include a plane extending in the vertical direction, The data output program according to claim 1, characterized in that the video size setting process includes, when setting the video size by the first method, a process to set the video size such that the vertical length of the video projected onto the model projection surface is a predetermined ratio of the vertical length of the three-dimensional model.

3. The data output program according to claim 1, characterized in that the output process includes a process of outputting the input image data or setting data that causes the image of the image size, which is positioned at a predetermined position on a predetermined surface of the model projection surface, to be projected onto the projector.

4. The data output program according to claim 1, characterized in that when the video size is set by the second method, the output process includes outputting the input image data or setting data which causes the video of the video size, positioned at a predetermined position indicated by relative coordinates with a predetermined point as the origin, set on the model projection surface, to be projected onto the projector.

5. The data output program according to claim 1, characterized in that the output processing includes, when the image size is set by the second method, a process to output input image data or setting data which projects the image of the image size onto the projector, in which the reference point of the image of the image size is placed at a predetermined position indicated by relative coordinates set on the model projection surface.

6. The aforementioned plurality of planes include planes extending in the vertical and horizontal directions, The aforementioned data output device, The instruction includes further instructions to execute a coordinate acquisition process to acquire the relative coordinates, which are represented by an XY coordinate system where the horizontal direction is the X-axis and the vertical direction is the Y-axis. The data output program according to claim 5, characterized in that when the video size is set by the second method, the output process includes a process to output input image data or setting data that projects the video of the video size, which is positioned at the predetermined position indicated by the relative coordinates obtained in the coordinate acquisition process of the model projection surface, onto the projector.

7. The data output program according to claim 5, characterized in that the video size setting process includes, when the video size is set by the second method, a process of obtaining the absolute value of the model projection plane video size and setting the size of the video to the obtained absolute value.

8. The aforementioned video size setting process includes, when the first image data and the second image data to be processed are acquired in the image acquisition process, a process to set the first video size, which is the size of the first video shown by the first image data on the three-dimensional projection plane, and a process to set the second video size, which is the size of the second video shown by the second image data on the three-dimensional projection plane. The data output program according to claim 1, characterized in that the output processing includes a process of outputting the input image data or the setting data that superimposes the first image of the first image size and the second image of the second image size onto the model projection surface and projects this image onto the projector.

9. The aforementioned setting data includes the original image data and the video size settings, The aforementioned output processing is performed as follows: The data output program according to claim 8, characterized in that it includes a process for outputting the aforementioned setting data to a storage device.

10. The aforementioned setting data includes the original image data and the video size settings, The image acquisition process includes a process of acquiring the original image data from the setting data output by another data output device. The video size setting process includes a process of setting the video size based on the setting data output by the other data output device. The aforementioned output processing is performed as follows: The data output program according to claim 1, characterized in that it includes a process of outputting the input image data to the projector and projecting the image onto the projector.

11. The instructions include further instructions for performing a projection type setting process to select the projection type from a plurality of options, including equirectangular projection and planar projection, by the aforementioned projector. The aforementioned image size setting process is performed when the projection type is planar projection. The data output program according to claim 1, characterized in that the output processing includes processing to output the input image data corresponding to the projection type.

12. The data output program according to claim 11, characterized in that the output processing includes, when the projection type is equirectangular projection, the process of outputting the input image data or setting data that causes the image, positioned at a location specified by the azimuth angle and elevation angle with respect to a reference, to be projected onto the projector.

13. The projection type setting process includes, when the first image data and the second image data are acquired in the image acquisition process, a process to set the first type which is the projection type of the first video shown by the first image data, and a process to set the second type which is the projection type of the second video shown by the second image data. The data output program according to claim 11, characterized in that the output processing includes a process of outputting the input image data or the setting data that superimposes the first image projected in the first type and the second image of the second type onto the model projection surface and projects this image onto the projector.

14. The aforementioned projection lens is a fisheye lens, The data output program according to claim 1, characterized in that the input image data is data that causes one projector to project the image onto the model projection surface.

Images