Projection control device, projection system, projection method and program

Abstract

To provide a projection control apparatus configured to allow a user to view a corrected projection image immediately even when a position of the user viewing the projection image moves, a projection system, a projection method, and a program.SOLUTION: A projection control apparatus 30 includes a control unit 31 which is at least one processor, and a communication unit 36. The control unit 31 acquires a first captured image (correction image) captured at a viewpoint 1 and a second captured image (correction image) captured at a viewpoint 2 through the communication unit 36, acquires first correction data based on the first captured image, acquires second correction data based on the second captured image, acquires first position data of the viewpoint 1, second position data of the viewpoint 2, and third position data of a prediction point, determines third correction data corresponding to the prediction point by using the first correction data, the second correction data, the first position data, the second position data, and the third position data, and corrects a projection image to be projected by a projection device 10 (projection unit 11) on the basis of the determined third correction data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a projection control device, a projection system, a projection method, and a program. [Background technology]

[0002] Conventionally, projection devices have been disclosed that have a correction function for performing correction such as deformation on an input image in order to project the image into a neat rectangle as viewed by a user. For example, the projection device disclosed in Patent Document 1 can photograph a projected adjustment image, calculate a correction coefficient from the photographed image (captured image), and perform keystone correction on the input image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-186080 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned conventional projection device, if the position of the user viewing the projected image relative to the installation position of the projection device is fixed, the user can continuously view a clear rectangular image. However, if the user moves the position from which the projected image is viewed, a process such as recalculation of the correction coefficient is required, which is cumbersome.

[0005] The present invention aims to provide a projection control device, a projection system, a projection method, and a program that enable a user to instantly view a corrected projection image even if the user moves the position from which the projection image is viewed. [Means for solving the problem]

[0006] The projection control device of the present invention includes at least one processor and a communication unit, and the at least one processor The measurement image projected by the projection unit The first image taken at the first location and The measurement image projected by the projection unit A second captured image captured at a second location different from the first location is acquired via the communication unit, and the first captured image is determining a range of a rectangle that can be projected from the measurement image having an aspect ratio corresponding to a display element of the projection unit; The first correction data is obtained, and the second captured image is determining a range of a projectable rectangle having an aspect ratio corresponding to the display element from within the measurement image in acquiring second correction data, respectively acquiring first position data of the first point, second position data of the second point, and third position data of a third point different from the first point and the second point, determining third correction data corresponding to the third point using the first correction data, the second correction data, the first position data, the second position data, and the third position data, and based on the determined third correction data The above The projection image projected on the projection unit is corrected.

[0007] The projection system of the present invention includes the above-mentioned projection control device and a plurality of micromirrors. The above A display element is provided The above and a terminal device configured to be capable of capturing the first captured image and the second captured image, and capable of transmitting the first captured image, the second captured image, and the first position data, the second position data, and the third position data to the projection device via the communication unit.

[0008] The projection method of the present invention comprises the steps of: The measurement image projected by the projection unit The first image taken at the first location and The measurement image projected by the projection unit acquiring a second captured image captured at a second location different from the first location via the communication unit; determining a range of a rectangle that can be projected from the measurement image having an aspect ratio corresponding to a display element of the projection unit; The first correction data is obtained, and the second captured image is determining a range of a projectable rectangle having an aspect ratio corresponding to the display element from within the measurement image in a step of acquiring second correction data, a step of acquiring first position data of the first point, second position data of the second point, and third position data of a third point different from the first point and the second point, respectively, a step of determining third correction data corresponding to the third point using the first correction data, the second correction data, the first position data, the second position data, and the third position data, and a step of determining third correction data corresponding to the third point based on the determined third correction data. The aboveand correcting the projection image to be projected by the projection unit.

[0009] The program of the present invention is a program executed by a computer, the computer comprising: The measurement image projected by the projection unit The first image taken at the first location and The measurement image projected by the projection unit A second captured image captured at a second location different from the first location is acquired via a communication unit, and the first captured image is determining a range of a rectangle that can be projected from the measurement image having an aspect ratio corresponding to a display element of the projection unit; The first correction data is obtained, and the second captured image is determining a range of a projectable rectangle having an aspect ratio corresponding to the display element from within the measurement image in acquiring second correction data, respectively acquiring first position data of the first point, second position data of the second point, and third position data of a third point different from the first point and the second point, determining third correction data corresponding to the third point using the first correction data, the second correction data, the first position data, the second position data, and the third position data, and based on the determined third correction data The above It functions to correct the image projected onto the projection unit. Effect of the Invention

[0010] According to the present invention, it is possible to provide a projection control device, a projection system, a projection method, and a program that allow a user to instantly view a corrected projection image even if the user moves the position from which the projection image is viewed. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a schematic diagram showing a state in which a projection image is projected onto a curtain by a projection system according to an embodiment of the present invention. [Diagram 2] 1 is a control block diagram showing a projection control device and a projection unit according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a control block diagram of a mobile terminal according to an embodiment of the present invention. [Figure 4] 6 is a flowchart showing how correction information is calculated by an image correction processing unit according to an embodiment of the present invention. [Diagram 5]5A and 5B are schematic diagrams showing a correspondence relationship between a captured image and a display element according to the embodiment of the present invention. [Figure 6] 5A to 5C are schematic diagrams showing a correspondence relationship between a projected image, a deformed projected image, and a display element according to an embodiment of the present invention. [Figure 7] 1 is a schematic diagram showing the positional relationship between viewpoints 1 and 2 and prediction points 1, 2, and 3 according to an embodiment of the present invention. [Figure 8] 11 is a flow diagram illustrating a calculation of correction information at a prediction point according to an embodiment of the present invention. FIG. [Figure 9] FIG. 4 is a diagram showing a UI screen displayed on a display unit of a mobile terminal according to an embodiment of the present invention. [Figure 10] FIG. 4 is a diagram showing a UI screen displayed on a display unit of a mobile terminal according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described. Fig. 1 shows a state where a projection system 1 including a projection device 10 equipped with a projection control device 30 and a mobile terminal 20 projects a projection image onto a part of a curtain 2 as a projection surface 3 in a room. According to the projection system 1 of this embodiment, when projected onto the projection surface 3 (curtain 2) that is irregularly curved like a wave, a distorted projection image (also called a projection image, including a still image and a video) P1 is generated as shown by a two-dot chain line in Fig. 1, but this can be corrected to a rectangular projection image P2 as shown by a solid line. Even if a user U moves the position where he or she views the projection image P2, the projection image is immediately corrected, and the rectangular projection image P2 can be viewed.

[0013] 2 is a control block diagram including a projection control device 30 provided in the projection device 10. The projection control device 30 includes a CPU (also called a processor, or a computer) including a control unit 31, an image correction processing unit 34, and a measurement processing unit 35, a front-end unit including an image input unit 33, and a formatter unit including a projection image processing unit 37. Image signals of various standards input from the image input unit 33 are converted by an image conversion unit of the control unit 31 via a system bus SB so as to be unified into image signals of a predetermined format suitable for display, and then processed by the image correction processing unit 34 and the measurement processing unit 35, which will be described later, and output to the projection image processing unit 37.

[0014] The projection image processing unit 37 drives the display element 50 in response to the output image processed by the image correction processing unit 34 and the measurement processing unit 35, and controls the light source device 60 so that light in a predetermined wavelength band required for image generation is emitted from the light source device 60. The projection optical system 15 can emit the image light generated by the display element 50 as projection light. The projection optical system 15 includes a fixed lens and a movable lens, and can perform focus adjustment by a motor 16. The projection unit 11 formed to be able to project the image light includes the projection image processing unit 37, the display element 50, the light source device 60, and the projection optical system 15. The display element 50 can be a DMD (Digital Micromirror Device) including a plurality of micromirrors.

[0015] The operation unit 32 includes key switches and indicators that enable various operations and settings of the projection device 10 (projection control device 30). An operation signal from the operation unit 32 is transmitted to the control unit 31.

[0016] The control unit 31 is connected to an audio processing unit 38 via a system bus SB. The audio processing unit 38 includes a sound source circuit such as a PCM sound source, and converts audio data into analog form in the projection mode and the playback mode, and drives a speaker 38a to emit amplified sound.

[0017] The projection control device 30 also has a storage unit 40. The storage unit 40 is connected to the control unit 31, the image correction processing unit 34, and the projection image processing unit 37 via a system bus SB. The storage unit 40 is configured, for example, with an SSD (Solid State Drive) or an SRAM (Static Random Access Memory). The storage unit 40 stores a measurement image 41 projected during correction, a control program 42 (program) that causes the control unit 31, the image correction processing unit 34, and the measurement processing unit 35 to function, and measurement information 43, which is information on an image (correction image) captured by a user U at an arbitrary position of the projected measurement image 41 and a position, etc., and correction information 44, which is information for image correction corresponding to the position where the user U captured the measurement image 41, as will be described in detail later.

[0018] The communication unit 36 ​​is, for example, a network interface controller (NIC), and is configured to be able to transmit and receive communication data to and from the communication unit 22 of the mobile terminal 20 via a wireless communication path. The communication unit 36 ​​is connected to the control unit 31 and the storage unit 40 via a system bus SB.

[0019] 3 is a control block diagram of a mobile terminal 20 (terminal device). The mobile terminal 20 is assumed to be a smartphone here, but may also be a tablet PC (Personal Computer), a notebook PC, or the like. The mobile terminal 20 includes a control unit 21, a communication unit 22, a storage unit 23, a display unit 24, and an imaging unit 26, and each unit is connected by a bus 29. The control unit 21 is a processor that controls each unit. The storage unit 23 is composed of a ROM (Read Only Memory) and a RAM (Random Access Memory). The processor of the control unit 21 loads an application program stored in the ROM of the storage unit 23 into the RAM and executes it.

[0020] The communication unit 22 of the mobile terminal 20 is composed of, for example, an antenna and a transmission / reception circuit, and is formed so as to be able to transmit and receive communication data to and from the communication unit 36 ​​of the projection control device 30 via a wireless communication path. The display unit 24 and the operation unit 25 can be composed of a touch panel display having both a display function and an input function, in which a transparent touch panel as the operation unit 25 is laminated on the surface of a display panel composed of a liquid crystal display or an organic EL display as the display unit 24. The touch panel is composed of a capacitance type or the like that detects a position touched by a finger or the like. The user U can operate the operation screen displayed on the display unit 24 via the operation unit 25 to operate the mobile terminal 20 and the projection device 10. In addition, the mobile terminal 20 has an imaging unit 26 that functions as a camera, and can capture an image projected on the projection surface 3 and store the captured image in the storage unit 23.

[0021] The outline of the projection method for correcting and projecting the image so that the image appears rectangular as if it were projected onto a smooth screen or the like even when the user U views the image projected onto the projection surface 3 from an arbitrary position (prediction point, see FIG. 1) is as follows: first, the projection unit 11 of the projection device 10 projects a measurement image 41 (see FIG. 2) onto the projection surface 3, and the measurement image 41 projected from the measurement point (also called viewpoint 1 or first point) on the left side of the projection surface 3 shown in FIG. 1 is captured to calculate correction information 44 (see FIG. 2). Similarly, the measurement image 41 is captured from the measurement point (also called viewpoint 2 or second point) on the right side of the projection surface 3 to calculate the correction information 44. The weighted average of these correction information 44 is used to calculate the correction information 44 of the prediction point (i.e., the position where the user U views the projection image, also called the third point), and the projection image is corrected. Note that the second point is different from the first point, and the third point is different from the first point and the second point.

[0022] Here, each correction information 44 calculated corresponding to viewpoint 1 and viewpoint 2 (hereinafter referred to as correction information 44-1 (first correction data) and 44-2 (second correction data)) is correction information 44 for correcting the projection surface 3 so that it appears rectangular when viewed from each viewpoint. The calculation of correction information 44 at each viewpoint can use a known correction method based on measurement information 43 including a captured image (hereinafter also simply referred to as a "correction image") obtained by capturing the projected measurement image 41.

[0023] An example of calculation of the correction information 44 at each viewpoint will be described below with reference to the flowchart in Fig. 4. In the process of calculating the correction information 44, first, in step S103, the projection unit 11 of the projection control device 30 projects the measurement image 41 stored in the storage unit 40 of the projection control device 30 onto the projection surface 3. Here, the measurement image 41 may be an image of a checkered pattern, a horizontal stripe pattern, a vertical stripe pattern, or the like, and a plurality of measurement images 41 with different widths of the grid or stripe pattern may be used.

[0024] Next, in step S105, the projected measurement image 41 is captured, and a captured image (correction image) is generated. In the projection system 1, an image is captured by the imaging unit 26 of the portable terminal 20, and the captured image (correction image) is stored in the memory unit 23 of the portable terminal 20. The correction image stored in the memory unit 23 of the portable terminal 20 is transmitted from the memory unit 23 of the portable terminal 20 to the projection control device 30 via the communication unit 36 ​​of the projection control device 30, and is stored in the memory unit 40 as measurement information 43.

[0025] Next, in step S107, the image correction processing unit 34 calculates the pixel position (Pi) of the display element 50 corresponding to the pixel position (Si) of the captured image (correction image). This is processing indicated by the arrow S107a in which the coordinate Pi of the display element 50 as a DMD (Digital Micromirror Device) projecting the measurement image 41 is obtained with respect to the coordinate Si of the pixel position of the captured image (correction image) 26a obtained by capturing the projected measurement image 41, as shown in FIG.

[0026] Next, in step S109, the image correction processing unit 34 calculates pixel positions (Si) of the captured image (correction image) corresponding to pixel positions of the output video (or output image, i.e., measurement image 41). This is the process indicated by arrow S109a in FIG. 5. The correspondence relationship determined in step S107 is used to calculate the pixel positions (Si) of the captured image (correction image) corresponding to the pixel positions of the output video. In steps S107 and S109, the positional correspondence relationship between the captured image (correction image) and the pixels of the output video is acquired, and correction information 44 is calculated. In other words, the correction information 44 is calculated using the coordinates Pi of the display element 50 and the coordinates Si of the captured image.

[0027] Next, in step S111, a clipping rectangle is determined from the captured image (correction image). That is, as shown in Fig. 6, within the range of the projection image (measurement image 41 in Fig. 6) that is deformed and projected, a rectangle S111a having an aspect ratio corresponding to the display element 50 is the range that can be projected onto a rectangle by the projection device 10 (projection unit 11). The image correction processing unit 34 determines the range of this projectable rectangle from the measurement image 41 of the captured image (correction image) 26a.

[0028] Next, in step S113, the image correction processing unit 34 deforms the input image so that it fits within the cutout rectangular frame based on the correction information 44 of the shape of the projection image. That is, the calculation results (correction information 44) of steps S107 and S109 are used to deform the projection image for the user U to view so that it fits within the range of the rectangle S111a determined in step S111. The calculation of the correction information 44 is based on the fact that the correction image 50-41 of the display element 50 in FIG. 6 is deformed and projected like the correction image 3-41 on the projection target surface 3, and therefore, in order to project like the rectangle S111a on the projection target surface 3, it is sufficient to project the deformed image S113a on the display element 50.

[0029] Therefore, the coordinates Pj of the display element 50 after relative (projected image) deformation are calculated with respect to the pixel coordinates of the projected image (i.e., the coordinates of the display element 50), and further, the coordinates Pk of the display element 50 corresponding to the coordinates Pj of the display element 50 after deformation are calculated.

[0030] In this way, the image correction processing unit 34 can calculate correction information 44 for making the projection surface 3 appear as a neat rectangle when viewed from each viewpoint. This correction information 44 can be calculated for each measurement point (viewpoints 1 and 2). For viewpoint 1, correction information 44-1 is calculated, and for viewpoint 2, correction information 44-2 is calculated.

[0031] Then, the measurement processing unit 35 calculates correction information 44-F (third correction data) of the prediction point by using a weighted average of each correction information 44 (correction information 44-1 (first correction data), 44-2 (second correction data)) at a plurality of different viewpoints and the position of the prediction point (any position from which the user U views the projection surface 3, third position data) relative to the position (first position data, second position data) of each measurement point (viewpoint 1, viewpoint 2). As shown in FIG. 7, for example, the prediction deformation (calculation of correction information 44-F at prediction point 1, also called prediction correction) at the position where prediction point 1 divides the distance between viewpoint 1 and viewpoint 2 at l1:l2 uses formula (1). P=(l1P2+l2P1) / (l1+l2) ···(1)

[0032] Moreover, to perform prediction transformation of prediction point 2, which is externally divided by l1:l2, equation (2) is used, and to perform prediction transformation of prediction point 3, equation (3) is used. Prediction point 2;l1>l2P=(-l2P1+l1P2) / (l1-l2) ···(2) Prediction point 3;l1 <l2P=(l2P1-l1P2) / (-l1+l2) ···(3)

[0033] Here, P, P1, and P2 in formulas (1) to (3) are any of the coordinates Si, Pi, Pj, and Pk used when calculating the correction information 44-1 and 44-2. In other words, the weighted average is applied to the coordinates (Pi, Pj, Pk) of the display element 50 or the coordinate (Si) of the captured image (correction image) 26a. In summary, the weighted average can be performed for the following four cases. (A) Coordinates (Si) of the captured image 26a relative to the coordinates of the display element 50 (B) Coordinates (Pi) of the display element 50 relative to the coordinates (Si) of the captured image 26a (C) Coordinates (Pj) of the display element 50 after deformation relative to the coordinates of the display element 50 (D) Coordinates (Pk) of the display element 50 relative to the coordinates (Pj) of the display element 50 after deformation

[0034] The captured image 26a captured from each viewpoint and position information of each viewpoint (each measurement point) are stored in the storage unit 23 as measurement information 43. The predicted point and position information of each viewpoint can be acquired and determined from any of GPS information output from a GPS receiving unit included in the mobile terminal 20, angle information at the time of capturing the captured image 26a detected by an acceleration sensor included in the mobile terminal 20, or settings of the user U (user settings).

[0035] Thus, the projection method for calculating correction information 44-F (third correction data) at the prediction point (third location) and correcting the projection image includes the steps of: acquiring a first captured image (correction image) captured at a first location (viewpoint 1) and a second captured image (correction image) captured at a second location (viewpoint 2) via the communication unit 36; acquiring first correction data (correction information 44-1) based on the first captured image and acquiring second correction data (correction information 44-2) based on the second captured image; acquiring first position data of the first location, second position data of the second location, and third position data of a third location (prediction point); determining third correction data (correction information 44-F) corresponding to the third location using the first correction data, the second correction data, and the first to third position data; and correcting the projection image to be projected by the projection device 10 (projection unit 11) based on the determined third correction data.

[0036] The third correction data is calculated using a weighted average from the third position data for the first position data and the second position data for the first correction data and the second correction data. The step of acquiring the first correction data based on the first captured image and acquiring the second correction data based on the second captured image includes a step of determining coordinates of the display element for coordinates of pixel positions of the first captured image and the second captured image, and calculating the first correction data and the second correction data using the coordinates of the first captured image and the second captured image corresponding to the pixel positions of the output image using this correspondence. The step of determining the third correction data corresponding to the third point using the first correction data, the second correction data, the first position data, the second position data, and the third position data applies a weighted average to the coordinates of any of the above (A) to (D).

[0037] Specifically, the correction information 44-F (prediction correction, third correction data) of the prediction point is calculated according to the flowchart shown in FIG. 8. When the process of prediction correction is started, in step S203, the mobile terminal 20 captures the measurement image 41 projected on the projection surface 3 at the viewpoint 1, and measurement information 43-1 (see FIG. 1) including the captured image (correction image) is stored in the storage unit 40 of the projection control device 30. Next, in step 205, the image correction processing unit 34 calculates correction information 44-1 (first correction data) from the measurement information 43-1 (including the first captured image and the first position data) of the viewpoint 1 and stores it in the storage unit 40. Then, in step S206, the projection unit 11 projects a corrected image corresponding to the viewpoint 1 based on the correction information (first correction data) of the viewpoint 1.

[0038] In steps S207 and S209, similarly to steps S203 and S205, the measurement of the viewpoint 2 is performed (step S203), and the correction information 44-2 (second correction data) (see FIG. 1) of the viewpoint 2 is calculated from the measurement information 43-2 (including the second captured image and the second position data) of the viewpoint 2, and is stored in the storage unit 40 (step S205). Then, in step S210, the projection unit 11 projects a corrected image corresponding to the viewpoint 2 based on the correction information (second correction data) of the viewpoint 2. Then, in step S211, the position information (third position data) of the prediction point is detected, transmitted to the projection control device 30, and stored in the storage unit 40. Next, in step S213, the measurement processing unit 35 calculates the correction information 44-F (third correction data) (prediction correction) of the prediction point using a weighted average of the correction information 44-1 of the viewpoint 1 and the correction information 44-2 of the viewpoint 2. In step S215, the measurement processing unit 35 stores the calculated correction information 44-F of the predicted point in the storage unit 40.

[0039] Using the correction information 44-F of the predicted point calculated in this manner, the image correction processing unit 34 deforms the input image, and based on the projection image after the correction and deformation, the light source device 60 and the display element 50 of the projection unit 11 are controlled by the projection image processing unit 37 of the projection device 10, and the projection light of the projection image is projected from the projection optical system 15. Then, the user U can instantly see a beautiful rectangular projection image even if the image is projected onto the projection target surface 3 having an irregular surface, and even if the user U moves to any position.

[0040] Operations such as projecting and capturing a correction image at each measurement point, setting a predicted point position, and calculating correction information can be performed by the mobile terminal 20. Fig. 9 shows an example of a user interface screen 300 (UI screen 300) displayed on the display unit 24 (operation unit 25) of the mobile terminal 20. A "connect" button 301 and a "disconnect" button 302 at the upper left of the UI screen 300 are buttons for connecting and disconnecting with the projection device 10 (projection control device 30) via the communication units 22, 36.

[0041] A camera screen 320 is displayed on the left side of the UI screen 300. An image captured by the imaging unit 26 of the mobile terminal 20 is displayed on the camera screen 320. A measurement correction operation unit 310 is provided on the upper right of the UI screen 300. The measurement correction operation unit 310 includes a measurement unit 311 having a "Viewpoint 1 Measurement" button 311-1 and a "Viewpoint 2 Measurement" button 311-2 (these are the first operation unit), a correction unit 312 having a "Viewpoint 1 Correction" button 312-1 and a "Viewpoint 2 Correction" button 312-2 (these are the second operation unit), and a "Viewpoint +" button 315 and a "Correction +" button 316. The control unit 21 instructs the imaging unit 26 to capture the measurement image projected by the projection device 10 in response to the operation of the first operation unit, and acquires the correction image captured by the imaging unit 26. Furthermore, the control unit 21 generates instruction data for correcting the projection image projected by the projection device 10 (projection unit 11) based on the correction image in response to the operation of the second operation unit.

[0042] 1, the control unit 21 transmits projection instruction information for causing the projection control device 30 to project a measurement image 41 via the communication units 22, 36. Then, the measurement image 41 is emitted from the projection device 10 and projected onto the projection surface 3. Then, the imaging unit 26 captures the projected measurement image 41 and stores it in the storage unit 23. Similarly, when the user U presses the "measurement viewpoint 2" button 311-2 at the position of the viewpoint 2, the measurement image 41 is projected and captured, and stored in the storage unit 23.

[0043] Furthermore, for example, when the user U presses the "Viewpoint 1 Correction" button 312-1, instruction data is generated for correcting the projection image to be projected by the projection device 10 (projection unit 11) based on the correction image from the correction image and the position information of the viewpoint 1 stored in the storage unit 23. Then, the mobile terminal 20 transmits this instruction data to the projection device 10, and the projection device 10 projects the projection image corrected at the viewpoint 1.

[0044] A "Viewpoint +" button 315 is displayed to the right of the measurement unit 311. The "Viewpoint +" button 315 is a view point increase unit that can further increase two view points (viewpoints 1 and 2) (i.e., the measurement unit 311). The "Correction +" button 316 arranged to the right of the correction unit 312 is a correction increase unit for forming the correction unit 312 corresponding to the view point increased by the "Viewpoint +" button 315.

[0045] A "No correction" button 303 displayed in the approximate center on the right side of the UI screen 300 is used as a preview operation section for previewing the capture of the projected measurement image 41 by the imaging section 26 of the mobile terminal 20. Before capturing an image by the measurement section 311, a preview can be performed to check whether the projected image fits within the angle of view, etc. In this case, too, the state of the image can be confirmed on the camera screen 320.

[0046] A "Settings" button 340 is displayed to the right of the "No correction" button 303. The "Settings" button 340 allows the user to set the settings for the measurement image 41 projected by the projection device 10. When the "Settings" button 340 is pressed, the display unit 24 switches to a UI screen 349 in FIG. 10. The "Brightness of dot pattern" 341 in the upper part of the UI screen 349 allows the user to adjust the brightness of the measurement image 41. The "Pitch width of dot pattern" 342 allows the user to set the pitch width of the stripe pattern of the measurement image 41, which may be a lattice pattern (checkered pattern), horizontal stripes, or vertical stripes. The "Number of dot patterns" 343 allows the user to set the number (type) of the measurement images 41 to be projected. These settings can be made when an error occurs in the calculation of the measurement information 43 by the measurement unit 311.

[0047] A prediction point position determination unit 330 is displayed below the "No correction" button 303 and the "Set" button 340. The prediction point position determination unit 330 can determine a prediction point position for calculating correction information of a prediction point using a weighted average from a plurality of correction information corresponding to the operation by a combination of a first operation unit and a second operation unit provided in a plurality of correspondence with different measurement points and the position of a prediction point relative to the position of a measurement point. The prediction point position determination unit 330 displays a "prediction point correction" button 331 and a prediction point position determination slide bar 332. The prediction point position determination slide bar 332 displays a point 332a indicating the position of viewpoint 1, a point 332b indicating the position of viewpoint 2, a line 332c connecting viewpoints 1 and 2, and a point 332d of a prediction point that can be moved by sliding on the line 332c.

[0048] When the user U wants to determine a prediction point (position), the user U can set the prediction point using the prediction point position determination slide bar 332. The prediction point can be set by moving the point 332d indicating the prediction point along the line 332c. Therefore, it is possible to determine whether the prediction point is a position that divides the measurement point (viewpoints 1, 2) internally or externally (on which side of the viewpoints 1, 2 the prediction point is to be externally divided). The measurement processing unit 35 of the projection control device 30 determines whether to apply the weighted average using any of the above-mentioned formulas (1) to (3) based on the prediction point position set using the prediction point position determination slide bar 332. This operation is not necessary when the prediction point is based on GPS information output from the GPS receiver included in the mobile terminal 20 or angle information at the time of capturing the captured image detected by the acceleration sensor included in the mobile terminal 20.

[0049] After the prediction point is determined by sliding the point 332d indicating the prediction point, the position information of the prediction point (prediction point position) set by the prediction point position determination slide bar 332 is transmitted to the projection control device 30 by pressing the "prediction point correction" button 331. Based on the received position information of the prediction point, the measurement processing unit 35 calculates correction information 44-F at the prediction point using a weighted average. The projection control device 30 corrects the projection image based on the correction information 44-F of the prediction point. When the prediction point is calculated from a GPS or a captured image, the prediction point position determination slide bar 332 may be displayed as inoperable, the "prediction point correction" button 331 may be blinked, or the like, and the "prediction point correction" button 331 may be pressed to cause the measurement processing unit 35 to calculate the correction information 44-F at the prediction point using a weighted average.

[0050] Although the embodiment of the present invention has been described above, the present invention is not limited to the present embodiment, and various modifications can be made to the present invention. For example, the image correction processing unit 34 and the measurement processing unit 35 may be provided in the mobile terminal 20, and the mobile terminal 20 may calculate the correction information 44 of each viewpoint (viewpoints 1 and 2) and the prediction point. In this case, the measurement information 43 and the correction information 44 may be stored in the storage unit 23 of the mobile terminal 20. Also, the projection device 10 may be a projector, and the projection control device 30 may be a PC (personal computer).

[0051] In addition, in this embodiment, two measurement points (viewpoints) are used, but three or more measurement points can be used, and correction information 44-F of the predicted point can be calculated using a weighted average of correction information 44 at these multiple measurement points.

[0052] According to the above embodiment of the present invention, the projection control device 30 includes at least one processor, that is, a control unit 31, and a communication unit 36. The control unit 31 acquires a first captured image (correction image) captured at a first point (viewpoint 1) and a second captured image (correction image) captured at a second point (viewpoint 2) via the communication unit 36, acquires first correction data based on the first captured image, acquires second correction data based on the second captured image, acquires first position data of the first point, second position data of the second point, and third position data of the third point, respectively, determines third correction data corresponding to the third point using the first correction data, second correction data, first position data, second position data, and third position data, and corrects the projection image to be projected by the projection device 10 (projection unit 11) based on the determined third correction data. This allows the user U to instantly view the corrected projection image even if the user U moves the position at which the projection image is viewed, without having to project and capture the correction image one by one to calculate correction information.

[0053] Moreover, the third correction data is calculated using a weighted average from the first position data for the first correction data and the second correction data, and the third position data for the second position data, thereby making it possible to accurately calculate correction information 44 of the third position, which is a predicted point.

[0054] Furthermore, the correction information 44 (first correction data, second correction data) is calculated by finding the coordinates of the display element 50 for the coordinates of the pixel positions of the first captured image and the second captured image, and using this correspondence, using the coordinates of the first captured image and the second captured image corresponding to the pixel positions of the output image, and the weighted average is applied to the coordinates of the display element 50 or the coordinates of the captured image. This makes it possible to select the coordinates where appropriate correction is performed according to the environment in which the projection device 10 is placed, and calculate the correction information 44 of the predicted point by the weighted average.

[0055] The position of the predicted point, which is the third position data, is determined by any one of GPS, angle information at the time of capturing the captured images (first captured image, second captured image), and user setting. This allows the projection device 10 and the user U to determine an appropriate position of the predicted point depending on the environment in which the user U views the projected image.

[0056] Moreover, the projection control device 30 further includes a storage unit 40 that stores measurement information 43 (measurement information 43-1, 43-2, etc.) including captured images (correction images) (first captured image, second captured image) captured from the measurement points and position information (first to third position data) of the measurement points, and a plurality of correction information 44 (correction information 44-1, 44-2, etc.). As a result, correction information 44-F of the prediction point can be calculated based on the information stored in the storage unit 40, so that the correction information of the prediction point can be calculated without the need to perform measurements from the viewpoints 1 and 2 every time projection is performed.

[0057] The projection system 1 also includes a projection control device 30 including a communication unit 36, a projection device 10 having a projection unit 11 provided with a display element 50 having a plurality of micromirrors, and a portable terminal 20 formed to be able to capture a first captured image and a second captured image and capable of transmitting the first captured image, the second captured image, and the first position data, the second position data, and the third position data to the projection device 10 via the communication unit 36. This makes it possible to provide a projection system 1 that allows a user U to instantly view a corrected projection image even if the user U moves the position from which the projection image is viewed, without having to project and capture a correction image one by one to calculate correction information.

[0058] The projection method by the projection system 1 (projection device 10) includes the steps of acquiring a first captured image captured at a first location and a second captured image captured at a second location via the communication unit 36, acquiring first correction data based on the first captured image and acquiring second correction data based on the second captured image, acquiring first position data of the first location, second position data of the second location, and third position data of the third location, determining third correction data corresponding to the third location using the first correction data, the second correction data, the first position data, the second position data, and the third position data, and correcting the projection image to be projected by the projection device 10 (projection unit 11) based on the determined third correction data. This makes it possible to provide a projection method that allows the user U to instantly view a corrected projection image even if the user U moves the position where the projection image is viewed, without having to project and capture a correction image one by one to calculate correction information.

[0059] The third correction data is calculated using a weighted average of the first position data and the second correction data, and the third position data for the second position data, so that the correction information 44 at the third point can be generated immediately by simple calculation.

[0060] In addition, the step of acquiring first correction data based on the first captured image and acquiring second correction data based on the second captured image includes a step of determining coordinates of the display element 50 with respect to coordinates of pixel positions of the first captured image and the second captured image, and calculating the first correction data and the second correction data using the coordinates of the first captured image and the second captured image corresponding to the pixel positions of the output image using the correspondence relationship. The step of determining third correction data corresponding to a third point using the first correction data, the second correction data, the first position data, the second position data, and the third position data applies a weighted average to any of (A) coordinates (Si) of the captured image 26a with respect to the coordinates of the display element 50, (B) coordinates (Pi) of the display element 50 with respect to the coordinates (Si) of the captured image 26a, (C) coordinates (Pj) of the display element 50 after deformation with respect to the coordinates of the display element 50, and (D) coordinates (Pk) of the display element 50 with respect to the coordinates (Pj) of the display element 50 after deformation. This makes it possible to calculate correction information 44-F (third correction data) at the predicted point for the display element 50 that is the DMD. Then, depending on the situation in which the projection device 10 is installed, the weighted average can be applied using appropriate coordinates from (A) to (D) above.

[0061] The projection method may further include a step of obtaining the position of the prediction point from any of GPS, angle information at the time of capturing the captured image 26a, and user settings. This allows the position of the prediction point to be obtained accurately according to the situation.

[0062] The program (control program 42) causes the control unit 31 to function as follows: acquire a first captured image captured at a first location and a second captured image captured at a second location via the communication unit 36, acquire first correction data based on the first captured image, acquire second correction data based on the second captured image, acquire first position data of the first location, second position data of the second location, and third position data of the third location, respectively, determine third correction data corresponding to the third location using the first correction data, second correction data, first position data, second position data, and third position data, and correct the projection image to be projected by the projection device 10 (projection unit 11) based on the determined third correction data. This makes it possible to provide a program that causes the projection device 10 to function so that the user U can instantly view the corrected projection image even if the user U moves the position where the projection image is viewed, without having to project and capture a correction image one by one and calculate correction information.

[0063] Furthermore, the third correction data is calculated using a weighted average from the first position data and the third position data for the second position data, for the first correction data and the second correction data. This makes it possible to provide the above program that uses a simple calculation formula to increase the program execution speed.

[0064] The above-described embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims.

[0065] The invention described in the first claim of this application is set forth below. [1] At least one processor; A communication unit, The at least one processor: acquiring a first captured image captured at a first location and a second captured image captured at a second location different from the first location via the communication unit; acquiring first correction data based on the first captured image, and acquiring second correction data based on the second captured image; Acquire first position data of the first location, second position data of the second location, and third position data of a third location different from the first location and the second location, determining third correction data corresponding to the third location using the first correction data, the second correction data, the first position data, the second position data, and the third position data; correcting the projection image to be projected by the projection unit based on the determined third correction data; 1. A projection control device comprising: [2] The projection control device described in [1], characterized in that the third correction data is calculated using a weighted average from the first position data and the third position data for the second position data, for the first correction data and the second correction data. [3] The first correction data and the second correction data are calculated by obtaining coordinates of a display element for coordinates of pixel positions of the first captured image and the second captured image, and using the correspondence relationship obtained by the calculation using the coordinates of the first captured image and the second captured image corresponding to pixel positions of an output image; The weighted average is applied to the coordinates of the display element or the coordinates of the first captured image and the second captured image. The projection control device according to [2] above, [4] The projection control device described in any of [1] to [3], characterized in that the third position data is determined by any of GPS, angle information at the time of capturing the first captured image and the second captured image, or user settings. [5] The projection control device described in any of [1] to [4], further comprising a memory unit that stores the first position data, the second position data, the third position data, the first correction data, and the second correction data. [6] A projection device having the projection control device according to any one of [1] to [5] above and a projection unit provided with a display element having a plurality of micromirrors; a terminal device configured to be able to capture the first captured image and the second captured image, and capable of transmitting the first captured image, the second captured image, the first position data, the second position data, and the third position data to the projection device via the communication unit; 1. A projection system comprising: [7] acquiring a first captured image captured at a first location and a second captured image captured at a second location different from the first location via a communication unit; acquiring first correction data based on the first captured image and acquiring second correction data based on the second captured image; acquiring first position data of the first location, second position data of the second location, and third position data of a third location different from the first location and the second location; determining third correction data corresponding to the third location using the first correction data, the second correction data, the first position data, the second position data, and the third position data; correcting a projection image to be projected by a projection unit based on the determined third correction data; 13. A projection method comprising: [8] The projection method described in [7], characterized in that the third correction data is calculated using a weighted average from the first position data and the third position data for the second position data for the first correction data and the second correction data. [9] The step of acquiring the first correction data based on the first captured image and acquiring the second correction data based on the second captured image includes a step of determining coordinates of a display element for coordinates of pixel positions of the first captured image and the second captured image, and calculating the first correction data and the second correction data using the coordinates of the first captured image and the second captured image corresponding to pixel positions of an output image using the correspondence relationship; The projection method described in [8], wherein the step of determining the third correction data corresponding to the third location using the first correction data, the second correction data, the first position data, the second position data, and the third position data applies the weighted average to any of the coordinates of the first captured image, the coordinates of the second captured image relative to the coordinates of the display element, the coordinates of the display element relative to the first captured image, the second captured image, the coordinates of the display element after deformation relative to the coordinates of the display element, and the coordinates of the display element relative to the coordinates of the display element after deformation.

[10] The projection method described in any one of [7] to [9], further comprising a step of obtaining position information of the third position data from any one of GPS, angle information at the time of capturing the first captured image and the second captured image, and user settings.

[11] A program executed by a computer, comprising: acquiring a first captured image captured at a first location and a second captured image captured at a second location different from the first location via a communication unit; acquiring first correction data based on the first captured image, and acquiring second correction data based on the second captured image; Acquire first position data of the first location, second position data of the second location, and third position data of a third location different from the first location and the second location, determining third correction data corresponding to the third location using the first correction data, the second correction data, the first position data, the second position data, and the third position data; correcting the projection image to be projected by the projection unit based on the determined third correction data; A program that makes it work like this.

[12] The program described in

[11] , characterized in that the third correction data is calculated using a weighted average from the first position data and the third position data for the second position data, for the first correction data and the second correction data. [Explanation of symbols]

[0066] 1 Projection system 2 Curtain 3 Projection surface 10 Projection device 11 Projection section 15 Projection optical system 16 Motor 20 Mobile terminal 21 Control section 22 Communication section 23 Storage section 24 Display section 25 Operation unit 26 Imaging unit 26a Image 29 Bus 30 Projection control device 31 Control unit 32 Operation section 33 Image input section 34 Image correction processing unit 35 Measurement processing unit 36 Communication unit 37 Projection image processing unit 38 Audio processing unit 38a Speaker 40 Memory unit 41 Measurement image 42 Control program 43 Measurement information 44 Correction information 50 Display element 60 Light source device 300 User interface screen (UI screen) 310 Measurement correction operation section 311 Measurement section 312 correction unit 320 camera screen 330 Prediction point position determination section 332 Prediction point position determination slide bar 349 UI screen U User

Claims

1. At least one processor; A communication unit, The at least one processor acquiring, via the communication unit, a first captured image obtained by capturing a measurement image projected by a projection unit at a first location and a second captured image obtained by capturing the measurement image projected by the projection unit at a second location different from the first location; acquiring first correction data by determining a range of a projectable rectangle having an aspect ratio corresponding to a display element of the projection unit from within the measurement image in the first captured image, and acquiring second correction data by determining a range of a projectable rectangle having an aspect ratio corresponding to the display element from within the measurement image in the second captured image; Acquiring first position data of the first location, second position data of the second location, and third position data of a third location different from the first location and the second location, determining third correction data corresponding to the third location using the first correction data, the second correction data, the first position data, the second position data, and the third position data; correcting the projection image to be projected by the projection unit based on the determined third correction data; 1. A projection control device comprising:

2. 2. The projection control device according to claim 1, wherein the third correction data is calculated using a weighted average of the first correction data and the second correction data based on the first position data, the second position data, and the third position data.

3. 3. The projection control device according to claim 2, characterized in that the first correction data and the second correction data are calculated using the first imaging coordinates and the second imaging coordinates corresponding to the coordinates of the display pixel position of the display element in an output image output to the display element, based on a relationship between the coordinates of a display pixel position of the display element and first imaging coordinates of a pixel position in the first captured image and second imaging coordinates of a pixel position in the second captured image, which correspond to the coordinates of the display pixel position, respectively.

4. 4. The projection control device according to claim 1, wherein the third position data is determined by either a GPS or a user setting.

5. 5. The projection control device according to claim 1, further comprising a storage unit that stores the first position data, the second position data, the third position data, the first correction data, and the second correction data.

6. a projection device comprising: the projection control device according to claim 1; and the projection unit provided with the display element having a plurality of micromirrors; a terminal device configured to be able to capture the first captured image and the second captured image, and capable of transmitting the first captured image, the second captured image, the first position data, the second position data, and the third position data to the projection device via the communication unit; 1. A projection system comprising:

7. A process of acquiring, via a communication unit, a first captured image of a measurement image projected by a projection unit captured at a first location and a second captured image of the measurement image projected by the projection unit captured at a second location different from the first location; acquiring first correction data by determining a range of a projectable rectangle having an aspect ratio corresponding to a display element of the projection unit from within the measurement image in the first captured image, and acquiring second correction data by determining a range of a projectable rectangle having an aspect ratio corresponding to the display element from within the measurement image in the second captured image; acquiring first position data of the first location, second position data of the second location, and third position data of a third location different from the first location and the second location; determining third correction data corresponding to the third location using the first correction data, the second correction data, the first position data, the second position data, and the third position data; correcting a projection image to be projected by the projection unit based on the determined third correction data; 13. A projection method comprising:

8. 8. The projection method according to claim 7, wherein the third correction data is calculated using a weighted average of the first correction data and the second correction data based on the first position data, the second position data, and the third position data.

9. 9. The projection method according to claim 8, wherein the step of acquiring the first correction data based on the first captured image and the second correction data based on the second captured image includes a step of calculating the first correction data and the second correction data using the first captured coordinates and the second captured coordinates corresponding to coordinates of a display pixel position of the display element in an output image that is output to the display element, based on a relationship between coordinates of a display pixel position of a display element and first captured coordinates of a pixel position in the first captured image and second captured coordinates of a pixel position in the second captured image that respectively correspond to the coordinates of the display pixel position.

10. 10. The projection method according to claim 7, further comprising the step of obtaining the position information of the third position data by using either a GPS or a user setting.

11. A program executed by a computer, the computer comprising: acquiring, via a communication unit, a first captured image obtained by capturing a measurement image projected by a projection unit at a first location and a second captured image obtained by capturing the measurement image projected by the projection unit at a second location different from the first location; acquiring first correction data by determining a range of a projectable rectangle having an aspect ratio corresponding to a display element of the projection unit from within the measurement image in the first captured image, and acquiring second correction data by determining a range of a projectable rectangle having an aspect ratio corresponding to the display element from within the measurement image in the second captured image; Acquiring first position data of the first location, second position data of the second location, and third position data of a third location different from the first location and the second location, determining third correction data corresponding to the third location using the first correction data, the second correction data, the first position data, the second position data, and the third position data; correcting the projection image to be projected by the projection unit based on the determined third correction data; A program that makes it work like this.

12. 12. The program according to claim 11, wherein the third correction data is calculated using a weighted average of the first correction data and the second correction data based on the first position data, the second position data, and the third position data.

Images