Control method, program, and projection system for terminal devices

Abstract

Even when there is an object other than the projection target inside the projection target, the reduction in the accuracy of the correction amount for correcting the shape and position of the image is suppressed. [Solution] The processor 370 displays a query image 325 on the touch panel 320, acquires an image of the screen 30 captured by the camera 330, and when it receives a first response indicating that the detection light irradiator 50 is in the area inside the first side, it calculates a first correction value to correct at least one of the shape or position of the image based on the second side of the screen 30, which is different from the first side in the captured image. When it receives a second response indicating that the detection light irradiator 50 is outside the screen 30 beyond the first side, it calculates a second correction value to correct at least one of the shape or position of the image based on both the first and second sides in the captured image.

Description

[[Technical Field]]

[0001] The present invention relates to a control method for a terminal device, a program, and a projection system. [[Background Art]]

[0002] Conventionally, there has been known a technique for correcting the shape of an image projected onto a projection surface based on a captured image obtained by a camera. For example, in the image display device of Patent Document 1, an imaging unit captures an image of a pattern projected onto a projection surface by a projection unit and an image of a projection target object, and the relative positional relationship between an image of a projection panel corresponding to the pattern image and the image of the projection target object is detected based on the captured image. Thereafter, a correction unit of the image display device performs correction based on a detection result obtained by an image analysis unit. [[Prior Art Documents]] [[Patent Documents]]

[0003] [[Patent Document 1]] Japanese Unexamined Patent Application Publication No. 2014-27457 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0004] However, when an obstacle exists within the projection surface, the calculation accuracy of a correction value obtained by performing image analysis on the captured image decreases, and the accuracy of image correction based on the correction value also decreases. [[Means for Solving the Problem]]

[0005] This disclosure relates to a control method for a terminal device comprising a camera that captures an image of a projection target onto which an image is projected from a projection device, a display, and one or more processors, wherein the one or more processors are instructed to display on the display an inquiry image asking whether an object different from the projection target is located in an area inside or outside the first edge of the projection target; to acquire an image of the projection target captured by the camera; to calculate a first correction value for correcting at least one of the shape or position of the image based on a second edge of the projection target different from the first edge in the image, when a first response indicating that the object is located in an area inside the first edge is received; and to calculate a second correction value for correcting at least one of the shape or position of the image based on both the first and second edges in the image, when a second response indicating that the object is located outside the projection target beyond the first edge is received.

[0006] This disclosure is a program that causes a computer to control a terminal device comprising a camera that images a projection target onto which an image is projected from a projection device, and a display, to display on the display an inquiry image asking whether an object different from the projection target is located in an area inside or outside the projection target beyond the first edge of the projection target; to acquire an image of the projection target captured by the camera; to calculate a first correction value for correcting at least one of the shape or position of the image based on a second edge of the projection target different from the first edge in the image, when a first response is received indicating that the object is located inside the area inside the first edge; and to calculate a second correction value for correcting at least one of the shape or position of the image based on both the first and second edges in the image, when a second response is received indicating that the object is located outside the projection target beyond the first edge.

[0007] This disclosure provides a terminal device comprising: a projection device that projects an image onto a projection target and includes a receiving circuit for receiving data; a camera for capturing images of the projection target; a display; one or more processors; and a transmitting circuit for transmitting the data, wherein the one or more processors cause the display to show a query image asking whether an object different from the projection target is in an area inside or outside the first edge of the projection target; acquires an image of the projection target captured by the camera; and, upon receiving a first response indicating that the object is in an area inside the first edge, the terminal device determines whether the object is different from the first edge in the image of the projection target. This projection system performs the following actions: calculate a first correction value to correct at least one of the shape or position of the image based on the second side; if a second response is received indicating that the object is outside the projection target beyond the first side, calculate a second correction value to correct at least one of the shape or position of the image based on both the first and second sides of the captured image; transmit the first correction value or the second correction value to the projection device via the transmission circuit; and the projection device projects the corrected image, which has been corrected based on the first correction value or the second correction value received by the reception circuit, onto the projection target. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram showing the system configuration of the projection system. [Figure 2] A perspective view showing the installation of the projection device. [Figure 3] A block diagram showing the configuration of the projection device. [Figure 4] A diagram showing the configuration of a projector. [Figure 5] A block diagram showing the configuration of the terminal device. [Figure 6] This diagram shows the corrected shape of the projection area when the detection light irradiation device is mistakenly recognized as part of the screen frame. [Figure 7] A flowchart illustrating the operation of the terminal device's control unit. [Figure 8] Figure illustrating an example of a guide image. [Figure 9] Figure illustrating an example of an inquiry image. [Figure 10] Figure illustrating an example of a first projection pattern image. [Figure 11] Figure illustrating an example of a second projection pattern image. [Figure 12] Flowchart illustrating the overall operation of a terminal device. [Figure 13] Explanatory diagram for explaining insufficient edge complement processing. [Figure 14] Explanatory diagram for explaining insufficient edge complement processing. [Figure 15] Explanatory diagram for explaining insufficient edge complement processing. [Figure 16] Explanatory diagram for explaining insufficient edge complement processing. [Figure 17] Explanatory diagram for explaining insufficient edge complement processing. [Figure 18] Flowchart illustrating the processing operation of insufficient edge complement processing when an upper edge and a lower edge are detected. [Figure 19] Explanatory diagram for explaining insufficient edge complement processing. [Figure 20] Explanatory diagram for explaining insufficient edge complement processing. [Figure 21] Explanatory diagram for explaining insufficient edge complement processing. [Figure 22] Flowchart illustrating the processing operation of insufficient edge complement processing when an upper edge and a left edge are detected. [Figure 23] Explanatory diagram for explaining insufficient edge complement processing. [Figure 24] Explanatory diagram for explaining insufficient edge complement processing. [Figure 25] Explanatory diagram for explaining insufficient edge complement processing. [Figure 26] Explanatory diagram for explaining insufficient edge complement processing. [Figure 27] Explanatory diagram for explaining insufficient edge complement processing. [Figure 28]A flowchart illustrating the process of calculating a correction value to adjust the shape of the projection area based on the screen's normal vector. [Figure 29] A diagram showing the output of a 3-axis accelerometer, the screen normal vector, and the vertical and horizontal vanishing points. [Figure 30] A diagram showing the four vertices of a liquid crystal panel in a normalized panel coordinate system. [Figure 31] A diagram showing the coordinates of the four vertices in the screen coordinate system after the transformation. [Figure 32] A diagram showing the intersection of a rectangle centered at the intersection of its diagonals and a quadrilateral formed by the four vertices of the transformed screen coordinate system. [Figure 33] A diagram showing the enlarged rectangular shape. [Figure 34] A diagram showing the enlarged rectangular shape. [Modes for carrying out the invention]

[0009] [1. Projection System Configuration] The embodiments will be described below with reference to the attached drawings. Figure 1 shows the system configuration of projection system 1. The projection system 1 comprises a projection device 100 that projects an image onto a screen 30, which is the projection surface, and a terminal device 300, and these devices are connected via a network 5. The screen corresponds to an example of the object to be projected.

[0010] The projection device 100 displays a projected image in the projection area 70 of the screen 30 by projecting image light onto the screen 30. The projected image is the image displayed when the projection device 100 projects image light onto the screen 30. The projection area 70 is the area of ​​the screen 30 from which the projection device 100 can project image light.

[0011] Next, the configuration of the projection device 100 will be described with reference to Figures 1 and 2. Figure 2 is a perspective view showing the projection device in its installed state. Figures 1 and 2 show the mutually orthogonal X, Y, and Z axes. The Y axis is parallel to the vertical direction, and the X and Z axes are parallel to the horizontal direction. In Figures 1 and 2, the screen 30 is parallel to the Y axis. When standing facing the screen 30, the X axis indicates the left-right direction, and the Z axis indicates the front-back direction. The positive direction of the X axis indicates the right direction, the positive direction of the Y axis indicates the upward direction, and the positive direction of the Z axis indicates the forward direction.

[0012] A screen plate 33 is positioned where the projection device 100 projects image light. The front surface of the screen plate 33 is used as a screen 30. In this embodiment, the screen 30 is a rectangle with a long side parallel to the X-axis and a short side parallel to the Y-axis. The projection device 100 is fixed to the screen plate 33 in the forward and upward directions by a support member 35. In Figure 1, the screen 30 is positioned vertically, but the screen 30 may also be positioned horizontally. Furthermore, this embodiment illustrates the case where the projection device 100 projects onto a flat screen 30, but the projection target is not limited to the screen 30; it may be a flat surface such as a building wall, or it may be a curved or uneven surface. The screen 30 is surrounded by four screen frames 40 in the drawing: the upper frame 41, the right frame 42, the lower frame 43, and the left frame 44. When a blackboard or whiteboard is used as the screen 30, the lower frame 43 functions, for example, as a tray for chalk or markers. It is also possible for the screen 30 to have only the lower frame 43, and for there to be no screen frames 40 surrounding the screen 30.

[0013] Furthermore, in the projection system 1, position indication operations can be performed on the screen 30 using the indicator 10, and the projection device 100 detects the indicated position indicated by the indicator 10. The indicator 10 can be a pen, the user's fingers, etc. In this embodiment, the case where the indicator 10 is the user's fingers will be described.

[0014] The projection device 100 includes a projector 200, an imaging unit 150 including a camera 155, a detection light irradiation device 50, and an operation panel (not shown) including a plurality of operation buttons. The projector 200 projects a projection image onto the screen 30. The camera 155 captures the projected image and outputs the captured image. The field of view of the camera 155, i.e., the imaging range, is at least the range on the screen 30 that includes the projected image. The projection lens of the projector 200 and the imaging lens of the camera 155 are located on the lower surface of the projection device 100.

[0015] The detection light emitting device 50 corresponds to an example of a detection device as an object. As shown in Figure 2, the detection light emitting device 50 emits detection light 55 to detect the indicated position of the indicator 10. The object does not have to be the detection light emitting device 50, but could be, for example, furniture or an obstacle. The detection light emitting device 50 is positioned, for example, above the upper frame edge 41, overlapping with the upper frame edge 41, or below the upper frame edge 41, i.e., in the upper part of the screen 30 area. The detection light irradiator 50 emits detection light 55 in a direction that covers the screen 30 to detect the tip of the indicator 10. Specifically, the detection light irradiator 50 emits detection light 55 in a planar manner along the screen 30. Near-infrared light is used as the detection light 55 emitted by the detection light irradiator 50. The detection light irradiation device 50 includes an emission unit 51 and an adjustment mechanism 53 for adjusting the emission direction in which the emission unit 51 emits infrared light. The detection light irradiation device 50 may also be configured to include multiple emission units 51.

[0016] The area from which the detection light irradiation device 50 emits detection light 55 includes the projection area 70, preferably the area including the screen 30. In the normal mode, when the detection light 55 of the projection device 100 is adjusted, the projector 100 detects the reflected detection light 57, which is the reflected light of the detection light 55 emitted by the detection light irradiator 50 and reflected by the indicator 10, as a bright spot in the image captured by the camera 155. In this case, the operation of the indicator 10 can be detected in the area where the detection light 55 is emitted and which is included in the field of view of the camera 155. In other words, the area of ​​the area where the detection light irradiator 50 emits the detection light 55 and which is included in the field of view of the camera 155 is the detection area in which the operation of the indicator 10 can be detected.

[0017] The camera 155 has at least a first imaging function that receives and captures light in a wavelength range including the wavelength of the detection light 55 emitted by the detection light irradiator 50. Preferably, the camera 155 further has a second imaging function that receives and captures light including visible light, and these two imaging functions are configured to be switchable. For example, the camera 155 preferably includes a near-infrared filter switching mechanism (not shown) that can position a near-infrared filter, which blocks visible light and allows only near-infrared light to pass through, in front of the lens or move it back from in front of the lens.

[0018] The projection device 100 detects the instruction operation of the indicator 10, identifies the instruction position, and performs an operation corresponding to the instruction position. For example, the projection device 100 operates in whiteboard mode, drawing characters, shapes, line drawings, etc., in response to the operation of the indicator 10, and projecting them as a projected image. Furthermore, the projection device 100 can operate in modes other than whiteboard mode, and can perform an operation mode in which it projects an image based on image data input from an image source (not shown), such as a personal computer.

[0019] [2. Projection Device Configuration] Figure 3 is a block diagram showing the configuration of the projection device 100. The projection device 100 includes a wireless communication interface 110, an image processing unit 120, a frame memory 125, a remote control light receiver 130, a short-range wireless communication interface 140, an imaging unit 150, a projector 200, and a control unit 170. Hereinafter, interfaces will be abbreviated as I / F.

[0020] The wireless communication interface (I / F) 110 is equipped with an interface circuit that supports wireless communication standards such as Wi-Fi, and is connected to the network 5. The wireless communication interface (I / F) 110 communicates data with external devices, including the terminal device 300, via the network 5. Wi-Fi is a registered trademark. The wireless communication interface (I / F) 110 corresponds to an example of a receiving circuit.

[0021] The image processing unit 120 receives image data received from an external device via the wireless communication interface 110. A frame memory 125 is connected to the image processing unit 120. The frame memory 125 has multiple banks. Each bank has a storage capacity capable of writing image data for one frame. The frame memory 125 is configured, for example, as SDRAM (Synchronous Dynamic Random Access Memory). The image processing unit 120 expands the image data input from the wireless communication interface 110 into the frame memory 125.

[0022] The image processing unit 120 performs image processing on the image data expanded in the frame memory 125. The image processing performed by the image processing unit 120 includes, for example, resolution conversion or resizing, distortion correction, shape correction, digital zoom, and adjustment of image hue and brightness. The image processing unit 120 executes the processing specified by the control unit 170 and, if necessary, uses parameters input from the control unit 170. The image processing unit 120 can also combine and execute multiple image processing operations as described above. The image processing unit 120 reads the image data expanded in the bank selected by the control unit 170 from the frame memory 125 and outputs the read image data to the projector 200.

[0023] The image processing unit 120 and the frame memory 125 are composed of, for example, integrated circuits. Integrated circuits include LSIs (Large Scale Integration), ASICs (Application Specific Integrated Circuits), PLDs (Programmable Logic Devices), FPGAs (Field-Programmable Gate Arrays), SoCs (System-on-a-chip), etc. Furthermore, analog circuits may be included as part of the configuration of the integrated circuit, and the control unit 170 may be configured in combination with an integrated circuit.

[0024] The remote control receiver 130 receives infrared signals transmitted from the remote control 135. The remote control 135 has multiple buttons, such as a power button and a source switching button. The remote control 135 transmits infrared signals corresponding to the buttons operated by the user to the projection device 100. The remote control receiver 130 outputs an operation signal corresponding to the received infrared signal to the control unit 170. The operation signal is a signal corresponding to the button on the remote control 135 operated by the user.

[0025] The short-range wireless communication interface (I / F) 140 is equipped with an interface circuit compatible with wireless communication standards such as Bluetooth, and is wirelessly connected to the detection light irradiator 50. The short-range wireless communication interface (I / F) 140 transmits control signals input from the control unit 170 to the detection light irradiator 50. Bluetooth is a registered trademark.

[0026] The imaging unit 150 has a camera 155. The camera 155 receives and captures light in a wavelength range that includes the wavelength of the detection light 55 described with reference to Figure 2. The camera 155 receives and captures the reflected detection light 57, which is the reflected light when the detection light 55 emitted by the detection light irradiation device 50 is reflected off the indicator 10. Therefore, the reflected detection light 57 can be detected from the image captured by the camera 155.

[0027] Preferably, the camera 155 has a function to capture images using light including visible light, in addition to a function to capture images using light including near-infrared light. In this case, the projected image projected onto the screen 30 can be captured by the camera 155, and the control unit 170 can use that image to perform trapezoidal distortion correction, color correction, etc.

[0028] The position detection unit 160 detects reflected light 57 from the image captured by the camera 155. The position detection unit 160 identifies the position of the detected light image in the captured image and detects the position of the indicator 10.

[0029] Figure 4 shows the configuration of the projector 200. Now, we will explain the configuration of projector 200 with reference to Figure 4. The projector 200 generates image light by modulating light emitted from the light source 210 using a liquid crystal panel 230, and then projects the generated image light onto an optical unit 250. The projector 200 comprises a light source 210, three liquid crystal panels 230R, 230G, and 230B as light modulation devices, an optical unit 250, and a panel drive unit 270. Hereinafter, the liquid crystal panels 230R, 230G, and 230B of the projector 200 will be collectively referred to as the liquid crystal panel 230.

[0030] The light source 210 includes discharge-type light sources such as ultra-high pressure mercury lamps and metal halide lamps, or solid-state light sources such as light-emitting diodes and semiconductor lasers. Light emitted from the light source 210 is incident on the liquid crystal panel 230. Liquid crystal panels 230R, 230G, and 230B are each composed of transmissive liquid crystal panels in which liquid crystal is sealed between a pair of transparent substrates. Liquid crystal panel 230R modulates red light, liquid crystal panel 230G modulates green light, and liquid crystal panel 230B modulates blue light. Each liquid crystal panel has a pixel region formed therein, consisting of multiple pixels arranged in a matrix, and a driving voltage can be applied to each pixel of the liquid crystal.

[0031] Image data processed by the image processing unit 120 is input to the panel drive unit 270. The panel drive unit 270 applies a drive voltage corresponding to the input image data to each pixel in the pixel area, setting each pixel to a light transmittance corresponding to the image data. Light emitted from the light source 210 is modulated for each pixel as it passes through the pixel areas of the liquid crystal panels 230R, 230G, and 230B, forming image light corresponding to the image data for each color of light. The formed image light of each color is combined for each pixel by a color synthesis optical system (not shown) to form image light representing a color image. The optical unit 250, equipped with a projection lens and the like, projects the image light modulated by the liquid crystal panels 230R, 230G, and 230B onto the screen 30.

[0032] Returning to Figure 1, we will continue to explain the configuration of the projection device 100. The control unit 170 is a computer device comprising a storage unit 180 and a processor 190.

[0033] The memory unit 180 includes RAM (Random Access Memory) and ROM (Read Only Memory). The RAM is used, for example, as the arithmetic area of ​​the processor 190. The ROM stores control programs 185 that control the operation of the processor 190 and various setting data.

[0034] The processor 190 is an arithmetic processing unit that includes a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit). The processor 190 may consist of a single processor or multiple processors. Furthermore, the processor 190 may consist of part or all of the memory unit 180, or an SoC integrated with other circuits. The processor 190 may also consist of a combination of a CPU that executes programs and a DSP (Digital Signal Processor) that performs predetermined arithmetic processing. Additionally, the processor 190 may be configured with all its functions implemented in hardware, or it may be configured using programmable devices.

[0035] [3. Terminal Device Configuration] Next, the configuration of the terminal device 300 will be described with reference to Figure 5. The terminal device 300 includes a wireless communication interface 310, a touch panel 320, a camera 330, a 3-axis accelerometer 340, and a control unit 350.

[0036] The wireless communication interface (I / F310) is equipped with an interface circuit that supports wireless communication standards such as Wi-Fi, and is connected to the network 5. The wireless communication interface (I / F310) communicates data with the projection device 100 via the network 5. The wireless communication interface (I / F310) corresponds to an example of a transmission circuit.

[0037] The touch panel 320 corresponds to an example of a display. It comprises a display panel such as a liquid crystal panel or an organic EL (Electro-Luminescence) panel, and a touch sensor. The display panel displays an image generated by the control unit 350. The touch sensor is a sensor that detects touch operations on the display panel. The touch sensor outputs a signal to the control unit 350 indicating the position of the display panel touched by the user.

[0038] The camera 330 includes an image sensor such as a CCD (Charge-Coupled Device) or CMOS (Complementary Metal-Oxide-Semiconductor), and a data processing circuit that generates an image from the output of the image sensor.

[0039] The 3-axis accelerometer 340 is a sensor that measures acceleration in three axes.

[0040] The control unit 350 is a computer device comprising a storage unit 360 and a processor 370.

[0041] The memory unit 360 includes RAM and ROM. RAM is used, for example, as the arithmetic area for the processor 370. ROM stores the OS (Operating System) and application programs that control the operation of the processor 370. Hereinafter, application programs will be abbreviated as APP.

[0042] The processor 370 is an arithmetic processing unit, such as a CPU or MPU. The processor 370 may consist of a single processor or multiple processors. Furthermore, the processor 370 may consist of part or all of the memory unit 360, or an integrated SoC with other circuits. Alternatively, the processor 370 may consist of a combination of a CPU that executes programs and a DSP that performs predetermined arithmetic processing. In addition, the processor 370 may be configured with all its functions implemented in hardware, or it may be configured using programmable devices.

[0043] Figure 6 shows the corrected shape of the projection area 70 when the detection light irradiation device 50 is mistakenly recognized as part of the screen frame 40. The projection system 1 is a system that uses a terminal device 300 to correct at least one of the position and shape of the projection area 70, which is the area in which the projection device 100 can display an image on the screen 30. The projection device 100 displays the projection pattern image 90 on the screen 30, and the terminal device 300 captures an image of the area including the screen 30 on which the projection pattern image 90 is displayed, generating correction parameters to correct the shape of the projection area 70. The projection pattern image 90 includes the first projection pattern image 91 and the second projection pattern image 93, which will be described later. In this case, if the detection light irradiation device 50 is installed inside the screen frame 40, the detection light irradiation device 50 may be recognized as part of the screen frame 40, causing the shape of the projection area 70 to be corrected to an incorrect shape, and sometimes resulting in distortion of the shape of the projected image. The operation of the projection system 1 to solve this problem will be described below.

[0044] [4. Overall Operation] Figure 7 is a flowchart showing the operation of the control unit 350 of the terminal device 300. The operation of the control unit 350 of the terminal device 300 will be explained with reference to the flowchart shown in Figure 7. First, when APP363 is selected by the user's touch operation, the control unit 350 executes the selected APP363 to connect the terminal device 300 to the network 5 (step S1). The APP363 selected here is an application program that corrects the projection position and shape of the projection area 70 of the projection device 100.

[0045] Next, the control unit 350, which executes APP363, instructs the projection device 100 to display the guide image 80 (step S2). The projection device 100 displays the guide image 80 on the screen 30 at the instruction of the terminal device 300. Alternatively, the projection device 100 may display the guide image 80 on the screen 30 at its own discretion once communication with the projection device 100 via the network 5 becomes possible.

[0046] The guide image 80 may be projected not by instructions from the terminal device 300, but in response to operations on the operation panel of the projection device 100 or operations on the OSD menu using the remote control. In this case, the control unit 350 may display a prompting image on the touch panel 320 to encourage the user to project the guide image 80 onto the projection device 100 by operating on the operation panel of the projection device 100 or operations on the OSD menu using the remote control. In this case, step S2 may be omitted from the flowchart in Figure 7, or a step of displaying a prompting image may be added to the flowchart instead of step S2.

[0047] Figure 8 shows an example of a guide image 80. Guide image 80 is an image in which the color or thickness of the upper guide edge 81 of guide image 80 differs from the other edges of guide image 80, such as the left edge, bottom edge, and right edge. The upper guide edge 81 is the edge that constitutes the top edge of guide image 80 in the drawing view, and is the edge closest to the projection device 100. At this time, the control unit 350 may display a guidance display (message) on the touch panel 320 that says, "Please adjust the adjustment mechanism so that the upper guide edge 81 of guide image 80 overlaps with the detection light irradiation device 50." The illustration of the adjustment mechanism is omitted.

[0048] The guide image 80 is not limited to the above configuration; the color or thickness of the upper guide edge 81 of the guide image 80 may be the same as the other edges of the guide image 80, such as the left, right, bottom, and right edges. In addition, a guide image that indicates the positional relationship between the upper guide edge 81 and the detection light irradiation device 50 may be projected onto the projection area 70 instead of a guide display. This guide image is, for example, a schematic diagram that shows at least part of the procedure for adjusting the positional relationship between the upper guide edge 81 and the detection light irradiation device 50, or an ideal positional relationship, using diagrams and text. This guide image may be projected together with the guide image 80, or it may be projected prior to projecting the guide image 80. When the guide image is projected together with the guide image 80, it may be projected in the area inside the guide image 80.

[0049] The user operates the adjustment mechanism to adjust the display position of the guide image 80 so that the upper guide edge 81 of the guide image 80 overlaps with the detection light irradiation device 50. Once the user has finished adjusting the display position of the guide image 80, they input a predetermined touch operation to indicate that the adjustment of the display position of the guide image 80 has been completed. When the control unit 350 detects the predetermined touch operation, it displays an inquiry image 325 on the touch panel 320 (step S3).

[0050] Figure 9 shows an example of inquiry image 325. The inquiry image 325 shown in Figure 9 is an image displayed on the touch panel 320, but the projection device 100 may also display it on the screen 30 as an OSD (On Screen Display) image.

[0051] Inquiry image 325 includes a schematic diagram illustrating the positional relationship between the detection light irradiation device 50 and the upper frame edge 41, which is an example of the first edge of the screen 30. The inquiry image 325 includes the first schematic diagram 325A, the second schematic diagram 325B, radio buttons 325C, radio buttons 325D, and the confirmation button 325E. Schematic diagram 325A is a schematic diagram showing the detection light irradiation device 50 installed within the frame of the screen frame 40. Schematic diagram 325B is a schematic diagram showing the detection light irradiation device 50 installed outside the frame of the screen frame 40. Radio button 325C is selected when selecting schematic diagram 325A. Radio button 325D is selected when selecting schematic diagram 325B.

[0052] When the inquiry image 325 shown in Figure 9 is displayed on the touch panel 320, the user selects either radio button 325C or radio button 325D, and then selects the confirmation button 325E to confirm the operation.

[0053] The configuration is not limited to requiring the user to select either radio button 325C or radio button 325D. For example, radio buttons 325C and 325D may be omitted, and instead, schematic diagram 325A may be given the function of radio button 325C, while schematic diagram 325B may be given the function of radio button 325D. Furthermore, the configuration is not limited to requiring the user to select a confirmation button 325E after making a selection. The confirmation button 325E may be omitted, and the system may transition to the next screen when either radio button 325C or radio button 325D is selected.

[0054] Figure 10 shows an example of the first projection pattern image 91 displayed on the screen 30. When the confirmation button 325E for the inquiry image 325 is pressed, the control unit 350 instructs the projection device 100 to project the first projection pattern image 91 (step S4). The projection device 100 reads the image data that will be the basis of the first projection pattern image 91 from the storage unit 180 according to the instructions of the terminal device 300 and outputs it to the image processing unit 120. The image processing unit 120 performs predetermined image processing on the input image data and outputs the processed image data to the projector 200. The projector 200 generates image light based on the input image data and projects the generated image light onto the screen 30.

[0055] The first projection pattern image 91 may be projected not by instructions from the terminal device 300, but in response to operations on the operation panel of the projection device 100 or operations on the OSD menu using a remote control. In this case, the control unit 350 may display an image on the touch panel 320 prompting the user to project the first projection pattern image 91 onto the projection device 100 by operations on the operation panel of the projection device 100 or operations on the OSD menu using a remote control. In this case, step S4 may be omitted from the flowchart in Figure 7, or a step to display the image may be added to the flowchart instead of step S4. Alternatively, the projection device 100 may be configured to generate the first projection pattern image 91 after receiving instructions from the terminal device 300. Or, the terminal device 300 may send the first projection pattern image 91 it generated to the projection device 100 via the network 5, and the projection device 100 may project the first projection pattern image 91 received from the terminal device 300 onto the screen 30.

[0056] When the projection device 100 displays the first projection pattern image 91 on the screen 30, the user adjusts the imaging position and presses the imaging button provided on the terminal device 300.

[0057] As shown in Figure 10, the first projection pattern image 91 includes a checkerboard pattern 91A. The checkerboard pattern 91A is an image in which white and black rectangular shapes are arranged alternately. The area of ​​the first projection pattern image 91 other than the checkerboard pattern 91A is a white image area. This area is called the white area 91B. In the initial state, the installation status of the projection device 100 is unknown. The projection device 100 assumes that the central area of ​​the first projection pattern image 91 will be displayed somewhere on the screen 30, and therefore displays the first projection pattern image 91, which includes a relatively small checkerboard pattern 91A, on the screen 30.

[0058] The control unit 350 acquires the captured image taken by the camera 330, performs image analysis on the captured image, and detects the first projection pattern image 91 included in the captured image (step S5). The control unit 350 detects the checker pattern 91A included in the captured image and determines the position of the checker pattern 91A in the captured image. By determining the position of the checker pattern 91A in the captured image, the control unit 350 establishes a correspondence between the image capture coordinate system, which is the coordinate system set for the captured image, and the panel coordinate system, which is the coordinate system of the liquid crystal panel 230.

[0059] Next, the control unit 350 detects the screen frame 40 based on the contrast ratio (step S6). A white area 91B is formed around the checker pattern 91A. The control unit 350 performs image analysis from the center toward the edges of the first projection pattern image 91 included in the captured image and detects the position where the contrast ratio is greater than or equal to a threshold as the screen frame 40. The control unit 350 converts the detected coordinates of the imaging coordinate system to coordinates of the panel coordinate system to determine the coordinates of the screen frame 40 in the panel coordinate system.

[0060] Next, the control unit 350 instructs the projection device 100 to project the second projection pattern image 93 (step S7). Following the instructions of the terminal device 300, the projection device 100 reads the image data that will be the basis for the second projection pattern image 93 from the storage unit 180 and outputs it to the image processing unit 120. The image processing unit 120 performs predetermined image processing on the input image data and outputs the processed image data to the projector 200. The projector 200 generates image light based on the input image data and projects the generated image light onto the screen 30.

[0061] The second projection pattern image 93 may be projected not by instructions from the terminal device 300, but in response to operations on the operation panel of the projection device 100 or operations on the OSD menu using the remote control. In this case, the control unit 350 may display an image on the touch panel 320 prompting the user to project the second projection pattern image 93 on the projection device 100 by operations on the operation panel of the projection device 100 or operations on the OSD menu using the remote control. In this case, step S7 may be omitted from the flowchart in Figure 7, or a step to display the image may be added to the flowchart instead of step S7. Also, as with step S4, the projection device 100 may be configured to generate the second projection pattern image 93 after receiving instructions from the terminal device 300. Alternatively, the terminal device 300 may send the second projection pattern image 93 generated by the terminal device 300 to the projection device 100 via the network 5, and the projection device 100 may project the first projection pattern image 91 received from the terminal device 300 onto the screen 30.

[0062] Figure 11 shows an example of a second projection pattern image 93 displayed on the screen 30. The second projection pattern image 93, like the first projection pattern image 91, includes a checkerboard pattern 93A. In this embodiment, the size of the white and black rectangles constituting the checkerboard pattern 93A is the same as the size of the white and black rectangles constituting the checkerboard pattern 91A. The second projection pattern image 93 displays the checkerboard pattern 93A on the screen 30 at a large size so that the distance between the checkerboard pattern 93A and the screen frame 40 is as close as possible. That is, the display range of the checkerboard pattern 93A in the second projection pattern image 93 is larger than the display range of the checkerboard pattern 91A in the first projection pattern image 91. By reducing the distance between the outer edge of the checkerboard pattern 93A and the screen frame 40, the checkerboard pattern around the screen frame 40 can be utilized, suppressing the effects of distortion from the imaging lens and improving the accuracy of converting the position of the screen frame 40 in the imaging coordinate system to its position in the panel coordinate system. In this embodiment, the display range of the checker pattern 93A is determined based on the position of the screen frame 40 in the panel coordinate system identified based on the checker pattern 91A. Geometric correction based on the correspondence between the imaging coordinate system and the panel coordinate system identified based on the checker pattern 91A may or may not be performed on the checker pattern 93A.

[0063] When the second projection pattern image 93 is displayed on the screen 30 by the projection device 100, the user adjusts the imaging position and presses the imaging button provided on the terminal device 300.

[0064] The control unit 350 acquires the captured image taken by the camera 330, performs image analysis on the captured image, and detects the second projection pattern image 93 included in the captured image (step S8). The control unit 350 detects the checker pattern 93A included in the captured image and determines the position of the checker pattern 93A in the captured image. Then, the control unit 350 re-establishes the correspondence between the image coordinate system, which is the coordinate system set for the captured image, and the panel coordinate system, which is the coordinate system of the liquid crystal panel 230.

[0065] Similarly, the control unit 350 performs a detection process to detect the screen frame 40 from the captured image (step S9). The control unit 350 performs image analysis from the center toward the edges of the second projection pattern image 93 included in the captured image and detects the position where the contrast ratio is greater than or equal to a threshold as the screen frame 40.

[0066] Next, the control unit 350 determines the number of frame sides of the detected screen frame 40 (step S10). Here, even if the control unit 350 detects that the screen frame 40 has four frame sides, if radio button 325C is selected in step S3, it is determined that the screen frame 40 does not have four frame sides. In other words, the determination in step S10 is affirmative only when radio button 325D is selected in step S3 and the control unit 350 detects four screen frames 40.

[0067] If the detected screen frame 40 has four sides (step S11 / YES), the control unit 350 generates correction parameters to correct the position and shape of the projection area 70 so that the projection area 70 fits within the detected screen frame 40 (step S15).

[0068] Subsequently, the control unit 350 transmits the generated correction parameters to the projection device 100 via the wireless communication interface 310 (step S16). When the projection device 100 receives the correction parameters via the wireless communication interface 110, the image processing unit 120 corrects the image data based on the received correction parameters and outputs the corrected image data to the projector 200. The projector 200 generates image light based on the input image data and projects the generated image light onto the screen 30.

[0069] Furthermore, if the number of frame sides of the detected screen frame 40 is not four (step S11 / NO), the control unit 350 determines whether or not one or more screen frames 40 have been detected by the screen frame 40 detection process performed in step S9 (step S12). If the control unit 350 detects one or more screen frames 40 through the detection process (step S12 / YES), it executes a missing edge completion process to complete the missing edges of the screen frames 40 that could not be detected (step S13).

[0070] Here, if radio button 325C is selected in step S3, the control unit 350 invalidates the detection result of the top edge 510 and fills in the missing top edge 510. Subsequently, the control unit 350 generates correction parameters to correct the position and shape of the projection area 70 so that it fits within the four screen frames 40 detected and filled in by the detection process and the missing edge completion process (step S15). The correction parameters generated here correspond to an example of the first correction value. Also, the edges other than the top edge 510 correspond to the second edge.

[0071] Furthermore, if radio button 325D is selected in step S3 and the top edge 510 is detected, the detection result for the top edge 510 is not invalidated. If there are any edges other than the top edge 510 that have not been detected, the control unit 350 performs a missing edge completion process to complete the missing edges. Subsequently, the control unit 350 generates correction parameters to correct the position and shape of the projection area 70 so that it fits within the four screen frames 40 detected and completed by the detection process and the missing edge completion process (step S15). The correction parameters generated here correspond to an example of a second correction value. The top edge 510 corresponds to the first edge, and the edges other than the top edge 510 correspond to the second edge.

[0072] Subsequently, the control unit 350 transmits the generated correction parameters to the projection device 100 via the wireless communication interface 310 (step S16).

[0073] Furthermore, if the control unit 350 fails to detect any screen frames 40 during the detection process performed in step S9 (step S12 / NO), it proceeds to step S14. In step S14, the control unit 350 generates correction parameters to correct the shape of the projection area 70 from the output of the 3-axis acceleration sensor 340 and the normal vector of the screen 30. Subsequently, the control unit 350 transmits the generated correction parameters to the projection device 100 (step S16).

[0074] [5. Behavior when only the bottom edge of the screen frame is detected] Figure 12 is a flowchart showing the processing operation of the missing edge interpolation process when only the lower edge 43 of the four edges constituting the screen frame 40 is detected from the captured image. Next, we will explain the operation of the control unit 350 when only the lower edge 43 of the screen frame 40 is detected. Figures 13 to 17 are explanatory diagrams for explaining the missing edge completion process, and show the normalized panel coordinate system obtained by normalizing the panel coordinate system of the liquid crystal panel 230. The rectangular shapes shown in Figures 13 to 17 represent the panel region 500 of the liquid crystal panel 230 in the normalized panel coordinate system.

[0075] Figure 13 shows the upper edge 501 and lower edge 530 of the screen frame 40 detected from the captured image, and the normal vector N of the screen 30. The upper edge 501 and lower edge 530 represent the upper and lower edges in the normalized panel coordinate system. Here, the upper edge 501 is data in which the detection light irradiation device 50 was mistakenly detected as part of the upper edge of the screen 30. First, the control unit 350 calculates the normal vector of the screen 30. Let the calculated normal vector of the screen 30 be N(nx,ny,nz) (step SA131).

[0076] Next, the control unit 350 substitutes the coordinate values ​​of the lower edge 530 detected in step S9 shown in Figure 7 into the linear equation ax+by+c=0 representing the lower edge 530 to obtain the value of the normal vector L(a,b,c) of the lower edge 530 (step SA132).

[0077] Figure 14 shows the LCD panel edge line 503. If radio button 325C is selected in step S2, the control unit 350 invalidates the detection result of the top edge 501. The control unit 350 invalidates the detection result of the top edge 501 and identifies the LCD panel edge line 503 in the normalized panel coordinate system (step SA133).

[0078] Figure 15 shows the horizontal vanishing point VH. Next, the control unit 350 calculates the cross product of the normal vector N(nx,ny,nz) and the normal vector L(a,b,c) of the lower side 530 to determine the coordinates (VHx,VHy,VHz) of the horizontal vanishing point VH (step SA134). In Figure 15, the side obtained by extending the lower side 530 to the horizontal vanishing point VH is shown as side 531.

[0079] Next, the control unit 350 identifies the top edge 510 (step SA135). The control unit 350 identifies the innermost straight line among the group of straight lines passing through the LCD panel edge line 503 and the horizontal vanishing point VH as the top edge 510 (step SA135). The control unit 350 identifies the straight line passing through the left end of the LCD panel edge line 503 as the top edge 510 in the drawing view.

[0080] Figure 16 shows the vertical vanishing point VV. Next, the control unit 350 calculates the cross product of the normal vector N(nx,ny,nz) and the horizontal vanishing point VH(VHx,VHy,VHz) to determine the coordinates of the vertical vanishing point VV (VVx,VVy,VVz) (step SA136).

[0081] Next, the control unit 350 identifies the innermost line among the group of lines passing through the vertical vanishing point VV and the LCD panel edge line 503 as the left-hand side 540 (step SA137). The control unit 350 identifies the line passing through the left end of the LCD panel edge line 503 as the left-hand side 540, as viewed from the drawing.

[0082] Next, the control unit 350 identifies the innermost line among the group of lines passing through the vertical vanishing point VV and the LCD panel edge line 503 as the right-hand side 520 (step SA138). The control unit 350 identifies the line passing through the right end of the LCD panel edge line 503 as the right-hand side 520, as viewed from the drawing.

[0083] Next, the control unit 350 recalculates the positions of the four vertices that make up the screen frame 40 (step SA139). The control unit 350 executes step SA139 in order to fit the corrected shape within the range (maximum resolution) of the panel 500. In other words, the control unit 350 executes step SA139 to improve the accuracy of fitting the corrected shape within the range of the panel 500, because if only the innermost straight line is identified in step SA135, etc., there is a risk that the shape will extend beyond the range of the panel 500 due to calculation errors, etc.

[0084] The control unit 350 determines the four intersection points of the identified top edge 510, right edge 520, bottom edge 530, and left edge 540 as the four vertices q0a, q0b, q0c, and q0d of the screen frame 40. As shown in Figure 16, the four vertices q0a, q0b, q0c, and q0d of the screen frame 40 may extend beyond the projection area 70. Therefore, the control unit 350 adjusts the positions of the four vertices q0a, q0b, q0c, and q0d so that the extending vertices are located within the panel area 500. In the example shown in Figure 17, since vertices q0c and q0d are outside the panel area 500, the positions of vertices q0c and q0d are moved along the bottom edge 530 to the positions of vertices q1c and q1d. For example, the control unit 350 may perform the step of identifying the innermost straight line in step SA135 as the upper edge 510 in step SA139.

[0085] Figure 17 shows the lines drawn from the horizontal vanishing point VH and the vertical vanishing point VV to the four adjusted vertices q0a, q0b, q1c, and q1d. Next, the control unit 350 draws straight lines toward the four adjusted vertices q0a, q0b, q1c, and q1d. The line passing through the vertical vanishing point VV and vertex q1c is denoted as 521 on the right-hand side. The line passing through the vertical vanishing point VV and vertex q1d is denoted as 541 on the left side. Let vertex q1b be the intersection of the right-hand side 521 and the top side 510. Also, let vertex q1a be the intersection of the left-hand side 541 and the top side 510.

[0086] The control unit 350 identifies the four vertices q1a, q1b, q1c, and q1d as the frame vertices after interpolation. That is, the region demarcated by the top edge 510, the right edge 521, the bottom edge 530, and the left edge 541 is identified as the screen frame 40.

[0087] [6. Action taken when the top and bottom edges of the screen frame are detected] Figure 18 is a flowchart showing the processing operation of the missing edge completion process when the top edge 510 and bottom edge 530 are detected among the four sides that make up the screen frame 40. Figures 19 to 21 are explanatory diagrams for explaining the missing edge completion process, and show the normalized panel coordinate system obtained by normalizing the panel coordinate system of the liquid crystal panel 230. The rectangular shapes shown in Figures 19 to 21 represent the panel region 500 of the liquid crystal panel 230 in the normalized panel coordinate system.

[0088] Figure 19 shows the upper and lower edges 510 and 530 of the screen frame 40 detected from the captured image, and the normal vector N of the screen 30. The upper and lower edges 510 and 530 are the upper and lower edges 501 and 530 in the normalized panel coordinate system. First, the control unit 350 calculates the normal vector of the screen 30. Let the calculated normal vector of the screen 30 be N(nx,ny,nz) (step SB131).

[0089] Next, the control unit 350 substitutes the coordinate values ​​of the upper edge 510 detected in step S9 shown in Figure 7 into the linear equation a1x+b1y+c1=0 which represents the upper edge 510, to obtain the value of the normal vector L1(a1,b1,c1) of the upper edge 510 (step SB132).

[0090] Next, the control unit 350 substitutes the coordinate values ​​of the lower edge 530 detected in step S9 shown in Figure 7 into the linear equation a2x+b2y+c2=0 which represents the lower edge 530, to obtain the value of the normal vector L2(a2,b2,c2) of the lower edge 530 (step SB133).

[0091] Figure 20 shows the horizontal vanishing point VH. Next, the control unit 350 determines the coordinates of the horizontal vanishing point VH. The control unit 350 finds the vertical vanishing point VV, which is the intersection of the equation of the upper side 510 calculated in step S131B and the equation of the lower side 530 calculated in step S132B (step SB133). Specifically, it finds the coordinates of the horizontal vanishing point VH from the cross product of the normal vector L1(a1,b1,c1) of the upper side 510 and the normal vector L2(a2,b2,c2) of the lower side 530. The normalized panel coordinate system utilizes the property that the coordinates of the horizontal vanishing point VH can be found by finding the cross product of the normal vectors of the lines.

[0092] Figure 21 shows the vertical vanishing point VV. Next, the control unit 350 determines the vertical vanishing point VV (step SB134). The control unit 350 determines the vertical vanishing point VV from the cross product of the normal vector N(nx,ny,nz) of the screen 30 and the normal vector L2(a2,b2,c2) of the lower edge 530.

[0093] Next, the control unit 350 recalculates the positions of the four vertices of the projection area 70, similar to step SA139 in the flowchart shown in Figure 12. The control unit 350 executes step SA139 to fit the corrected shape within the range (maximum resolution) of the panel 500. In other words, the control unit 350 executes step SA139 to improve the accuracy of fitting the corrected shape within the range of the panel 500, because simply identifying the innermost straight line in step SA135, etc., may result in the shape exceeding the range of the panel 500 due to calculation errors, etc. This processing operation is the same as step SA139 explained with reference to Figure 17, so a detailed explanation is omitted. Here again, for example, the control unit 350 may perform the process of identifying the innermost straight line in step SA135 as the top edge 510 together in step SA139.

[0094] [7. Action taken when the top and left edges of the screen frame are detected] Figure 22 is a flowchart showing the processing operation of the missing edge completion process when the top edge 510 and left edge 540 are detected among the four sides that make up the screen frame 40. Figures 23 to 27 are explanatory diagrams for explaining the missing edge completion process, and show the normalized panel coordinate system obtained by normalizing the panel coordinate system of the liquid crystal panel 230. The rectangular shapes shown in Figures 23 to 27 represent the panel region 500 of the liquid crystal panel 230 in the normalized panel coordinate system.

[0095] Figure 23 shows the normal vector N(nx,ny,nz) of the screen 30 and the upper edge 510 and left edge 540 of the detected screen frame 40. First, the control unit 350 calculates the normal vector of the screen 30. Let the calculated normal vector of the screen 30 be N(nx,ny,nz) (step SC131).

[0096] Next, the control unit 350 substitutes the coordinate values ​​of the upper edge 510 detected in step S9 shown in Figure 7 into the linear equation a1x+b1y+c1=0 which represents the upper edge 510, to obtain the value of the normal vector L1(a1,b1,c1) of the upper edge 510 (step SC132).

[0097] Next, the control unit 350 substitutes the coordinate values ​​of the left-hand side 540 detected in step S9 shown in Figure 7 into the linear equation a3x+b3y+c3=0 which represents the left-hand side 540, to obtain the value of the normal vector L3(a3,b3,c3) of the left-hand side 540 (step SC133).

[0098] Figure 24 shows the coordinates of the vertical vanishing point VV. Next, the control unit 350 determines the coordinates of the vertical vanishing point VV (step SC134). The control unit 350 calculates the coordinates of the vertical vanishing point VV (VVx, VVy, VVz) by finding the cross product of the normal vector L3(a3, b3, c3) of the left-hand side 540 and the normal vector N(nx, ny, nz) of the screen 30.

[0099] Figure 25 shows the right-hand side 520 of the screen frame 40. Next, the control unit 350 identifies the right-hand side 520 of the screen frame 40 (step SC135). The control unit 350 identifies the innermost straight line among the group of straight lines passing through the vertical vanishing point VV determined in step SC134 and the top side 510 as the right-hand side 520. The control unit 350 identifies the straight line passing through the right end of the top side 510 as the right-hand side 520 in the drawing view.

[0100] Figure 26 shows the horizontal vanishing point VH. Next, the control unit 350 determines the coordinates of the horizontal vanishing point VH (step SC136). The control unit 350 calculates the coordinates (VHx, VHy, VHz) of the horizontal vanishing point VH by calculating the cross product of the normal vector L1(a1, b1, c1) of the upper edge 510 and the normal vector N(nx, ny, nz) of the screen 30.

[0101] Figure 27 shows the bottom edge of the screen frame 40. Next, the control unit 350 identifies the bottom edge 530 of the screen frame 40 (step SC137). The control unit 350 identifies the bottom edge 530 as the innermost straight line among the straight lines passing through the horizontal vanishing point VH determined in step SC135C and the left edge 540.

[0102] After this, the control unit 350 recalculates the positions of the four vertices of the projection area 70, similar to step SA139 in the flowchart shown in Figure 12. The control unit 350 executes step SA139 to fit the corrected shape within the range (maximum resolution) of the panel 500. In other words, the control unit 350 executes step SA139 to improve the accuracy of fitting the corrected shape within the range of the panel 500, because if only the innermost straight line in step SA135, etc., there is a risk that the shape will extend beyond the range of the panel 500 due to calculation errors, etc. This processing operation is the same as step SA139 explained with reference to Figure 17, so a detailed explanation is omitted. Here, for example, the control unit 350 may perform the process of identifying the innermost straight line in step SA135 as the top edge 510 together in step SA139.

[0103] The flowchart in Figure 22 explains the case where the top edge 510 and left edge 540 of the screen 30 are detected, but the same process can be applied when the bottom edge 530 and right edge 520 of the screen 30 are detected. The changes in the processing operation when the bottom edge 530 and right edge 520 are detected are shown below. In step SC133, the process of finding the normal vector of the linear equation representing the right-hand side is executed. Step SC134 executes the process of finding the coordinates (VHx, VHy, VHz) of the horizontal vanishing point VH. The coordinates (VHx, VHy, VHz) of the horizontal vanishing point VH are found by calculating the cross product of the normal vector of 520 on the right side and the normal vector of screen 30. In step SC135, the process of identifying the left-hand side is performed. The left-hand side 540 is identified from the group of lines passing through the horizontal vanishing point VH and the lower edge 530. In step SC136, the process of finding the coordinates (VVx, VVy, VVz) of the vertical vanishing point VV is executed. The coordinates (VVx, VVy, VVz) of the vertical vanishing point VV are found by calculating the cross product of the normal vector of the bottom edge 530 and the normal vector of the screen 30. In step SC137, the process of identifying the upper edge 510 is performed. The upper edge 510 is identified from the set of lines passing through the vertical vanishing point VV and the right edge 520.

[0104] [8. Detailed flow of step S14] Figure 28 is a flowchart detailing step S14 of the flowchart shown in Figure 7. The details of step S14 will be explained with reference to the flowchart shown in Figure 28.

[0105] Figure 29 shows the output of the 3-axis accelerometer, the normal vector of the screen 30, and the vertical and horizontal vanishing points. The XYZ axes shown in Figure 29 represent three mutually orthogonal axes, with the Y axis parallel to the vertical direction, and the X and Z axes each parallel to the horizontal direction. First, the control unit 350 calculates a coordinate transformation matrix to convert the normalized panel coordinate system to the screen coordinate system (step S141). The coordinate transformation matrix is ​​a matrix whose components are the horizontal vanishing point, the vertical vanishing point, and the normal vector of the screen 30. Here, the property that the cross product of the gravity direction detected by the 3-axis accelerometer and the normal vector of the screen 30 becomes the X-axis direction of the roll-compensated screen coordinate system is utilized.

[0106] Let N(nx,ny,nz) be the normal vector of screen 30, and let (gx,gy,gz) be the gravity vector output by the 3-axis accelerometer. The normal vector (nx,ny,nz) of the screen 30 and the gravity direction vector (gx,gy,gz) are values ​​in the local coordinate system of the projection device 100, with the center of the projection lens of the projection device 100 as the origin, and composed of the optical axis of the projection lens and two axes orthogonal to this optical axis. The gravity direction vector (gx,gy,gz) output by the 3-axis accelerometer is output as a value in the local coordinate system of the 3-axis accelerometer, but the value of the gravity direction vector (gx,gy,gz) is converted to the local coordinate system of the projection device 100 based on the mounting position relationship between the 3-axis accelerometer and the projection lens.

[0107] The horizontal vanishing point in the homogeneous normalized coordinate system is obtained by the cross product of the normal vector N(nx,ny,nz) of the screen 30 and the output (gx,gy,gz) of the 3-axis accelerometer using the following equation (1).

[0108]

number

[0109] Furthermore, the vertical vanishing point in the homogeneous normal coordinate system is obtained by the cross product of the normal vector N(nx,ny,nz) of screen 30 and the horizontal vanishing point in the homogeneous normalized coordinate system obtained in equation (1), as shown in equation (2).

[0110]

number

[0111] Since the matrix whose components are the horizontal vanishing point, the vertical vanishing point, and the normal vector of screen 30 functions as a coordinate transformation matrix that transforms the normalized panel coordinate system to the screen coordinate system, the coordinate transformation matrix is ​​given by the following equation (3).

[0112]

number

[0113] Next, the control unit 350 uses the above coordinate transformation example (3) to transform the coordinates of the four vertices of the liquid crystal panel 230 into the coordinates of the four vertices of the screen coordinate system (step S142). Figure 30 shows the four vertices of the liquid crystal panel 230 in the normalized panel coordinate system. The coordinates of the four vertices of the liquid crystal panel 230 in the normalized panel coordinate system are p0(c), p1(c), p2(c), and p3(c), respectively. The letter "c" in parentheses indicates that the coordinates are in the normalized panel coordinate system. Figure 31 shows the coordinates p0(s), p1(s), p2(s), and p3(s) of the four vertices in the transformed screen coordinate system. The letter "s" in parentheses indicates that the coordinates are in the screen coordinate system.

[0114] Next, the control unit 350 determines the coordinates of the intersection point Q of the diagonals of the coordinates of the four vertices in the transformed screen coordinate system (step S143). Figure 32 shows the intersection point Q of a rectangle 610 centered at the intersection point Q of its diagonals and a quadrilateral 620 composed of four vertices p0(s), p1(s), p2(s), and p3(s).

[0115] Next, the control unit 350 calculates a rectangular figure 610 with the aspect ratio of the liquid crystal panel 230 centered at the intersection point Q of the diagonals (step S144). Then, the control unit 350 enlarges the rectangular figure 610 while maintaining the aspect ratio and searches for the intersection point between the rectangular figure 610 and the quadrilateral 620. That is, the control unit 350 searches for the intersection point of the quadrilateral 620 closest to the intersection point Q of the diagonals.

[0116] Figure 33 shows the enlarged rectangular shape 610. Next, the control unit 350 enlarges the rectangular figure 610 and finds the intersection point with the quadrilateral 620 formed by the four vertices p0(s), p1(s), p2(s), and p3(s) (step S145). The control unit 350 enlarges the rectangular figure 610, which has the aspect ratio of the liquid crystal panel 230 centered on the found diagonal intersection point Q, while maintaining the aspect ratio, and finds the intersection point between the rectangular figure 610 and the quadrilateral 620 formed by the four vertices p0(s), p1(s), p2(s), and p3(s). The found intersection point is designated as intersection point w3(s).

[0117] Figure 34 shows the enlarged rectangular shape 610. When the control unit 350 detects the intersection point w3(s) with the quadrilateral 620, it further enlarges the rectangular figure 610 while maintaining its aspect ratio. At this time, the control unit 350 enlarges the size of the rectangular figure 610 without changing the position of the previously detected intersection point w3(s) with the quadrilateral 620. Then, the control unit 350 finds another intersection point between the enlarged rectangular figure 610 and the quadrilateral 620. This intersection point is called w2(s).

[0118] Furthermore, the control unit 350 designates the vertices of the rectangular figure 610 other than the intersection points w3(s) and w2(s) as w0(s) and w1(s), respectively. The control unit 350 then uses the inverse matrix of the coordinate transformation row example (3) to transform the screen coordinate system coordinates of w0(s), w1(s), w2(s), and w3(s) into the normalized panel coordinate system coordinates (step S146).

[0119] [9. Other Embodiments] The embodiment described above is a preferred embodiment. However, it is not limited to the embodiment described above, and various modifications can be made without departing from the spirit of the invention. For example, in the embodiment described above, the control unit 350 of the terminal device 300 that executes APP363 is described as generating the correction parameters, but the projection device 100 may also generate the correction parameters. Furthermore, the camera may be the camera 155 provided by the projection device 100, the camera 330 provided by the terminal device 300, or a separate camera may be provided independently of the projection device 100 and the terminal device 300.

[0120] Furthermore, Figure 9 shows an inquiry image 325 that asks whether the detection light irradiation device 50 is located in the inner or outer region of the screen frame 40. However, the inquiry image 325 may also include guidance asking whether the detection light irradiation device 50 itself is present or absent.

[0121] Furthermore, the position of the detection light irradiation device 50 is not limited to the upper frame edge 41 of the screen frame 40, but may be located near the right frame edge 42, the lower frame edge 43, or the left frame edge 44, and may be configured to be located inside the screen 30.

[0122] Furthermore, the form of the inquiry image 325 is not limited to the embodiments described above. For example, the inquiry image 325 may be an image prompting the user to specify at least one of the top edge 41, the right edge 42, the bottom edge 43, or the left edge 44. In this case, the form of the inquiry image 325 is not particularly limited as long as at least one of the top edge 41, the right edge 42, the bottom edge 43, or the left edge 44 can be specified.

[0123] Furthermore, the configurations of the projection device 100 shown in Figure 3 and the terminal device 300 shown in Figure 5 represent functional configurations, and the specific implementation form is not particularly limited. In other words, it is not necessarily required that hardware corresponding to each functional unit be implemented individually, and it is certainly possible to have a configuration in which a single processor executes a program to realize the functions of multiple functional units. Also, some of the functions realized by software in the above embodiment may be realized by hardware, or some of the functions realized by hardware may be realized by software. In addition, the specific detailed configurations of other parts of the projection device 100 and the terminal device 300 can also be arbitrarily changed without departing from the intent.

[0124] Furthermore, the processing units in the flowcharts shown in Figures 7, 12, 18, 22, and 28 are divided according to their main processing content to facilitate understanding of the processing of the control unit 700. The way the processing units are divided and the names of the processing units shown in the flowcharts in Figures 7, 12, 18, 22, and 28 do not limit the process; it is possible to further divide the processing into more processing units depending on the processing content, or to divide it so that one processing unit contains even more processing. Also, the processing order in the flowcharts described above is not limited to the examples shown.

[0125] Furthermore, the control method for the terminal device 300 can be implemented by having the processor 370 of the terminal device 300 execute a control program corresponding to the control method for the terminal device 300. This control program can also be recorded on a recording medium that is readable by a computer. Magnetic, optical, or semiconductor memory devices can be used as the recording medium. Specifically, portable or fixed recording media such as flexible disks, HDDs (Hard Disk Drives), CD-ROMs (Compact Disk Read Only Memory), DVDs (Digital Versatile Discs), Blu-ray® Discs, magneto-optical disks, flash memory, and card-type recording media can be used. The recording medium may also be a non-volatile storage device such as RAM, ROM, or HDD, which is an internal storage device of the projection device 100. Alternatively, the control method for the terminal device 300 can be implemented by storing a control program corresponding to the display method of the projection device 100 in a server device, and downloading the control program from the server device to the projection device 100.

[0126] [10. Summary of this disclosure] A summary of this disclosure is provided below.

[0127] (Note 1) A control method for a terminal device comprising a camera that captures an image of a projection target onto which an image is projected from a projection device, a display, and one or more processors, wherein the one or more processors are instructed to display on the display an inquiry image asking whether an object different from the projection target is located in an area inside or outside the projection target beyond the first edge of the projection target; to acquire an image of the projection target captured by the camera; to calculate a first correction value for correcting at least one of the shape or position of the image based on a second edge of the projection target different from the first edge in the image, when a first response indicating that the object is located inside the first edge is received; and to calculate a second correction value for correcting at least one of the shape or position of the image based on both the first and second edges in the image, when a second response indicating that the object is located outside the projection target beyond the first edge is received.

[0128] According to the control method for the terminal device described in Appendix 1, if a first response is received indicating that the object is in the area inside the first edge, a first correction value is calculated to correct at least one of the shape or position of the image based on the second edge of the projection target. If a second response is received indicating that the object is outside the projection target beyond the first edge, a first correction value is calculated to correct at least one of the shape or position of the image based on both the first and second edges of the projection target. Therefore, when a first response is received, a first correction value is calculated to correct at least one of the shape or position of the image based on the second edge of the captured image, making it possible to calculate an appropriate correction value that reduces the influence of the object.

[0129] (Note 2) The control method for the terminal device described in Appendix 1, wherein the object is a detection device for detecting an indicator that indicates the position of the projection target on the projection surface.

[0130] According to the control method for the terminal device described in Appendix 2, the object is a detection device for detecting an indicator that indicates the position of the projection target on the projection surface. Therefore, even when a detection device is used to detect the indicator, an appropriate correction value can be calculated.

[0131] (Note 3) The control method for a terminal device according to Appendix 1 or 2, wherein the inquiry image includes at least one schematic diagram illustrating the positional relationship between the object and one of the first and second sides.

[0132] According to the control method for the terminal device described in Appendix 3, the inquiry image includes at least one schematic diagram that simulates the positional relationship between the object and one of its first and second edges. This makes it easier to understand the content of the inquiry.

[0133] (Note 4) A control method for a terminal device as described in Appendix 1, further causing one or more processors to calculate the horizontal vanishing point by calculating the cross product of the normal vector of the right side, which is the third side and different from the first and second sides, and the normal vector of the projection surface, which is the projection target; calculate the vertical vanishing point by calculating the cross product of the normal vector of the bottom side, which is the second side, and the normal vector of the projection surface; identify a straight line that will be the left side, which is the fourth side, from among a group of straight lines passing through the horizontal vanishing point and the bottom side, which is the second side; and identify a straight line that will be the top side, which is the first side, from among a group of straight lines passing through the vertical vanishing point and the right side.

[0134] According to the control method for the terminal device described in Appendix 4, by identifying the bottom edge and the third edge (right edge) from the captured image, the other edges, the top edge and the left edge, can be identified. Therefore, even if the object is located in the area inside the projection surface, the four edges of the projection surface that are the target of the projection can be identified, and appropriate correction values ​​can be calculated.

[0135] (Note 5) A control method for a terminal device as described in Appendix 1, further causing one or more processors to calculate the vertical vanishing point by calculating the cross product of the normal vector of the left side, which is the fourth side different from the first and second sides, and the normal vector of the projection surface, which is the projection target; calculate the horizontal vanishing point by calculating the cross product of the normal vector of the top side, which is the first side, and the normal vector of the projection surface; identify a straight line that will be the right side, which is the third side, from among a group of straight lines passing through the vertical vanishing point and the top side; and identify a straight line that will be the bottom side, which is the second side, from among a group of straight lines passing through the horizontal vanishing point and the left side.

[0136] According to the control method for the terminal device described in Appendix 5, by identifying the top edge and the fourth edge (left edge) from the captured image, the other edges, the bottom edge and the right edge, can be identified. Therefore, even if the object is located in the area inside the projection surface, the four edges of the projection surface that are the target of the projection can be identified, and appropriate correction values ​​can be calculated.

[0137] (Note 6) A program that causes a computer to control a terminal device comprising a camera that images a projection target onto which an image is projected from a projection device, and a display, to display on the display an inquiry image asking whether an object different from the projection target is located in an area inside or outside the projection target beyond the first edge of the projection target; to acquire an image of the projection target captured by the camera; to calculate a first correction value for correcting at least one of the shape or position of the image based on the second edge of the projection target, which is different from the first edge in the image, when a first response is received indicating that the object is located inside the area inside the first edge; and to calculate a second correction value for correcting at least one of the shape or position of the image based on both the first and second edges in the image, when a second response is received indicating that the object is located outside the projection target beyond the first edge.

[0138] According to the program described in Appendix 6, if a first response is received indicating that the object is located within the area inside the first edge, a first correction value is calculated to correct at least one of the shape or position of the image based on the second edge of the projection target. If a second response is received indicating that the object is located outside the projection target beyond the first edge, a first correction value is calculated to correct at least one of the shape or position of the image based on both the first and second edges of the projection target. Therefore, when a first response is received, a first correction value is calculated to correct at least one of the shape or position of the image based on the second edge of the captured image, thereby enabling the calculation of an appropriate correction value that reduces the influence of the object.

[0139] (Note 7) A terminal device comprising a projection device that projects an image onto a projection target and includes a receiving circuit for receiving data, a camera for capturing images of the projection target, a display, one or more processors, and a transmitting circuit for transmitting the data, wherein the one or more processors cause the display to show a query image asking whether an object different from the projection target is in an area inside or outside the projection target beyond the first edge of the projection target, acquire an image of the projection target captured by the camera, and, upon receiving a first response indicating that the object is in an area inside the first edge, the terminal device includes the projection target different from the first edge in the image. A projection system that performs the following: calculates a first correction value for correcting at least one of the shape or position of the image based on a second edge; when a second response is received indicating that the object is outside the projection target beyond the first edge, calculates a second correction value for correcting at least one of the shape or position of the image based on both the first and second edges in the captured image; transmits the first correction value or the second correction value to the projection device via the transmission circuit; and the projection device projects the corrected image, which has been corrected based on the first correction value or the second correction value received by the reception circuit, onto the projection target.

[0140] According to the projection system described in Appendix 7, if a first response is received indicating that the object is located within the area inside the first edge, a first correction value is calculated to correct at least one of the shape or position of the image based on the second edge of the projection target. If a second response is received indicating that the object is located outside the projection target beyond the first edge, a first correction value is calculated to correct at least one of the shape or position of the image based on both the first and second edges of the projection target. Therefore, when a first response is received, a first correction value is calculated to correct at least one of the shape or position of the image based on the second edge of the captured image, thereby enabling the calculation of an appropriate correction value that reduces the influence of the object. [Explanation of symbols]

[0141] 1…Projection system, 5…Network, 10…Indicator, 30…Screen, 31…Clean frame, 33…Screen plate, 35…Support member, 40…Screen frame, 41…Upper frame edge, 43…Lower frame edge, 50…Detection light irradiation device, 51…Ejection unit, 53…Adjustment mechanism, 55…Detection light, 57…Reflection detection light, 70…Projection area, 80…Guide image, 81…Upper guide edge, 91…First projection pattern image, 91A…Checker pattern Turn, 91B...White area, 93...Second projection pattern image, 93A...Checker pattern, 100...Projection device, 110...Wireless communication I / F, 120...Image processing unit, 125...Frame memory, 130...Remote control light receiver, 135...Remote control, 140...Short-range wireless communication I / F, 150...Imaging unit, 155...Camera, 160...Position detection unit, 170...Control unit, 180...Storage unit, 185...Control program, 19 0…Processor, 200…Projector, 210…Light source, 230, 230R, 230G, 230B…LCD panel, 250…Optical unit, 270…Panel drive unit, 300…Terminal device, 310…Wireless communication I / F, 320…Touch panel, 325…Inquiry image, 325A…First schematic diagram, 325B…Second schematic diagram, 325C, 325D…Radio buttons, 325E…Confirm button, 330…Camera 340...3-axis accelerometer, 350...control unit, 360...memory unit, 363...APP, 370...processor, 500...panel area, 501...top edge, 503...LCD panel edge line, 510...top edge, 520...right edge, 530...bottom edge, 540...left edge, 610...rectangle shape, 620...quadrilateral, 700...control unit, L, L1, L2, L3...normal vector, N...normal vector, VH...horizontal vanishing point, VV...vertical vanishing point.

Claims

1. A control method for a terminal device comprising a camera that captures an image of a projection target from which an image is projected by a projection device, a display, and one or more processors, The one or more processors: The display is shown a query image asking whether an object different from the projection target is located inside or outside the area of ​​the projection target beyond the first edge of the projection target. The camera captures an image of the projection target, and the camera captures the image of the projection target. When a first response is received indicating that the object is in the region inside the first side, a first correction value is calculated to correct at least one of the shape or position of the image based on the second side of the projection target which is different from the first side in the captured image, When a second response is received indicating that the object is outside the projection target beyond the first side, a second correction value is calculated to correct at least one of the shape or position of the image based on both the first side and the second side of the captured image, A method for controlling a terminal device that executes an action.

2. The control method for a terminal device according to claim 1, wherein the object is a detection device for detecting an indicator that indicates the position of the projection target on the projection surface.

3. The control method for a terminal device according to claim 1, wherein the inquiry image includes at least one schematic diagram illustrating the positional relationship between the object and one of the first and second sides.

4. A computer controls a terminal device that includes a camera that captures the projection target from which an image is projected, and a display. The display is shown a query image asking whether an object different from the projection target is located inside or outside the area of ​​the projection target beyond the first edge of the projection target. The camera captures an image of the projection target, and the camera captures the image of the projection target. When a first response is received indicating that the object is in the region inside the first side, a first correction value is calculated to correct at least one of the shape or position of the image based on the second side of the projection target which is different from the first side in the captured image, When a second response is received indicating that the object is outside the projection target beyond the first side, a second correction value is calculated to correct at least one of the shape or position of the image based on both the first side and the second side of the captured image, A program to be executed.

5. A projection device that includes a receiving circuit for receiving data and projects an image onto a projection target, A terminal device comprising a camera for capturing images of the projection target, a display, one or more processors, and a transmission circuit for transmitting the data, Equipped with, The one or more processors described above The display is shown a query image asking whether an object different from the projection target is located inside or outside the area of ​​the projection target beyond the first edge of the projection target. The camera captures an image of the projection target, and the camera captures the image of the projection target. When a first response is received indicating that the object is in the region inside the first side, a first correction value is calculated to correct at least one of the shape or position of the image based on the second side of the projection target which is different from the first side in the captured image, When a second response is received indicating that the object is outside the projection target beyond the first side, a second correction value is calculated to correct at least one of the shape or position of the image based on both the first side and the second side of the captured image, The transmission circuit transmits the first correction value or the second correction value to the projection device. Execute, The projection device, Projecting the corrected image, which has been corrected based on the first or second correction value received by the receiving circuit, onto the projection target. A projection system that performs this task.

Images