Calibration method, system and program product

Abstract

The invention provides a calibration method, a system and a program product to realize high-precision calibration of coordinate systems of a camera and a projector. The calibration method includes: projecting a first image including a phase shift pattern from a projector to a projection surface; acquiring a first captured image including the first image and captured by the camera; generating a mask image based on the first captured image; projecting a second image from the projector to the projection surface, the second image including the structured light pattern and being shaded by the mask image; acquiring a second captured image including the second image and captured by the camera; and associating the coordinates of the photographing coordinate system of the camera with the coordinates of the display coordinate system of the projector on the basis of the second photographed image.

Description

Technical Field

[0001] The present disclosure relates to a calibration method, a system, and a program product. Background Art

[0002] A projector that projects a display image onto a wall surface of a room is disclosed in Patent Document 1. Two side wall surfaces, a ceiling surface, and a floor surface are adjacent to the wall surface, respectively.

[0003] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2009-48015

[0004] In order for the projector to project an image onto the projection surface appropriately, it is necessary to determine how the coordinates in the display coordinate system of the projector correspond to the coordinates on the projection surface. However, conventionally, when there are other wall surfaces or the like near the outer edge of the projection surface, due to reflected light from the other wall surface to the projection surface, there has been a problem that it is difficult to measure this correspondence with high accuracy. Summary of the Invention

[0005] A calibration method according to one aspect of the present disclosure includes: projecting a first image including a phase shift pattern from a projector onto a projection surface; obtaining a first captured image including the first image and captured by a camera; generating a mask image based on the first captured image; projecting a second image from the projector onto the projection surface, where the second image includes a structured light pattern and is obtained by masking with the mask image; obtaining a second captured image including the second image and captured by the camera; and performing correspondence between the coordinates in the capture coordinate system of the camera and the coordinates in the display coordinate system of the projector based on the second captured image.

[0006] A system according to one aspect of the present disclosure includes: a camera; and a projector communicably connected to the camera, the projector performing the following processing: projecting a first image including a phase shift pattern onto a projection surface; obtaining a first captured image obtained by the camera capturing the first image; generating a mask image based on the first captured image; projecting a second image onto the projection surface, where the second image includes a structured light pattern and is obtained by masking with the mask image; obtaining a second captured image obtained by the camera capturing the second image; and performing correspondence between the coordinates in the capture coordinate system of the camera and the coordinates in the display coordinate system of the projector based on the second captured image.

[0007] A program product according to one aspect of the present disclosure causes a computer to perform the following processes: causing a first image including a phase shift pattern to be projected from a projector onto a projection surface; acquiring a first captured image by capturing the first image with a camera; generating a mask image based on the first captured image; causing a second image to be projected from the projector onto the projection surface, where the second image includes a structured light pattern and is obtained by masking with the mask image; acquiring a second captured image by capturing the second image with the camera; and corresponding coordinates in a capture coordinate system of the camera to coordinates in a display coordinate system of the projector based on the second captured image. Description of the Drawings

[0008] Figure 1 FIG. is a schematic diagram showing the system of the first embodiment.

[0009] Figure 2 FIG. is a block diagram of the projector used in the system of the first embodiment.

[0010] Figure 3 FIG. is a flowchart showing the process of the calibration method of the first embodiment.

[0011] Figure 4 FIG. is a diagram for explaining the first image.

[0012] Figure 5 FIG. is a diagram for explaining the captured image.

[0013] Figure 6 FIG. is a diagram for explaining the amplitude intensity image based on the first captured image.

[0014] Figure 7 FIG. is a diagram for explaining the determination of the mask shape.

[0015] Figure 8 FIG. is a diagram for explaining the mask image in the capture coordinate system.

[0016] Figure 9 FIG. is a diagram for explaining the mask image in the display coordinate system.

[0017] Figure 10 FIG. is a diagram for explaining the second image.

[0018] Figure 11 FIG. is a block diagram of the projector used in the system of the second embodiment.

[0019] Figure 12 FIG. is a flowchart showing the process of the calibration method of the second embodiment.

[0020] Figure 13 FIG. is a diagram for explaining the image for area designation.

[0021] Reference Signs

[0022] 10: Projector; 10A: Projector; 11: Storage device; 12: Processing device; 12a: Projection control unit; 12b: Shooting control unit; 12c: Generation unit; 12d: Generation unit; 13: Communication device; 14: Image processing circuit; 15: Optical device; 15a: Light source; 15b: Display panel; 15c: Optical system; 16: Operating device; 20: Camera; 30: Terminal device; 31: Display device; 100: System; 100A: System; CE: Top surface; D1: First shooting data; D2: Second shooting data; DA: Amplitude intensity information; DC: Corresponding information; DG1: First image information; DG2: Second image information; DM: Mask information; FL: Ground; G: Projected image; G1: First image; G1-1: First image; G1-2: First image; G1-3: First image; G1-4: First image; G2: Second image; GG: Shot image; GG1: First shot image; GG2: Amplitude intensity image; GG3: Mask shape image; GU: Image; IMG: Image data; L1: Line; L2: Line; L3: Outline; M: Mask image; M-C: Mask image; M-P: Mask image; OP: Transmissive area; PR1: Program; PR2: Program; PT: Phase shift pattern; RC: Area; RM: Non-transmissive area; RP: Area; S10: Step; S11: Step; S12: Step; S13: Step; S20: Step; S20A: Step; S21: Step; S22: Step; S22A: Step; S30: Step; S31: Step; S32: Step; S33: Step; S50: Step; S60: Step; SC: Projection surface; WA1: Wall surface; WA2: Wall surface; WA3: Wall surface. Detailed implementation manners

[0023] Hereinafter, preferred implementation manners of the present disclosure will be described with reference to the accompanying drawings. In addition, in the drawings, the sizes and scales of respective parts are appropriately different from the actual ones, and there are also parts schematically shown for easy understanding. In addition, the scope of the present disclosure is not limited to these manners as long as the gist of the present disclosure is not particularly limited in the following description.

[0024] 1. First implementation manner

[0025] 1-1. Outline of the system

[0026] Figure 1 It is a schematic diagram showing the outline of the system 100 of the first implementation manner. The system 100 is a projection system that projects a projected image G onto a projection surface SC. In Figure 1In [description], when observing from the direction perpendicular to the paper surface, the direction arranged in the order of the top surface CE, the wall surface WA1, and the floor surface FL is defined as the up-down direction. When observing from the wall surface WA1, the top surface CE side is defined as the upper side in the figure, and when observing from the wall surface WA1, the floor surface FL side is defined as the lower side in the figure. In Figure 1 In [description], when observing from the direction perpendicular to the paper surface, the direction arranged in the order of the wall surface WA2, the wall surface WA1, and the wall surface WA3 is defined as the left-right direction. When observing from the wall surface WA1, the wall surface WA2 side is defined as the left side in the figure, and when observing from the wall surface WA1, the wall surface WA3 side is defined as the right side in the figure.

[0027] The projection surface SC is the wall surface WA1. Adjacent to the left side in the figure of the wall surface WA1 is the wall surface WA2. On the other hand, adjacent to the right side in the figure of the wall surface WA1 is the wall surface WA3. In addition, the top surface CE is adjacent to the upper side in the figure of the wall surface WA1. On the other hand, the floor surface FL is adjacent to the lower side in the figure of the wall surface WA1. Thus, the wall surface WA1 is the surface surrounded by the wall surfaces WA2, WA3, the top surface CE, and the floor surface FL.

[0028] In addition, the projection surface SC is not limited to the form of the wall surface WA1. For example, it can also be the surface of an object such as a screen. Furthermore, the projection surface SC is not limited to a flat surface. For example, it can also be a surface that is concave or convexly curved.

[0029] As Figure 1 shown, the system 100 includes a projector 10, a camera 20, and a terminal device 30.

[0030] The projector 10 is a display device that projects the projection image G represented by the video data IMG output from the terminal device 30 onto the projection surface SC. In Figure 1 the example shown, the projection image G is projected onto a quadrilateral area that covers substantially the entire area of the projection surface SC. In addition, the projector 10 can project the projection image G onto the area RP that includes the projection surface SC. In addition, in Figure 1 [description], the projection image G is displayed in a grid. The projection position and shape of the projection image G with respect to the projection surface SC are not limited to Figure 1 the example shown and are arbitrary.

[0031] The projector 10 of the present embodiment has a function of controlling the operation of the camera 20 and a function of adjusting the shape of the projection image G using the shooting result of the camera 20.

[0032] The camera 20 is a digital camera having a shooting element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

[0033] The camera 20 photographs the area RC. The area RC is an area that includes the projected image G projected onto the projection surface SC. In Figure 1 the example shown, the area RC includes the area RP. Additionally, the camera 20 may also be a component of the projector 10.

[0034] The terminal device 30 is a computer that has a function of supplying the image data IMG to the projector 10. In Figure 1 the example shown, the terminal device 30 is a notebook computer. Additionally, the terminal device 30 is not limited to a notebook computer. For example, it may also be a desktop computer, a smart phone, a tablet terminal, etc., and may also be a video playback device, a DVD (Digital Versatile Disk) player, a Blu-ray Disc player, a hard disk recorder, a television tuner device, a set-top box for CATV (Cable television), a video game console, etc.

[0035] 1-2. Projector

[0036] Figure 2 is a block diagram of the projector 10 used in the system 100 of the first embodiment. In Figure 2 it, in addition to the projector 10, the connection states of the camera 20 and the terminal device 30 with respect to the projector 10 are also shown. In Figure 2 the example shown, the terminal device 30 includes a display device 31. The display device 31 is, for example, a display device including various display panels such as a liquid crystal display panel and an organic EL display panel.

[0037] As Figure 2 shown, the projector 10 has a storage device 11, a processing device 12, a communication device 13, an image processing circuit 14, an optical device 15, and an operation device 16. They are connected in a manner that enables communication with each other.

[0038] The storage device 11 is a storage device that stores the programs executed by the processing device 12 and the data processed by the processing device 12. The storage device 11 is configured, for example, to include a hard disk drive or a semiconductor memory. Additionally, part or all of the storage device 11 may also be provided in an external storage device or server, etc., of the projector 10.

[0039] Stored in the storage device 11 are a program PR1, first image information DG1, second image information DG2, first photographing data D1, second photographing data D2, mask information DM, amplitude intensity information DA, and correspondence information DC.

[0040] The program PR1 is a program for executing the calibration method described in detail later.

[0041] The first image information DG1 is information representing a first image G1 described later. The first image G1 includes a phase shift pattern PT described later and is projected onto a projection surface SC by a projector 10.

[0042] The first captured data D1 is information representing a first captured image GG1 described later, which is obtained by capturing the first image G1 projected onto the projection surface SC with a camera 20.

[0043] The amplitude intensity information DA is information representing an amplitude intensity image GG2 described later, which is generated based on the first captured data D1. The amplitude intensity image GG2 is an image that emphasizes the edges of the first captured image GG1 represented by the first captured data D1.

[0044] The mask information DM is information representing a mask image M described later.

[0045] The second image information DG2 is information representing a second image G2 described later. The second image G2 includes a structured light pattern and is masked by a mask image M described later represented by the mask information DM, and is projected onto the projection surface SC by the projector 10.

[0046] The second captured data D2 is information representing a second captured image obtained by capturing the second image G2 projected onto the projection surface SC with the camera 20.

[0047] The correspondence information DC is information representing the correspondence relationship between the coordinates in the display coordinate system of the projector 10 and the coordinates in the capture coordinate system of the camera 20. The display coordinate system of the projector 10 is a coordinate system using the pixels of a display panel 15b described later as coordinate values. The capture coordinate system of the camera 20 is a coordinate system using the pixels of the capture element of the camera 20 as coordinate values.

[0048] The processing device 12 is a processing device having functions of controlling respective parts of the projector 10 and processing various data. The processing device 12 is configured to include, for example, a processor such as a CPU (Central Processing Unit). In addition, the processing device 12 may be constituted by a single processor or may be constituted by a plurality of processors. In addition, part or all of the functions of the processing device 12 may also be implemented by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). In addition, the processing device 12 may also be integrated with at least a part of the image processing circuit 14.

[0049] The communication device 13 is a communication device capable of communicating with various devices, obtaining the image data IMG from the terminal device 30, or communicating with the camera 20. For example, the communication device 13 is a wired communication device such as a wired LAN (Local Area Network), USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), an LPWA (Low Power Wide Area), a wireless LAN including Wi-Fi, or a wireless communication device such as Bluetooth. "HDMI", "Wi-Fi", and "Bluetooth" are registered trademarks respectively.

[0050] The image processing circuit 14 is a circuit that performs necessary processing on the image data IMG from the communication device 13 and inputs it to the optical device 15. The image processing circuit 14 has, for example, a frame memory (not shown), expands the image data IMG in the frame memory, appropriately performs various processes such as resolution conversion processing, size adjustment processing, and distortion correction processing, and inputs it to the optical device 15. Here, the above-mentioned correspondence information DC is appropriately used in these various processes. In addition, the image processing circuit 14 may also perform processes such as OSD (On Screen Display) processing that generates image information for menu display or operation guidance as needed and synthesizes it into the image data IMG.

[0051] The optical device 15 is a device that projects image light onto the projection surface SC. The optical device 15 includes a light source 15a, a display panel 15b, and an optical system 15c.

[0052] The light source 15a includes, for example, a halogen lamp, a xenon lamp, an ultra-high pressure mercury lamp, an LED (Light Emitting Diode), or a laser light source, and emits red, green, and blue light respectively. The display panel 15b is a light modulator including three light modulation elements provided corresponding to red, green, and blue. Each light modulation element includes, for example, a transmissive liquid crystal panel, a reflective liquid crystal panel, or a DMD (Digital Micromirror Device), and generates image light of each color by modulating the light of the corresponding color. The image light of each color generated by the display panel 15b is synthesized by a color synthesis optical system to become full-color image light. The optical system 15c is a projection optical system including a projection lens that forms an image of the full-color image light from the display panel 15b and projects it onto the projection surface SC.

[0053] The operation device 16 is a device that accepts operations from the user. For example, the operation device 16 includes an operation panel (not shown) and an infrared remote control light receiving unit. The operation panel is provided on the outer casing of the projector 10 and outputs a signal based on an operation from the user. The infrared remote control light receiving unit receives an infrared signal from a remote control (not shown), decodes the infrared signal, and outputs a signal based on the operation of the remote control. In addition, the operation device 16 is provided as needed and may be omitted.

[0054] In the above projector 10, the processing device 12 functions as a projection control unit 12a, a shooting control unit 12b, and a generation unit 12c by executing the program PR1 stored in the storage device 11. Therefore, the processing device 12 includes the projection control unit 12a, the shooting control unit 12b, and the generation unit 12c.

[0055] The projection control unit 12a controls the operations of the image processing circuit 14 and the optical device 15. More specifically, the projection control unit 12a controls the projection of the projection image G onto the projection surface SC through the operation of the optical device 15. More specifically, the projection control unit 12a projects a first image G1 (described later) based on the first image information DG1 onto the projection surface SC, or projects a second image G2 (described later) based on the second image information DG2 onto the projection surface SC.

[0056] The shooting control unit 12b controls the operation of the camera 20. More specifically, the shooting control unit 12b obtains the first shooting data D1 by causing the camera 20 to shoot a first image G1 (described later) projected onto the projection surface SC, or obtains the second shooting data D2 by causing the camera 20 to shoot a second image G2 (described later) projected onto the projection surface SC. Then, the shooting control unit 12b stores the obtained first shooting data D1 and second shooting data D2 in the storage device 11.

[0057] The generation unit 12c generates provisional correspondence information DC, or generates amplitude intensity information DA and mask information DM based on the first shooting data D1. In addition, the generation unit 12c generates new correspondence information DC based on the second shooting data D2.

[0058] 1-3. Calibration method

[0059] Figure 3 It is a flowchart showing the flow of the calibration method of the first embodiment. This calibration method is performed by the processing device 12, which is an example of a "computer", executing the program PR1 using the above system 100. Here, as described above, the system 100 has a camera 20 and a projector 10 connected to the camera 20 in a communicable manner.

[0060] The calibration method includes step S10, step S20, and step S30. Here, the projector 10 executes steps S10 to S30. Additionally, the program PR1 causes the processing device 12 to execute steps S10 to S30.

[0061] In step S10, a measurement based on the phase shift method is performed over the entire area of the above-mentioned area RP. Step S10 includes step S11, step S12, and step S13. In step S11, a first image G1 described later is projected as a projection image G from the projector 10 onto the projection surface SC. This projection is performed by the projection control unit 12a controlling the operations of the image processing circuit 14 and the optical device 15 based on the first image information DG1.

[0062] In step S12, by photographing the first image G1 described later using the camera 20, a first photographed image GG1 described later is obtained. This obtaining is performed by the photographing control unit 12b controlling the operation of the camera 20. Through this obtaining, first photographed data D1 is generated, and the generated first photographed data D1 is stored in the storage device 11.

[0063] In step S13, based on the first photographed image GG1, the coordinates of the display coordinate system of the projector 10 and the coordinates of the photographing coordinate system of the camera 20 are corresponded over the entire area of the above-mentioned area RP. The generation unit 12c performs this correspondence based on the first photographed data D1 by a known measurement method. Through this correspondence, tentative correspondence information DC is generated, and the generated correspondence information DC is stored in the storage device 11.

[0064] In step S20, a mask image M is generated based on the first photographed image GG1. This generation is performed by the generation unit 12c using the result of extracting the contour of the projection surface SC based on the first photographed data D1. Through this generation, mask information DM is generated, and the generated mask information DM is stored in the storage device 11.

[0065] Step S20 of the present embodiment includes step S21 and step S22. In step S21, an amplitude intensity image represented by amplitude intensity information DA is generated based on the first photographed image GG1. In step S22, the shape of the mask image M is determined based on the amplitude intensity image represented by the amplitude intensity information DA.

[0066] In step S30, a measurement based on the phase shift method is performed over the regions in the above-described region RP other than the region represented by the mask information DM. Step S30 includes step S31, step S32, and step S33. In step S31, a second image G2, which will be described later, is projected as a projection image G from the projector 10 onto the projection surface SC. This projection is performed by the projection control unit 12a controlling the operations of the image processing circuit 14 and the optical device 15 based on the second image information DG2.

[0067] In step S32, the second image G2, which will be described later, is captured by the camera 20, thereby obtaining a second captured image. This obtaining is performed by the capture control unit 12b controlling the operation of the camera 20. Through this obtaining, second capture data D2 is generated, and the generated second capture data D2 is stored in the storage device 11.

[0068] In step S33, based on the second captured image represented by the second capture data D2, the coordinates in the capture coordinate system of the camera 20 are made to correspond to the coordinates in the display coordinate system of the projector 10. The generation unit 12c performs this correspondence based on the second capture data D2 by a known measurement method. Through this correspondence, new correspondence information DC is generated, and the generated correspondence information DC is stored in the storage device 11.

[0069] Figure 4 is a diagram for explaining the first images G1-1 to G1-4. In Figure 4 the first images G1-1 to G1-4 in the display coordinate system of the projector 10 are shown. In addition, in Figure 4 the horizontal direction, i.e., the x direction, and the vertical direction, i.e., the y direction, of the display coordinate system of the projector 10 are shown together.

[0070] The first images G1-1 to G1-4 are each an image including a phase shift pattern PT. Hereinafter, the first images G1-1 to G1-4 may sometimes be referred to as the first image G1 without distinguishing each of them.

[0071] The phase shift pattern PT is a stripe pattern in which the luminance value changes along a sine wave in the x direction. The first images G1-1 to G1-4 are the same except that the phases of the phase shift pattern PT are different from each other. The phases of the phase shift pattern PT included in the first images G1-1 to G1-4 are shifted from each other by π / 2.

[0072] In Figure 4 the example shown, the number of stripes of the phase shift pattern PT is four. In addition, the number of stripes of the phase shift pattern PT is not limited to Figure 4 the example shown, and may be two or more.

[0073] In step S11, the projector 10 sequentially projects the first images G1-1 to G1-4 onto the projection surface SC. In addition, in step S12, whenever such projection is performed, the camera 20 captures the first image G1 projected onto the projection surface SC. Therefore, in step S12, the camera 20 captures the first images G1-1 to G1-4 projected onto the projection surface SC, respectively. Thus, the first captured data D1 is obtained.

[0074] In step S13, throughout the entire area of the region RP, the correspondence between the coordinate values in the display coordinate system of the projector 10 and the coordinate values in the capture coordinate system of the camera 20 is temporarily obtained by the phase shift method. Thus, tentative correspondence information DC is generated. Such a correspondence is obtained using the following equations (1) and (2).

[0075] [Mathematical formula 1]

[0076]

[0077] In equation (1), i is the stripe number of the phase shift pattern PT, is the x coordinate in the display coordinate system of the projector 10, T is the number of pixels in one period of the phase shift pattern PT, I1(u, v) is the luminance value I(u, v) at the coordinate (u, v) of the image obtained by capturing the first image G1-1, I2(u, v) is the luminance value I(u, v) at the coordinate (u, v) of the image obtained by capturing the first image G1-2, I3(u, v) is the luminance value I(u, v) at the coordinate (u, v) of the image obtained by capturing the first image G1-3, and I4(u, v) is the luminance value I(u, v) at the coordinate (u, v) of the image obtained by capturing the first image G1-4.

[0078] [Mathematical formula 2]

[0079] I(u, v) = Acos(x) + B (2)

[0080]

[0081] In equation (2), A is the luminance value based on the reflection, diffusion, absorption of light in the capture area, the camera sensitivity, etc., and B is the luminance based on the background color, the camera screen style, the central luminance of the panel, etc.

[0082] In this way, even if the absolute value of the luminance value I(u, v) at the same coordinate x of the four captured images obtained by capturing the first images G1-1 to G1-4 changes due to the surface state or color of the measurement object at the coordinate x, the relative value also changes by the amount of the phase difference of the stripe pattern. Thus, it is possible to reduce the influence of ambient light or the surface state of the measurement object, etc., and it is possible to obtain the phase value of the stripe pattern at the coordinate x.

[0083] Here, first, the phase values are not continuous values in the captured image, but are obtained within the range of -π to +π for each stripe of the stripe pattern. Moreover, such phase values are phase-connected (phase-linked) by a known method so as to be continuous values in the captured image.

[0084] Figure 5 is a diagram for explaining the captured image GG. In Figure 5 , the captured image GG obtained by capturing with the camera 20 when a white image is projected over the entire area of the above-mentioned region RP is shown. In addition, in Figure 5 , the captured image GG in the capture coordinate system in the case where the camera 20 is a camera using a fish-eye lens is illustrated.

[0085] As Figure 5 shown, lines L1 and L2 indicating edges appear in the captured image GG. Line L1 is a line along the outer edge of the region RP. Line L2 is a line along the outer edge of the projection plane SC, that is, the outer edge of the aforementioned wall surface WA1.

[0086] The region between line L1 and line L2 is the region outside the projection plane SC, specifically, the regions of the wall surfaces WA2, WA3, the ceiling CE, or the floor FL. When projecting onto the region between such line L1 and line L2, the reflected light in this region reaches the outer peripheral part of the projection plane SC, etc. Therefore, in step S11, when the phase shift pattern PT is projected over the entire area of the above-mentioned region RP, an error occurs in the measurement of the projection plane SC based on the phase shift method.

[0087] Therefore, after step S10, in step S20, the system 100 generates a mask image M having a shape that is not projected onto the region between line L1 and line L2, and in step S30, uses the mask image M to perform measurement of the projection plane SC based on the phase shift method. Thereby, the above-mentioned error can be reduced.

[0088] Here, in step S30, it is necessary to detect line L2 as the outer edge of the projection plane SC and use the detection result to determine the shape of the mask image M. However, in the captured image GG, in addition to line L2, line L1 also appears, and the contrast difference between the inside and outside at line L2 is small, so it is difficult to detect the outer edge of the projection plane SC with high precision.

[0089] Therefore, in step S21, the system 100 generates an amplitude intensity image GG2 based on the first captured image GG1, and in step S22, determines the mask shape based on the amplitude intensity image GG2.

[0090] Figure 6 is a diagram for explaining the amplitude intensity image GG2 based on the first captured image GG1. InFigure 6 In this figure, a part of the first captured image GG1 is shown on the left side of the figure, and the amplitude intensity image GG2 is shown on the right side of the figure.

[0091] In Figure 6 In the amplitude intensity image GG2 shown on the right side in Figure 6 compared with the first captured image GG1 shown on the left side in

[0092] Figure 7 This is a diagram for explaining the determination of the mask shape. In step S22, by detecting the line L2 in the amplitude intensity image GG2, as Figure 7 shown, a mask shape image GG3 representing the mask shape is generated.

[0093] Figure 8 This is a diagram for explaining the mask image M in the imaging coordinate system. In step S20, image processing is performed in such a way that the inside of the line L2 of the mask shape image GG3 is the transmission area OP and the outside is the non - transmission area RM, thereby generating the mask image M - C. The mask image M - C is the mask image in the imaging coordinate system.

[0094] Figure 9 This is a diagram for explaining the mask image M in the display coordinate system. In step S20, using the tentative correspondence information DC obtained in step S10, the above - mentioned mask image M - C is converted into the display coordinate system of the projector 10, thereby generating the mask image M - P. Additionally, hereinafter, the mask images M - C and M - P may sometimes be referred to as the mask image M without distinction.

[0095] Figure 10 This is a diagram for explaining the second image G2. In step S30, after generating the second image G2 using the mask image M - P, in step S31, the second image G2 is projected onto the projection surface SC.

[0096] In the present embodiment, as Figure 10 shown, the second image G2 is an image that includes the phase - shift pattern PT and is masked by the mask image M - P. Here, the phase - shift pattern PT is displayed by the aforementioned transmission area OP and masked by the aforementioned non - transmission area RM. The phase - shift pattern PT of the second image G2 is an example of a "structured light pattern". Additionally, in Figure 10In this case, one second image G2 is shown. Similar to the first images G1-1 to G1-4, four second images G2 with phase shifts of π / 2 from each other in the phase of the phase shift pattern PT are used. In step S32, every time it is projected, the camera 20 captures these four second images G2. Thus, second captured data D2 is obtained.

[0097] In addition, the structured light pattern for the second image G2 is not limited to the phase shift pattern PT. For example, it can also be other structured light patterns such as binary code patterns, dot patterns, rectangular patterns, polygon patterns, checkerboard patterns, Gray code patterns, or random dot patterns.

[0098] In step S33, for the area in the region RP except for the area masked by the mask image M, the correspondence between the coordinate values in the display coordinate system of the projector 10 and the coordinate values in the capture coordinate system of the camera 20 is obtained. Thus, new correspondence information DC is generated. In the present embodiment, such a correspondence is obtained by the phase shift method as described above.

[0099] In the above calibration method, since the first image G1 includes the phase shift pattern PT, it is possible to accurately determine the area to be masked on the projection surface SC based on the first captured image GG1. As a result, it is possible to generate a mask image M that masks the desired area of the projection surface SC. On this basis, by using such a mask image M for the second image G2, it is possible to accurately measure the projection surface SC based on the second captured image represented by the second captured data D2. As a result, it is possible to accurately correspond the coordinates in the capture coordinate system of the camera 20 with the coordinates in the display coordinate system of the projector 10. By using this correspondence, it is possible to accurately determine the correspondence between the coordinates in the display coordinate system of the projector 10 and the coordinates on the projection surface SC.

[0100] In the present embodiment, as described above, the second image G2 uses the phase shift pattern PT as the structured light pattern, so the phase shift pattern PT can be shared between the first image G1 and the second image G2.

[0101] In addition, as described above, the step S20 of generating the mask image M includes step S21 and step S22. Here, in step S21, based on the first captured image GG1, an amplitude intensity image represented by amplitude intensity information DA is generated. In step S22, based on the amplitude intensity image represented by the amplitude intensity information DA, the shape of the mask image M is determined. According to such step S21 and step S22, the accuracy of edge detection in the first captured image GG1 can be improved.

[0102] 2. Second Embodiment

[0103] Hereinafter, a second embodiment of the present disclosure will be described. For elements that have the same functions and operations as those in the first embodiment in the embodiments exemplified below, the reference numerals used in the description of the first embodiment are used, and their detailed descriptions are appropriately omitted.

[0104] Figure 11 It is a block diagram of the projector 10A used in the system 100A of the second embodiment. The system 100A is configured in the same manner as the system 100 of the first embodiment, except that it includes a projector 10A instead of the projector 10 of the first embodiment. The projector 10A is configured in the same manner as the projector 10 of the first embodiment, except that it uses a program PR2 instead of the program PR1 of the first embodiment.

[0105] The program PR2 is the same as the program PR1 of the first embodiment, except that the processing device 12 functions as a generation unit 12d instead of the generation unit 12c of the first embodiment.

[0106] The generation unit 12d is the same as the generation unit 12c of the first embodiment, except that the display device 31 displays an image GU for region designation and the mask shape is determined using the result of region designation in addition to the amplitude intensity image GG2.

[0107] Figure 12 It is a flowchart showing the process of the calibration method of the second embodiment. This calibration method includes a step S20A instead of the step S20 of the first embodiment, and steps S50 and S60 are added. Other than that, it is the same as the calibration method of the first embodiment.

[0108] In the calibration method of the present embodiment, first, in step S50, an image GU for region designation, which will be described later, is displayed on the display device 31. This display is performed by the generation unit 12d controlling the operation of the display device 31. After that, in step S60, it is determined whether there is an input for region designation. This determination is made by the generation unit 12d monitoring the input result to the terminal device 30. Step S60 is repeatedly executed until there is an input for region designation (step S60: No).

[0109] When there is an input for region designation (step S60: Yes), step S10 is executed in the same manner as in the first embodiment. After that, step S20A is executed.

[0110] Step S20A is the same as step S20 of the first embodiment, except that step S22A is executed instead of step S22 of the first embodiment. Step S22A is the same as step S22 of the first embodiment, except that the mask shape is determined using not only the amplitude intensity image GG2 but also the result of the region designation. In addition, steps S50 and S60 may be performed before step S22A, and are not limited to the illustrated example. For example, they may be performed between step S10 and step S20A, or within step S10 or step S20A.

[0111] Figure 13 FIG. is a diagram for explaining the image GU for region designation. In step S50, the image GU is displayed on the display device 31. The image GU is a GUI (graphical user interface) image for accepting region designation.

[0112] The image GU can accept adjustment of the shape and size of the region within the shooting region. In Figure 13 the illustrated example, a plurality of points are arranged along the contour L3 of the region to be designated. The adjustment of the region is performed, for example, by selecting an arbitrary point from the plurality of points using the cursor and then moving the selected point. In addition, the display for the adjustment of the region is not limited to Figure 13 the illustrated example and is arbitrary.

[0113] Although not illustrated, the determination of the region designation is performed, for example, by operating a determination button displayed on the display device 31.

[0114] According to the second embodiment described above, high-precision calibration of the coordinate systems of the camera 20 and the projector 10 can also be achieved. As described above, the calibration method of the present embodiment includes step S50 and step S60. In step S50, the display device 31 is caused to display an image for designating a region within the projection plane SC. In step S60, an input of the designated region is accepted from the user. The step S22A of determining the shape of the mask image M includes: determining the shape of the mask image M based on the amplitude intensity image GG2 and the region. Thereby, the accuracy of edge detection in the first captured image GG1 can be improved.

[0115] 3. Variation

[0116] Each of the above-exemplified modes can be variably deformed in various ways. The following are specific deformation modes that can be applied to each of the foregoing modes. Two or more modes arbitrarily selected from the following examples can be appropriately combined within a range where they do not conflict with each other.

[0117] 3-1. Variation 1

[0118] In the foregoing embodiment, the manner in which the processing device 12 of the projector 10 executes the programs PR1 and PR2 is illustrated, but it is not limited to this manner. For example, the programs PR1 and PR2 may also be executed by a processing device of a computer connected to the projector 10 and the camera 20 in a communicable manner.

[0119] 3-2. Variant Example 2

[0120] In the above embodiment, the manner of using the correspondence information DC for adjusting the projection image G is illustrated, but it is not limited to this manner. For example, the correspondence information DC can be used to display a pattern such as a grid pattern with uniformity on the projection surface SC, or can be used to reflect a three-dimensional shape model of the projection surface SC observed from the camera 20 onto three-dimensional image editing software or the like, and draw a picture on this model, or display on a PC monitor or the like how the picture looks when observed from the projector 10, or cause the projector 10 to project it.

[0121] 4. Postscript

[0122] Hereinafter, a summary of the present disclosure is noted.

[0123] (Postscript 1) The first mode, which is a preferred example of the calibration method of the present disclosure, includes: projecting a first image including a phase shift pattern from a projector onto a projection surface; obtaining a first captured image including the first image and captured by a camera; generating a mask image based on the first captured image; projecting a second image from the projector onto the projection surface, where the second image includes a structured light pattern and is obtained by masking with the mask image; obtaining a second captured image including the second image and captured by the camera; and corresponding the coordinates of the camera's capture coordinate system to the coordinates of the projector's display coordinate system based on the second captured image.

[0124] In the above mode, since the first image includes a phase shift pattern, the area on the projection surface to be masked can be accurately determined based on the first captured image. Thus, a mask image that masks the desired area on the projection surface can be generated with high accuracy. By using such a mask image for the second image, the reflection of unwanted light from surrounding objects on the projection surface can be appropriately suppressed. Therefore, based on the second captured image, with the coordinates on the projection surface as a reference, the coordinates of the camera's capture coordinate system can be accurately corresponded to the coordinates of the projector's display coordinate system. By using this correspondence, the correspondence relationship between the coordinates of the projector's display coordinate system and the coordinates on the projection surface can be accurately determined.

[0125] (Postscript 2) In the second mode, which is a preferred example of the first mode, the structured light pattern is a phase shift pattern. In the above mode, the phase shift pattern can be shared by the first image and the second image.

[0126] (Supplementary Note 3) In the third mode, which is a preferred example of the first mode or the second mode, generating the mask image includes: generating an amplitude intensity image based on the first captured image; and determining the shape of the mask image based on the amplitude intensity image. In the above mode, the accuracy of edge detection in the first captured image can be improved.

[0127] (Supplementary Note 4) In the fourth mode, which is a preferred example of the third mode, it further includes: causing a display device to display an image for specifying a region within the projection surface; and receiving an input from a user for specifying the region, and determining the shape of the mask image includes: determining the shape of the mask image based on the amplitude intensity image and the region. In the above mode, the accuracy of edge detection in the first captured image can be improved.

[0128] (Supplementary Note 5) A fifth mode, which is a preferred example of the system of the present disclosure, includes: a camera; and a projector connected to the camera in a communicable manner, and the projector performs the following steps: projecting a first image including a phase shift pattern onto a projection surface; obtaining a first captured image obtained by the camera capturing the first image; generating a mask image based on the first captured image; projecting a second image onto the projection surface, where the second image includes a structured light pattern and is obtained by masking with the mask image; obtaining a second captured image obtained by the camera capturing the second image; and corresponding the coordinates of the camera's imaging coordinate system to the coordinates of the projector's display coordinate system based on the second captured image.

[0129] In the above mode, since the first image includes a phase shift pattern, the region to be masked on the projection surface can be determined with high accuracy based on the first captured image. Thus, a mask image for masking a desired region on the projection surface can be generated with high accuracy. By using such a mask image for the second image, the reflection of unnecessary light from objects around the projection surface to the projection surface can be appropriately suppressed. Therefore, based on the second captured image, with the coordinates on the projection surface as a reference, the coordinates of the camera's imaging coordinate system can be accurately corresponded to the coordinates of the projector's display coordinate system. By using this correspondence, the correspondence relationship between the coordinates of the projector's display coordinate system and the coordinates on the projection surface can be accurately determined.

[0130] (Supplementary Note 6) The sixth mode, which is a preferred example of the program of the present disclosure, causes a computer to perform the following steps: projecting a first image including a phase shift pattern from a projector onto a projection surface; obtaining a first captured image by capturing the first image using a camera; generating a mask image based on the first captured image; projecting a second image from the projector onto the projection surface, where the second image includes a structured light pattern and is obtained by masking with the mask image; obtaining a second captured image by capturing the second image using the camera; and corresponding the coordinates of the capture coordinate system of the camera with the coordinates of the display coordinate system of the projector based on the second captured image.

[0131] In the above mode, since the first image includes a phase shift pattern, it is possible to accurately determine the area to be masked on the projection surface based on the first captured image. Thus, it is possible to accurately generate a mask image that masks a desired area on the projection surface. By using such a mask image for the second image, it is possible to appropriately suppress the reflection of unnecessary light from objects around the projection surface to the projection surface. Therefore, based on the second captured image, it is possible to accurately correspond the coordinates of the capture coordinate system of the camera with the coordinates of the display coordinate system of the projector with reference to the coordinates on the projection surface. By using this correspondence, it is possible to accurately determine how the coordinates of the display coordinate system of the projector correspond to the coordinates on the projection surface.

Claims

1. A calibration method, comprising: projecting a first image including a phase shift pattern from a projector onto a projection surface; obtaining a first captured image that includes the first image and is captured by a camera; generating a mask image based on the first captured image; Cause a second image to be projected from the projector onto the projection surface, wherein, the second image includes a structured light pattern and is obtained by masking with the mask image; obtaining a second captured image that includes the second image and is captured by the camera; and based on the second captured image, performing correspondence between coordinates in the camera's capture coordinate system and coordinates in the projector's display coordinate system.

2. The calibration method according to claim 1, wherein the structured light pattern is a phase shift pattern.

3. The calibration method according to claim 1 or 2, wherein generating the mask image includes: generating an amplitude intensity image based on the first captured image; and determining the shape of the mask image based on the amplitude intensity image.

4. The calibration method according to claim 3, wherein the calibration method further includes: causing a display device to display an image for specifying a region within the projection surface; and receiving an input from a user specifying the region, determining the shape of the mask image includes: determining the shape of the mask image based on the amplitude intensity image and the region.

5. A system, comprising: a camera; and a projector communicably connected to the camera, the projector performs the following processing: projecting a first image including a phase shift pattern onto a projection surface; obtaining a first captured image obtained by the camera capturing the first image; generating a mask image based on the first captured image; Project the second image onto the projection surface, where the second image includes a structured light pattern and is obtained by masking with the mask image; obtaining a second captured image obtained by the camera capturing the second image; and based on the second captured image, performing correspondence between coordinates in the camera's capture coordinate system and coordinates in the projector's display coordinate system.

6. A program product that causes a computer to perform the following processing: projecting a first image including a phase shift pattern from a projector onto a projection surface; obtaining a first captured image by capturing the first image with a camera; generating a mask image based on the first captured image; Cause a second image to be projected from the projector onto the projection surface, where the second image includes a structured light pattern and is obtained by masking with the mask image; obtaining a second captured image by capturing the second image with the camera; and based on the second captured image, performing correspondence between coordinates in the camera's capture coordinate system and coordinates in the projector's display coordinate system.

Images