Calibration method, system, and program

Abstract

To realize a highly accurate calibration of a coordinate system of a camera and a projector.SOLUTION: A calibration method includes the steps of: causing a projector to project a first image including a phase shift pattern onto a projection surface; acquiring a first imaging image including the first image and imaged by a camera; generating a mask image based on the first imaging image; causing the projector to project a second image including a structured light pattern and masked by the mask image onto the projection surface; acquiring a second imaging image including the second image and imaged by the camera; and associating coordinates of an imaging coordinate system of the camera with coordinates of a display coordinate system of the projector based on the second imaging image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a calibration method, system, and program. [Background technology]

[0002] Patent Document 1 discloses a projector that projects a display image onto a wall surface of a room. The wall surface is adjacent to both side walls, a ceiling surface, and a floor surface. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-48015 Summary of the Invention [Problem to be solved by the invention]

[0004] In order for a projector to properly project an image onto a projection surface, it is necessary to identify the correspondence between the coordinates in the projector's display coordinate system and the coordinates on the projection surface. However, in the past, when there was another wall surface near the outer edge of the projection surface, it was difficult to measure this correspondence with high precision due to light reflected from the other wall surface onto the projection surface. [Means for solving the problem]

[0005] A projection method according to one embodiment of the present disclosure includes projecting a first image including a phase shift pattern from a projector onto a projection surface, acquiring a first captured image including the first image and captured by a camera, generating a mask image based on the first captured image, projecting a second image including a structured light pattern and masked by the mask image from the projector onto the projection surface, acquiring a second captured image including the second image and captured by the camera, and corresponding coordinates in the camera's imaging coordinate system to coordinates in the projector's display coordinate system based on the second captured image.

[0006] A system according to one aspect of the present disclosure includes a camera and a projector communicatively connected to the camera, wherein the projector performs the following operations: projecting a first image including a phase shift pattern onto a projection surface; acquiring a first captured image by capturing the first image with the camera; generating a mask image based on the first captured image; projecting a second image including a structured light pattern and masked by the mask image onto the projection surface; acquiring a second captured image by capturing the second image with the camera; and corresponding, based on the second captured image, coordinates of the camera's imaging coordinate system to coordinates of the projector's display coordinate system.

[0007] A program according to one embodiment of the present disclosure causes a computer to perform the following steps: projecting a first image including a phase shift pattern from a projector onto a projection surface; capturing the first image with a camera to obtain a first captured image; generating a mask image based on the first captured image; projecting a second image including a structured light pattern and masked with the mask image from the projector onto the projection surface; capturing the second image with the camera to obtain a second captured image; and corresponding, based on the second captured image, coordinates in the camera's imaging coordinate system to coordinates in the projector's display coordinate system. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an outline of a system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram of a projector used in the system according to the first embodiment. [Figure 3] 4 is a flowchart showing the flow of a calibration method according to the first embodiment. [Figure 4] FIG. 2 is a diagram for explaining a first image. [Figure 5] FIG. 2 is a diagram for explaining a captured image. [Figure 6] FIG. 10 is a diagram for explaining an amplitude intensity image based on the first captured image. [Figure 7] FIG. 10 is a diagram for explaining the determination of a mask shape. [Figure 8] FIG. 10 is a diagram for explaining a mask image in an imaging coordinate system. [Figure 9] FIG. 10 is a diagram for explaining a mask image in a display coordinate system. [Figure 10] FIG. 10 is a diagram for explaining a second image. [Figure 11] FIG. 10 is a block diagram of a projector used in a system according to a second embodiment. [Figure 12] 10 is a flowchart showing the flow of a calibration method according to a second embodiment. [Figure 13] FIG. 10 is a diagram illustrating an image for specifying an area. DETAILED DESCRIPTION OF THE INVENTION

[0009] Preferred embodiments of the present disclosure will be described below with reference to the accompanying drawings. Note that the dimensions and scale of each part in the drawings may differ from the actual dimensions and are shown schematically to facilitate understanding. Furthermore, the scope of the present disclosure is not limited to these embodiments unless otherwise specified in the following description to the effect that the present disclosure is limited.

[0010] 1. First embodiment 1-1. System Overview FIG. 1 is a diagram illustrating an outline of a system 100 according to a first embodiment. The system 100 is a projection system that projects a projection image G onto a projection surface SC. In FIG. 1, the direction in which the ceiling surface CE, wall surface WA1, and floor surface FL are arranged in this order as viewed perpendicular to the paper surface is referred to as the up-down direction. As viewed from wall surface WA1, the ceiling surface CE side is the upper side in the drawing, and as viewed from wall surface WA1, the floor surface FL side is the lower side in the drawing. In FIG. 1, the direction in which the wall surfaces WA2, WA1, and WA3 are arranged in this order as viewed perpendicular to the paper surface is referred to as the left-right direction. As viewed from wall surface WA1, the wall surface WA2 side is the left side in the drawing, and as viewed from wall surface WA1, the wall surface WA3 side is the right side in the drawing.

[0011] The projection surface SC is wall surface WA1. Wall surface WA2 is adjacent to wall surface WA1 on the left side in the drawing. Meanwhile, wall surface WA3 is adjacent to wall surface WA1 on the right side in the drawing. Furthermore, ceiling surface CE is adjacent to wall surface WA1 on the upper side in the drawing. Meanwhile, floor surface FL is adjacent to wall surface WA1 on the lower side in the drawing. In this way, wall surface WA1 is a surface surrounded by walls WA2, WA3, ceiling surface CE, and floor surface FL.

[0012] The projection surface SC is not limited to a wall surface WA1, and may be the surface of an object such as a screen, etc. The projection surface SC is also not limited to a flat surface, and may be a curved surface, for example, concavely or convexly.

[0013] As shown in FIG. 1, the system 100 includes a projector 10, a camera 20, and a terminal device 30.

[0014] The projector 10 is a display device that projects a projection image G indicated by video data IMG output from the terminal device 30 onto a projection surface SC. In the example shown in FIG. 1, the projection image G is projected onto a rectangular area that covers substantially the entire projection surface SC. The projector 10 can also project the projection image G onto an area RP that encompasses the projection surface SC. Note that in FIG. 1, the projection image G is displayed as a shaded area. The projection position and shape of the projection image G relative to the projection surface SC are not limited to the example shown in FIG. 1 and are arbitrary.

[0015] The projector 10 of this embodiment has a function of controlling the operation of the camera 20 and a function of adjusting the shape of the projected image G using the image pickup result of the camera 20.

[0016] The camera 20 is a digital camera having an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

[0017] The camera 20 captures an image of the area RC. The area RC is an area that includes the projected image G projected onto the projection surface SC. In the example shown in Fig. 1, the area RC includes the area RP. Note that the camera 20 may be a component of the projector 10.

[0018] The terminal device 30 is a computer that has a function of supplying video data IMG to the projector 10. In the example shown in Fig. 1, the terminal device 30 is a notebook computer. Note that the terminal device 30 is not limited to a notebook computer, and may be, for example, a desktop computer, a smartphone, a tablet terminal, a video playback device, a DVD (Digital Versatile Disk) player, a Blu-ray Disc player, a hard disk recorder, a television tuner device, a CATV (Cable television) set-top box, a video game console, or the like.

[0019] 1-2.Projector Fig. 2 is a block diagram of a projector 10 used in a system 100 according to the first embodiment. In addition to the projector 10, Fig. 2 also shows the connection state of a camera 20 and a terminal device 30 to the projector 10. In the example shown in Fig. 2, the terminal device 30 includes a display device 31. The display device 31 is a display device including various display panels such as a liquid crystal display panel and an organic EL display panel.

[0020] 2, the projector 10 has a storage device 11, a processing device 12, a communication device 13, an image processing circuit 14, an optical device 15, and an operation device 16. These are connected to each other so that they can communicate with each other.

[0021] The storage device 11 is a storage device that stores programs executed by the processing device 12 and data processed by the processing device 12. The storage device 11 includes, for example, a hard disk drive or a semiconductor memory. Note that part or all of the storage device 11 may be provided in an external storage device or server outside the projector 10.

[0022] The storage device 11 stores a program PR1, first image information DG1, second image information DG2, first imaging data D1, second imaging data D2, mask information DM, amplitude intensity information DA, and correspondence information DC.

[0023] The program PR1 is a program for executing a calibration method, which will be described in detail later.

[0024] The first image information DG1 is information indicating a first image G1, which will be described later. The first image G1 includes a phase shift pattern PT, which will be described later, and is projected by the projector 10 onto the projection surface SC.

[0025] The first imaging data D1 is information indicating a first captured image GG1 (to be described later) that is acquired by using the camera 20 to capture the first image G1 projected on the projection surface SC.

[0026] The amplitude intensity information DA is information indicating an amplitude intensity image GG2 (described later) generated based on the first imaging data D1. The amplitude intensity image GG2 is an image that emphasizes the edges of the first imaging image GG1 indicated by the first imaging data D1.

[0027] The mask information DM is information indicating a mask image M, which will be described later.

[0028] The second image information DG2 is information indicating a second image G2 (described later). The second image G2 includes a structured light pattern and is masked with a mask image M (described later) indicated by the mask information DM, and is projected onto the projection surface SC by the projector 10.

[0029] The second imaging data D2 is information indicating a second captured image obtained by using the camera 20 to capture the second image G2 projected on the projection surface SC.

[0030] The correspondence information DC is information indicating the correspondence relationship between the coordinates of the display coordinate system of the projector 10 and the coordinates of the imaging coordinate system of the camera 20. The display coordinate system of the projector 10 is a coordinate system in which the pixels of the display panel 15b (described later) are used as coordinate values. The imaging coordinate system of the camera 20 is a coordinate system in which the pixels of the imaging element of the camera 20 are used as coordinate values.

[0031] The processing device 12 has the function of controlling each unit of the projector 10 and the function of processing various data. The processing device 12 includes, for example, a processor such as a CPU (Central Processing Unit). The processing device 12 may be configured with a single processor or multiple processors. Some or all of the functions of the processing device 12 may be realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The processing device 12 may be integrated with at least a part of the image processing circuit 14.

[0032] The communication device 13 is a communication device capable of communicating with various devices, and acquires video data IMG from the terminal device 30 and communicates with the camera 20. For example, the communication device 13 is a wired communication device such as a wired LAN (Local Area Network), USB (Universal Serial Bus), or HDMI (High Definition Multimedia Interface), or a wireless communication device such as LPWA (Low Power Wide Area), wireless LAN including Wi-Fi, or Bluetooth. "HDMI," "Wi-Fi," and "Bluetooth" are all registered trademarks.

[0033] The image processing circuit 14 is a circuit that performs necessary processing on the video data IMG from the communication device 13 and inputs the data to the optical device 15. The image processing circuit 14 has, for example, a frame memory (not shown), and loads the video data IMG into the frame memory, appropriately performs various processes such as resolution conversion, resizing, and distortion correction, and inputs the data to the optical device 15. The various processes here use the aforementioned correspondence information DC as appropriate. Note that the image processing circuit 14 may also perform processes such as OSD (On Screen Display) processing, where necessary, that generate image information for menu display or operation guides, etc., and combine it with the video data IMG.

[0034] The optical device 15 is a device that projects image light onto the projection surface SC, and includes a light source 15a, a display panel 15b, and an optical system 15c.

[0035] The light source 15a includes a light source such as a halogen lamp, a xenon lamp, an ultra-high pressure mercury lamp, an LED (Light Emitting Diode), or a laser light source, and emits red, green, and blue light, respectively. The display panel 15b is an optical modulator including three light modulation elements corresponding to red, green, and blue. Each light modulation element includes, for example, a transmissive liquid crystal panel, a reflective liquid crystal panel, or a DMD (Digital Mirror Device), and generates image light of each color by modulating light of the corresponding color. The image light of each color generated by the display panel 15b is combined by a color combining optical system to become full-color image light. The optical system 15c is a projection optical system including a projection lens and the like that forms and projects the full-color image light from the display panel 15b onto the projection surface SC.

[0036] The operation device 16 is a device that accepts operations from the user. For example, the operation device 16 includes an operation panel and a remote control receiver, not shown. The operation panel is provided on the exterior housing of the projector 10, and outputs a signal based on an operation from the user. The remote control receiver receives an infrared signal from a remote control, not shown, decodes the infrared signal, and outputs a signal based on the operation of the remote control. The operation device 16 may be provided as needed, or may be omitted.

[0037] In the above-described projector 10, the processing device 12 functions as a projection control unit 12a, an imaging control unit 12b, and a generation unit 12c by executing the program PR1 stored in the storage device 11. Therefore, the processing device 12 includes the projection control unit 12a, the imaging control unit 12b, and the generation unit 12c.

[0038] The projection control unit 12a controls the operation of the image processing circuit 14 and the optical device 15. More specifically, the projection control unit 12a controls the operation of the optical device 15 to project a projection image G onto the projection surface SC. More specifically, the projection control unit 12a projects a first image G1 (described below) based on first image information DG1 onto the projection surface SC, and projects a second image G2 (described below) based on second image information DG2 onto the projection surface SC.

[0039] The imaging control unit 12b controls the operation of the camera 20. More specifically, the imaging control unit 12b acquires first imaging data D1 by having the camera 20 capture a first image G1 (described later) projected on the projection surface SC, and acquires second imaging data D2 by having the camera 20 capture a second image G2 (described later) projected on the projection surface SC. The imaging control unit 12b then stores the acquired first imaging data D1 and second imaging data D2 in the storage device 11.

[0040] The generating unit 12c generates provisional correspondence information DC, amplitude intensity information DA, and mask information DM based on the first imaging data D1, and also generates new correspondence information DC based on the second imaging data D2.

[0041] 1-3. Calibration method 3 is a flowchart showing the flow of the calibration method according to the first embodiment. The calibration method is performed by the processing device 12, which is an example of a "computer," executing the program PR1 using the system 100 described above. Here, as described above, the system 100 includes the camera 20 and the projector 10 communicatively connected to the camera 20.

[0042] The calibration method includes steps S10, S20, and S30. The projector 10 executes steps S10 to S30. The program PR1 also causes the processing device 12 to execute steps S10 to S30.

[0043] In step S10, measurement is performed using the phase shift method over the entire area RP described above. Step S10 includes steps S11, S12, and S13. In step S11, a first image G1 (described below) is projected onto the projection surface SC as a projection image G. This projection is performed by the projection control unit 12a controlling the operations of the image processing circuit 14 and the optical device 15 based on the first image information DG1.

[0044] In step S12, a first captured image GG1 (described later) is acquired by capturing a first image G1 (described later) using the camera 20. This acquisition is performed by the imaging control unit 12b controlling the operation of the camera 20. As a result of this acquisition, first captured data D1 is generated, and the generated first captured data D1 is stored in the storage device 11.

[0045] In step S13, based on the first captured image GG1, the coordinates in the display coordinate system of the projector 10 are associated with the coordinates in the image capture coordinate system of the camera 20 over the entire area RP. This association is performed by the generation unit 12c using a known measurement method based on the first captured data D1. By this association, provisional correspondence information DC is generated, and the generated correspondence information DC is stored in the storage device 11.

[0046] In step S20, a mask image M is generated based on the first captured image GG1. This generation is performed by using the result of the generation unit 12c extracting the contour of the projection surface SC based on the first captured image data D1. As a result of this generation, mask information DM is generated, and the generated mask information DM is stored in the storage device 11.

[0047] Step S20 of this embodiment includes steps S21 and S22. Step S21 generates an amplitude image indicated by the amplitude intensity information DA based on the first captured image GG1. Step S22 determines the shape of the mask image M based on the amplitude intensity image indicated by the amplitude intensity information DA.

[0048] In step S30, measurement is performed using the phase shift method over the aforementioned region RP excluding the region indicated by the mask information DM. Step S30 includes steps S31, S32, and S33. In step S31, a second image G2 (described below) is projected onto the projection surface SC as a projection image G. This projection is performed by the projection control unit 12a controlling the operations of the image processing circuit 14 and the optical device 15 based on the second image information DG2.

[0049] In step S32, a second captured image is acquired by capturing a second image G2 (described later) using the camera 20. This acquisition is performed by the imaging control unit 12b controlling the operation of the camera 20. As a result of this acquisition, second captured data D2 is generated, and the generated second captured data D2 is stored in the storage device 11.

[0050] In step S33, based on the second captured image indicated by the second captured data D2, the coordinates in the image capture coordinate system of the camera 20 are associated with the coordinates in the display coordinate system of the projector 10. This association is performed by the generation unit 12c using a known measurement method based on the second captured data D2. Through this association, new association information DC is generated, and the generated association information DC is stored in the storage device 11.

[0051] Fig. 4 is a diagram for explaining the first images G1-1 to G1-4. Fig. 4 shows the first images G1-1 to G1-4 in the display coordinate system of the projector 10. Note that Fig. 4 also shows the x direction, which is the horizontal direction, and the y direction, which is the vertical direction, of the display coordinate system of the projector 10.

[0052] Each of the first images G1-1 to G1-4 is an image including a phase shift pattern PT. Hereinafter, the first images G1-1 to G1-4 may be referred to as the first image G1 without being distinguished from one another.

[0053] The phase shift pattern PT is a stripe pattern in which the brightness value changes along a sine wave in the x direction. The first images G1-1 to G1-4 are similar except that the phases of the phase shift patterns PT are different from each other. The phases of the phase shift patterns PT included in the first images G1-1 to G1-4 are shifted from each other by π / 2.

[0054] In the example shown in Fig. 4, the number of stripes in the phase shift pattern PT is 4. Note that the number of stripes in the phase shift pattern PT is not limited to the example shown in Fig. 4, and may be two or more.

[0055] In step S11, the projector 10 projects the first images G1-1 to G1-4 in order onto the projection surface SC. Furthermore, in step S12, after each projection, the camera 20 captures an image of the first image G1 projected onto the projection surface SC. Therefore, in step S12, each of the first images G1-1 to G1-4 projected onto the projection surface SC is captured by the camera 20. As a result, first captured data D1 is obtained.

[0056] In step S13, the correspondence between the coordinate values in the display coordinate system of the projector 10 and the coordinate values in the image capture coordinate system of the camera 20 is provisionally determined using the phase shift method over the entire area RP. This generates provisional correspondence information DC. This correspondence is determined using the following equations 1 and 2.

[0057]

number

[0058] In equation (1), i is the stripe number of the phase shift pattern PT and the x-coordinate in the display coordinate system of the projector 10, T is the number of pixels in one period of the phase shift pattern PT, I1(u,v) is the luminance value I(u,v) at the coordinates (u,v) of the image obtained by capturing the first image G1-1, I2(u,v) is the luminance value I(u,v) at the coordinates (u,v) of the image obtained by capturing the first image G1-2, I3(u,v) is the luminance value I(u,v) at the coordinates (u,v) of the image obtained by capturing the first image G1-3, and I4(u,v) is the luminance value I(u,v) at the coordinates (u,v) of the image obtained by capturing the first image G1-4.

number

[0059] In equation (2), A is the luminance value due to reflection, diffusion, absorption of light in the imaging area, camera sensitivity, etc., and B is the luminance due to the background color, camera tone, center luminance of the panel, etc.

[0060] In this way, even if the absolute value of the luminance value I(u,v) at the same coordinate x in the four captured images obtained by capturing the first images G1-1 to G1-4 changes due to the surface condition or color of the object to be measured at that coordinate x, the relative value changes by the phase difference of the stripe pattern. This makes it possible to determine the phase value of the stripe pattern at that coordinate x while reducing the influence of ambient light or the surface condition of the object to be measured.

[0061] Here, the phase values are first calculated in the range of -π to +π for each stripe of the stripe pattern, rather than as continuous values in the captured image, and then these phase values are phase-connected (phase-unwrapped) by a known method so that they become continuous values in the captured image.

[0062] Fig. 5 is a diagram for explaining the captured image GG. Fig. 5 shows the captured image GG obtained by imaging with the camera 20 when a white image is projected over the entire area of the aforementioned region RP. Fig. 5 illustrates the captured image GG in an imaging coordinate system when the camera 20 is a camera using a fisheye lens.

[0063] 5, lines L1 and L2 indicating edges appear in the captured image GG. Line L1 is a line that follows the outer edge of the area RP. Line L2 is a line that follows the outer edge of the projection surface SC, i.e., the outer edge of the wall surface WA1 mentioned above.

[0064] The area between lines L1 and L2 is the area outside the projection surface SC, specifically the area of walls WA2, WA3, ceiling CE, or floor FL. If projection is performed in this area between lines L1 and L2, the light reflected in this area will reach the outer periphery of the projection surface SC. For this reason, if the phase shift pattern PT is projected over the entire area RP in step S11, an error will occur in the measurement of the projection surface SC using the phase shift method.

[0065] Therefore, after step S10, in step S20, the system 100 generates a mask image M having a shape that does not project onto the area between the line L1 and the line L2, and in step S30, measures the projection surface SC by the phase shift method using the mask image M. This makes it possible to reduce the above-mentioned error.

[0066] Here, in step S30, it is necessary to detect line L2 as the outer edge of projection surface SC and use the detection result to determine the shape of mask image M. However, in captured image GG, line L1 appears in addition to line L2, and the contrast difference between the inside and outside of line L2 is small, making it difficult to detect the outer edge of projection surface SC with high accuracy.

[0067] Therefore, in step S21, the system 100 generates an amplitude intensity image GG2 based on the first captured image GG1, and in step S22, determines a mask shape based on the amplitude intensity image GG2.

[0068] Fig. 6 is a diagram for explaining an amplitude intensity image GG2 based on the first captured image GG1. In Fig. 6, a part of the first captured image GG1 is shown on the left side of the drawing, and the amplitude intensity image GG2 is shown on the right side of the drawing.

[0069] In the amplitude intensity image GG2 shown on the right side of Fig. 6, the contrast difference between the inside and outside of the line L2 is greater than in the first captured image GG1 shown on the left side of Fig. 6, so the outer edge of the projection surface SC can be detected with high accuracy. The amplitude intensity image GG2 is an image in which the luminance value of each pixel is A in the above-mentioned equation 2.

[0070] 7 is a diagram for explaining the determination of the mask shape. In step S22, a mask shape image GG3 indicating the mask shape is generated by detecting a line L2 in the amplitude image GG2, as shown in FIG.

[0071] 8 is a diagram for explaining a mask image M in an imaging coordinate system. In step S20, image processing is performed so that the inside of the line L2 of the mask shape image GG3 is a transparent region OP and the outside is a non-transparent region RM, thereby generating a mask image MC. The mask image MC is a mask image in the imaging coordinate system.

[0072] 9 is a diagram for explaining a mask image M in a display coordinate system. In step S20, a mask image MP is generated by converting the mask image MC described above into the display coordinate system of the projector 10 using the provisional correspondence information DC obtained in step S10. Note that, hereinafter, the mask images MC and MP may be referred to as the mask image M without distinction.

[0073] 10 is a diagram for explaining the second image G2. In step S30, the second image G2 is generated using the mask image MP, and then in step S31, the second image G2 is projected onto the projection surface SC.

[0074] In this embodiment, as shown in FIG. 10, the second image G2 includes a phase shift pattern PT and is masked by a mask image MP. Here, the phase shift pattern PT is displayed in the aforementioned transparent region OP and masked in the aforementioned non-transparent region RM. The phase shift pattern PT of the second image G2 is an example of a "structured light pattern." Note that, although one second image G2 is shown in FIG. 10, four second images G2 are used, in which the phases of the phase shift patterns PT are shifted by π / 2 from one another, as with the first images G1-1 to G1-4. In step S32, the camera 20 captures these four second images G2 for each projection. This results in second captured data D2.

[0075] The structured light pattern used for the second image G2 is not limited to the phase shift pattern PT, but may be other structured light patterns such as a binary code pattern, a dot pattern, a rectangular pattern, a polygonal pattern, a checkered pattern, a gray code pattern, or a random dot pattern.

[0076] In step S33, for the region RP excluding the masked region of the mask image M, the correspondence relationship between the coordinate values in the display coordinate system of the projector 10 and the coordinate values in the imaging coordinate system of the camera 20 is found. As a result, new correspondence information DC is generated. In this embodiment, such correspondence relationship is found by the phase shift method as described above.

[0077] In the above calibration method, because the first image G1 includes the phase shift pattern PT, it is possible to determine with high accuracy the area to be masked on the projection surface SC based on the first captured image GG1. As a result, it is possible to generate a mask image M that masks the desired area on the projection surface SC. Then, by using such a mask image M for the second image G2, it is possible to measure the projection surface SC with high accuracy based on the second captured image indicated by the second captured data D2. As a result, it is possible to accurately associate coordinates in the imaging coordinate system of the camera 20 with coordinates in the display coordinate system of the projector 10. Using this association, it is possible to accurately determine the correspondence between coordinates in the display coordinate system of the projector 10 and coordinates on the projection surface SC.

[0078] In this embodiment, as described above, the second image G2 uses the phase shift pattern PT as the structured light pattern, so the phase shift pattern PT can be shared by the first image G1 and the second image G2.

[0079] As described above, step S30 of generating mask image M includes steps S31 and S32. Here, step S31 generates an amplitude image indicated by the amplitude intensity information DA based on the first captured image GG1. Step S32 determines the shape of mask image M based on the amplitude image indicated by the amplitude intensity information DA. According to steps S31 and S32, it is possible to improve the accuracy of edge detection in the first captured image GG1.

[0080] 2. Second embodiment A second embodiment of the present disclosure will be described below. In the following exemplary embodiment, for elements whose actions and functions are similar to those of the first embodiment, the reference numerals used in the description of the first embodiment will be used, and detailed descriptions of each element will be omitted as appropriate.

[0081] 11 is a block diagram of a projector 10A used in a system 100A according to the second embodiment. The system 100A is configured similarly to the system 100 of the first embodiment, except that it includes a projector 10A instead of the projector 10 of the first embodiment. The projector 10A is configured similarly to the projector 10 of the first embodiment, except that it uses a program PR2 instead of the program PR1 of the first embodiment.

[0082] The program PR2 is similar to the program PR1 of the first embodiment, except that the processing device 12 functions as the generating unit 12d instead of the generating unit 12c of the first embodiment.

[0083] The generation unit 12d is similar to the generation unit 12c of the first embodiment, except that it displays an image GU for area designation on the display device 31 and determines the mask shape using the result of the area designation in addition to the amplitude intensity image GG2.

[0084] 12 is a flowchart showing the flow of a calibration method according to the second embodiment. This calibration method is similar to the calibration method according to the first embodiment except that it includes step S20A instead of step S20 in the first embodiment and adds steps S50 and S60.

[0085] In the calibration method of this embodiment, first, in step S50, an image GU for area designation, which will be described later, is displayed on the display device 31. This display is performed by the generation unit 12d controlling the operation of the display device 31. Thereafter, in step S60, it is determined whether or not an area designation has been input. This determination is made by the generation unit 12d monitoring the input result to the terminal device 30. Step S60 is repeatedly executed until an area designation is input (step S60: NO).

[0086] If an area specification is input (step S60: YES), step S10 is executed as in the first embodiment, followed by step S20A.

[0087] Step S20A is the same as step S20 in the first embodiment, except that step S22A is executed instead of step S22 in the first embodiment. Step S22A is the same as step S22 in the first embodiment, except that the mask shape is determined using the result of the region designation in addition to amplitude intensity image GG2. Note that steps S50 and S60 only need to be executed before step S22A, and are not limited to the illustrated example. For example, steps S50 and S60 may be executed between step S10 and step S20A, or may be executed within step S10 or step S20A.

[0088] 13 is a diagram for explaining an image GU for area designation. In step S50, the image GU is displayed on the display device 31. The image GU is a GUI (graphical user interface) image for receiving area designation.

[0089] The image GU can accept adjustments to the shape and size of an area within the captured image. In the example shown in FIG. 13, multiple dots are arranged along the outline L3 of the area to be specified. The area is adjusted by, for example, using a cursor to select any one dot from the multiple dots and then moving the selected dot. Note that the display for adjusting the area is not limited to the example shown in FIG. 13 and is arbitrary.

[0090] The area designation is confirmed by operating a confirmation button displayed on the display device 31, for example, although this is not shown.

[0091] The second embodiment described above also makes it possible to achieve highly accurate calibration of the coordinate systems between the camera 20 and the projector 10. As described above, the calibration method of this embodiment includes steps S50 and S60. Step S50 causes the display device 31 to display an image for specifying an area within the projection surface SC. Step S60 receives input specifying the area from the user. Step S22A of determining the shape of the mask image M includes determining the shape of the mask image M based on the amplitude intensity image GG2 and the area. This makes it possible to improve the accuracy of edge detection in the first captured image GG1.

[0092] 3. Variations The above-described embodiments can be modified in various ways. Specific modifications that can be applied to the above-described embodiments are exemplified below. Two or more embodiments arbitrarily selected from the following examples can be combined as appropriate within the scope of not mutually contradictory.

[0093] 3-1. Variation 1 In the above-described embodiment, an example is given in which the processing device 12 of the projector 10 executes the programs PR1 and PR2, but this is not limited to this example, and for example, the processing device of a computer that is communicatively connected to the projector 10 and the camera 20 may execute the programs PR1 and PR2.

[0094] 3-2. Variation 2 In the above-described embodiment, an example is given in which the correspondence information DC is used to adjust the projection image G, but the present invention is not limited to this example. For example, the correspondence information DC may be used to display a uniform grid pattern or the like on the projection surface SC, or may be used to reflect a three-dimensional model of the projection surface SC as seen from the camera 20 in three-dimensional image editing software or the like, and then draw a picture on the model, and display how the picture looks when viewed from the projector 10 on a PC monitor or the like, or project it on the projector 10.

[0095] 4. Notes A summary of this disclosure is provided below.

[0096] (Supplementary Note 1) A first aspect, which is a preferred example of the calibration method of the present disclosure, includes projecting a first image including a phase shift pattern from a projector onto a projection surface, acquiring a first captured image including the first image and captured by a camera, generating a mask image based on the first captured image, projecting a second image including a structured light pattern and masked by the mask image from the projector onto the projection surface, acquiring a second captured image including the second image and captured by the camera, and corresponding coordinates in the imaging coordinate system of the camera to coordinates in the display coordinate system of the projector based on the second captured image.

[0097] In the above aspect, since the first image includes a phase shift pattern, it is possible to determine with high accuracy the area to be masked on the projection surface based on the first captured image. This allows for the generation of a mask image that masks a desired area on the projection surface with high accuracy. By using such a mask image for the second image, it is possible to effectively suppress unnecessary light reflection from objects around the projection surface onto the projection surface. Therefore, based on the second captured image, it is possible to accurately associate coordinates in the camera's imaging coordinate system with the projector's display coordinate system, using coordinates on the projection surface as a reference. By using this association, it is possible to accurately determine the correspondence between coordinates in the projector's display coordinate system and coordinates on the projection surface.

[0098] (Supplementary Note 2) In a second aspect which is a preferred example of the first aspect, the structured light pattern is a phase shift pattern. In the above aspect, the phase shift pattern can be shared by the first image and the second image.

[0099] (Supplementary Note 3) In a third aspect, which is a preferred example of the first or second aspect, generating the mask image includes generating an amplitude image based on the first captured image, and determining a shape of the mask image based on the amplitude image. In this aspect, it is possible to improve the accuracy of edge detection in the first captured image.

[0100] (Supplementary Note 4) In a fourth aspect, which is a preferred example of the third aspect, the method further includes displaying an image for specifying an area within the projection surface on a display device and receiving an input specifying the area from a user, and determining the shape of the mask image includes determining the shape of the mask image based on the amplitude intensity image and the area. In the above aspects, the accuracy of edge detection in the first captured image can be improved.

[0101] (Appendix 5) A fifth aspect, which is a preferred example of the system of the present disclosure, includes a camera and a projector communicatively connected to the camera, wherein the projector performs the following operations: projecting a first image including a phase shift pattern onto a projection surface; acquiring a first captured image by capturing the first image with the camera; generating a mask image based on the first captured image; projecting a second image including a structured light pattern and masked by the mask image onto the projection surface; acquiring a second captured image by capturing the second image with the camera; and corresponding, based on the second captured image, coordinates in the imaging coordinate system of the camera to coordinates in the display coordinate system of the projector.

[0102] In the above aspect, since the first image includes a phase shift pattern, it is possible to determine with high accuracy the area to be masked on the projection surface based on the first captured image. This allows for the generation of a mask image that masks a desired area on the projection surface with high accuracy. By using such a mask image for the second image, it is possible to effectively suppress unnecessary light reflection from objects around the projection surface onto the projection surface. Therefore, based on the second captured image, it is possible to accurately associate coordinates in the camera's imaging coordinate system with the projector's display coordinate system, using coordinates on the projection surface as a reference. By using this association, it is possible to accurately determine the correspondence between coordinates in the projector's display coordinate system and coordinates on the projection surface.

[0103] (Appendix 6) A sixth aspect, which is a preferred example of the program of the present disclosure, causes a computer to execute the following steps: projecting a first image including a phase shift pattern from a projector onto a projection surface; capturing the first image with a camera to obtain a first captured image; generating a mask image based on the first captured image; projecting a second image including a structured light pattern and masked by the mask image from the projector onto the projection surface; capturing the second image with the camera to obtain a second captured image; and corresponding, based on the second captured image, coordinates in the imaging coordinate system of the camera to coordinates in the display coordinate system of the projector.

[0104] In the above aspect, since the first image includes a phase shift pattern, it is possible to determine with high accuracy the area to be masked on the projection surface based on the first captured image. This allows for the generation of a mask image that masks a desired area on the projection surface with high accuracy. By using such a mask image for the second image, it is possible to effectively suppress unnecessary light reflection from objects around the projection surface onto the projection surface. Therefore, based on the second captured image, it is possible to accurately associate coordinates in the camera's imaging coordinate system with the projector's display coordinate system, using coordinates on the projection surface as a reference. By using this association, it is possible to accurately determine the correspondence between coordinates in the projector's display coordinate system and coordinates on the projection surface. [Explanation of symbols]

[0105] 10...projector, 10A...projector, 11...storage device, 12...processing device, 12a...projection control unit, 12b...imaging control unit, 12c...generation unit, 12d...generation unit, 13...communication device, 14...image processing circuit, 15...optical device, 15a...light source, 15b...display panel, 15c...optical system, 16...operation device, 20...camera, 30...terminal device, 31...display device, 100...system, 100A...system, CE...ceiling surface, D1...first imaging data, D2...second imaging data, DA...amplitude intensity information, DC...correspondence information, DG1...first image information, DG2...second image information, DM...mask information, FL...floor surface, G...projected image, G1...first image, G1-1...first image, G1-2...first image, G1-3...first image, G1-4...first image, G2...second image, GG ...captured image, GG1...first captured image, GG2...amplitude intensity image, GG3...mask shape image, GU...image, IMG...video data, L1...line, L2...line, L3...contour, M...mask image, MC...mask image, MP...mask image, OP...transparent area, PR1...program, PR2...program, PT...phase shift pattern, RC...area, RM...non-transparent area, RP...area, S10...step, S11...step, S12...step, S13...step, S20...step, S20A...step, S21...step, S22...step, S22A...step, S30...step, S31...step, S32...step, S33...step, S50...step, S60...step, SC...projection surface, WA1...wall surface, WA2...wall surface, WA3...wall surface.

Claims

1. projecting a first image including a phase shift pattern from a projector onto a projection surface; acquiring a first captured image including the first image and captured by a camera; generating a mask image based on the first captured image; projecting a second image, including a structured light pattern and masked by the mask image, from the projector onto the projection surface; obtaining a second captured image including the second image and captured by the camera; and associating coordinates in an imaging coordinate system of the camera with coordinates in a display coordinate system of the projector based on the second captured image. Calibration method.

2. the structured light pattern is a phase shift pattern; The calibration method according to claim 1 .

3. generating the mask image generating an amplitude image based on the first captured image; determining a shape of the mask image based on the amplitude intensity image; The calibration method according to claim 1 or 2.

4. displaying an image for designating an area within the projection surface on a display device; receiving an input from a user specifying the area; Determining the shape of the mask image includes: determining a shape of the mask image based on the amplitude intensity image and the region; The calibration method according to claim 3 .

5. A camera and a projector communicatively connected to the camera, The projector includes: projecting a first image including a phase shift pattern onto a projection surface; acquiring a first captured image by capturing the first image with the camera; generating a mask image based on the first captured image; projecting a second image onto the projection surface, the second image including a structured light pattern and masked by the mask image; acquiring a second captured image by capturing the second image with the camera; and associating coordinates in an imaging coordinate system of the camera with coordinates in a display coordinate system of the projector based on the second captured image. system.

6. projecting a first image including a phase shift pattern from a projector onto a projection surface; capturing the first image with a camera to obtain a first captured image; generating a mask image based on the first captured image; projecting a second image, including a structured light pattern and masked by the mask image, from the projector onto the projection surface; capturing the second image with the camera to obtain a second captured image; and causing a computer to execute the steps of: associating coordinates in an imaging coordinate system of the camera with coordinates in a display coordinate system of the projector based on the second captured image. program.

Images