Correction method, system, and program

Abstract

To reduce complexity of projection position correction calculation.SOLUTION: A correction method includes: acquiring a first picked-up image obtained by imaging a projection image and a plane in a first period; acquiring a second picked-up image obtained by imaging the projection image and the plane in a second period later than the first period; extracting a first feature point and a second feature point of the plane in the first picked-up image; extracting a third feature point corresponding to the first feature point of the plane and a fourth feature point corresponding to the second feature point in the second picked-up image; calculating a first normal vector of the plane in the first period; calculating a second normal vector of the plane in the second period; and correcting the position of the projection image relative to the plane in the second period to the position of the projection image relative to the plane in the first period, on the basis of a group of the first feature point, second feature point, and first normal vector, and a group of the third feature point, fourth feature point, and second normal vector.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a correction method, a system, and a program. [Background technology]

[0002] For example, Patent Document 1 discloses a projector that derives projective transformation parameters based on four or more first feature points extracted from a first captured image captured in a first period and four or more second feature points extracted from a second captured image captured in a second period. These projective transformation parameters are used to derive panel transformation parameters, which are transformation matrices for returning the positional relationship between the projection surface and the projected image to the initial positional relationship before the device body was moved, for example, in the first period. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-092169 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, even when the projection surface is flat, four or more first feature points and four or more second feature points are required to derive the projective transformation parameters, but depending on the presence of obstacles, the angle of view of the imaging device, the shape of the screen, etc., it may be difficult to find the four feature points. [Means for solving the problem]

[0005] A correction method according to one aspect of the present disclosure is a correction method for correcting a projection image projected onto a plane by a projector, the correction method including: acquiring a first captured image obtained by capturing the projection image and the plane by an imaging device during a first period; acquiring a second captured image obtained by capturing the projection image and the plane by the imaging device during a second period after the first period; extracting first feature points and second feature points of the plane in the first captured image; and extracting third feature points and second feature points corresponding to the first feature points of the plane in the second captured image. calculating a first normal vector that is a normal vector of the plane in the first period; calculating a second normal vector that is a normal vector of the plane in the second period; and correcting a position of the projected image with respect to the plane in the second period to a position of the projected image with respect to the plane in the first period based on a set of the first feature point, the second feature point, and the first normal vector, and a set of the third feature point, the fourth feature point, and the second normal vector.

[0006] A system according to one aspect of the present disclosure includes a projector that projects a projection image onto a plane, and an imaging device that captures the plane and the projection image, wherein the projector acquires a first captured image obtained by capturing the projection image and the plane using the imaging device during a first period, acquires a second captured image obtained by capturing the projection image and the plane using the imaging device during a second period after the first period, extracts first feature points and second feature points of the plane in the first captured image, and extracts the first feature points of the plane in the second captured image. and a fourth feature point corresponding to the second feature point; calculating a first normal vector that is a normal vector of the plane in the first period; calculating a second normal vector that is a normal vector of the plane in the second period; and correcting the position of the projected image with respect to the plane in the second period to the position of the projected image with respect to the plane in the first period based on a set of the first feature point, the second feature point, and the first normal vector, and a set of the third feature point, the fourth feature point, and the second normal vector.

[0007] A program according to one aspect of the present disclosure is a program for correcting a projection image projected onto a plane by a projector, the program comprising: acquiring a first captured image obtained by capturing the projection image and the plane by an imaging device during a first period; acquiring a second captured image obtained by capturing the projection image and the plane by the imaging device during a second period after the first period; extracting first feature points and second feature points of the plane in the first captured image; and extracting third feature points and second feature points corresponding to the first feature points of the plane in the second captured image. and extracting a corresponding fourth feature point from the plane during the first period, calculating a first normal vector that is a normal vector of the plane during the first period, calculating a second normal vector that is a normal vector of the plane during the second period, and correcting the position of the projected image with respect to the plane during the second period to the position of the projected image with respect to the plane during the first period based on a set of the first feature point, the second feature point, and the first normal vector, and a set of the third feature point, the fourth feature point, and the second normal vector. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an outline of a system used in a correction method according to an embodiment. [Figure 2] FIG. 1 is a block diagram of a projector used in a system according to an embodiment. [Figure 3] 1 is a flowchart illustrating a flow of a correction method according to an embodiment. [Figure 4] FIG. 2 is a diagram for explaining acquisition of a first captured image. [Figure 5] FIG. 2 is a diagram for explaining a first captured image. [Figure 6] FIG. 10 is a diagram for explaining acquisition of a second captured image. [Figure 7] FIG. 10 is a diagram for explaining a second captured image. [Figure 8] FIG. 2 is a diagram for explaining the relationship between the first feature point, the second feature point, the third feature point, and the fourth feature point. [Figure 9] 10A and 10B are diagrams for explaining correction of the position of a projected image. DETAILED DESCRIPTION OF THE INVENTION

[0009] Preferred embodiments of the present disclosure will be described below with reference to the accompanying drawings. Note that the dimensions and scale of each part in the drawings may differ from the actual dimensions and are shown schematically to facilitate understanding. Furthermore, the scope of the present disclosure is not limited to these embodiments unless otherwise specified in the following description to the effect that the present disclosure is limited.

[0010] 1. First embodiment 1-1. System Overview 1 is a diagram showing an outline of a system 100 used in a correction method according to an embodiment. The system 100 is a projection system that projects a projection image G onto a projection surface SC.

[0011] The projection surface SC is an example of a "plane" and is formed by the surface of an object such as a screen. In the example shown in FIG. 1, the outer shape of the projection surface SC is rectangular. Note that the outer shape of the projection surface SC is not limited to the example shown in FIG. 1 and is arbitrary. Furthermore, the projection surface SC is not limited to being a strict plane, and may be a surface that is slightly curved or distorted to the extent that it can be roughly perceived as a plane.

[0012] In this embodiment, a first physical marker MK-1 and a second physical marker MK-2 are removably installed at positions corresponding to two adjacent corners of the four corners of the projection surface SC. Each of the first physical marker MK-1 and the second physical marker MK-2 is a mark having a reflectance different from that of the projection surface SC or the background of the projection surface SC, for example, a sticker of a color different from that of the projection surface SC or the background of the projection surface SC. The first physical marker MK-1 is used to extract a first feature point P1 and a third feature point P3, which will be described later. Meanwhile, the second physical marker MK-2 is used to extract a second feature point P2 and a fourth feature point P4, which will be described later.

[0013] The first physical marker MK-1 and the second physical marker MK-2 are not limited to being removably installed on the projection surface SC, and may be fixedly installed on the projection surface SC. The first physical marker MK-1 and the second physical marker MK-2 may be provided as needed and omitted. In this case, two corners of the projection surface SC are used instead of the first physical marker MK-1 and the second physical marker MK-2 to extract the first feature point P1, the second feature point P2, the third feature point P3, and the fourth feature point P4, which will be described later.

[0014] As shown in FIG. 1, the system 100 includes a projector 10, an image capturing device 20, and a terminal device 30.

[0015] The projector 10 is a display device that projects a projection image G represented by video data IMG output from a terminal device 30 onto a projection surface SC. In the example shown in FIG. 1, the projection image G is projected onto a rectangular area that covers substantially the entire projection surface SC. Here, the projection image G is contained within the projection surface SC, and the four corners of the projection image G are positioned near the four corners of the projection surface SC. Note that the projection position and shape of the projection image G relative to the projection surface SC are not limited to the example shown in FIG. 1 and are arbitrary. However, when two corners of the projection surface SC are used instead of the first physical marker MK-1 and the second physical marker MK-2 to extract the first feature point P1, the second feature point P2, the third feature point P3, and the fourth feature point P4 described below, it is preferable that the two corners located at both ends of one side of the projection image G be positioned near the two corners located at both ends of the corresponding side of the projection surface SC.

[0016] The projector 10 of this embodiment has a function of controlling the operation of the image capturing device 20 and a function of correcting the projection position and shape of the projected image G using the image capturing result of the image capturing device 20.

[0017] The imaging device 20 is a digital camera having an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor).

[0018] The imaging device 20 captures an image of the area RC. The area RC is an area that includes the projection image G projected onto the projection surface SC. In this way, the imaging device 20 captures an image of the projection surface SC and the projection image G. Note that the imaging device 20 may be a component of the projector 10.

[0019] The terminal device 30 is a computer that has a function of supplying video data IMG to the projector 10. In the example shown in Fig. 1, the terminal device 30 is a notebook computer. Note that the terminal device 30 is not limited to a notebook computer, and may be, for example, a desktop computer, a smartphone, a tablet terminal, a video playback device, a DVD (Digital Versatile Disk) player, a Blu-ray Disc player, a hard disk recorder, a television tuner device, a CATV (Cable television) set-top box, a video game console, or the like.

[0020] 1-2.Projector Fig. 2 is a block diagram of a projector 10 used in a system 100 according to the embodiment. In addition to the projector 10, Fig. 2 shows the connection state of an imaging device 20 and a terminal device 30 to the projector 10.

[0021] 2, the projector 10 has a storage device 11, a processing device 12, a communication device 13, an image processing circuit 14, an optical device 15, and an operation device 16. These are connected to each other so that they can communicate with each other. The processing device 12 is an example of a "computer."

[0022] The storage device 11 is a storage device that stores programs executed by the processing device 12 and data processed by the processing device 12. The storage device 11 includes, for example, a hard disk drive or a semiconductor memory. Note that part or all of the storage device 11 may be provided in an external storage device or server outside the projector 10.

[0023] The memory device 11 stores a program PR1, first imaging data D1, second imaging data D2, first adjustment point cloud information DP1, second adjustment point cloud information DP2, third adjustment point cloud information DP3, fourth adjustment point cloud information DP4, first normal vector information DV1, second normal vector information DV2, first feature vector information DV3, second feature vector information DV4, first orthogonal vector information DV5, second orthogonal vector information DV6, first adjustment point cloud information DP1, second adjustment point cloud information DP2, and correction parameter information DC.

[0024] The program PR1 is a program for correcting the projection image G projected by the projector 10 onto the projection surface SC.

[0025] The first imaging data D1 is information representing a first imaging image GG1 (described later) acquired by imaging the projection surface SC by the imaging device 20 during a first period (described later). The second imaging data D2 is information representing a second imaging image GG2 (described later) acquired by imaging the projection surface SC by the imaging device 20 during a second period (described later).

[0026] The first adjustment point cloud information DP1 is information indicating a first feature point P1 (described later) that is a feature point of the projection surface SC in a first captured image GG1 (described later) that is represented by the first imaging data D1. The second adjustment point cloud information DP2 is information indicating a second feature point P2 (described later) that is a feature point of the projection surface SC in a first captured image GG1 (described later) that is represented by the first imaging data D1. The third adjustment point cloud information DP3 is information indicating a third feature point P3 (described later) that is a feature point of the projection surface SC in a second captured image GG2 (described later) that is represented by the second imaging data D2. The fourth adjustment point cloud information DP4 is information indicating a fourth feature point P4 (described later) that is a feature point of the projection surface SC in a second captured image GG2 (described later) that is represented by the second imaging data D2.

[0027] The first normal vector information DV1 is information indicating a first normal vector Vn1, which will be described later. The second normal vector information DV2 is information indicating a second normal vector Vn2, which will be described later. The first feature vector information DV3 is information indicating a first feature vector Va1, which will be described later. The second feature vector information DV4 is information indicating a second feature vector Va2, which will be described later. The first orthogonal vector information DV5 is information indicating a first orthogonal vector Vb1, which will be described later. The second orthogonal vector information DV6 is information indicating a second orthogonal vector Vb2, which will be described later.

[0028] The first adjustment point cloud information DP1 is information indicating, using coordinate values in a three-dimensional coordinate system, the positions of a plurality of adjustment points PR (described below) set in the projection image G on the projection surface SC during a first period (described below) and a period before correction within a second period (described below). The second adjustment point cloud information DP2 is information indicating, using coordinate values in a three-dimensional coordinate system, the positions of a plurality of adjustment points PR (described below) set in the projection image G on the projection surface SC during a period after correction within the second period (described below). The correction parameter information DC is information indicating parameters of an arithmetic formula for performing coordinate conversion to correct the position of the projection image G on the projection surface SC during the second period (described below) to the position of the projection image G on the projection surface SC during the first period (described below). The position indicated by the second adjustment point cloud information DP2 is obtained from the position indicated by the first adjustment point cloud information DP1 through this coordinate conversion.

[0029] The processing device 12 has the function of controlling each unit of the projector 10 and the function of processing various data. The processing device 12 includes, for example, a processor such as a CPU (Central Processing Unit). The processing device 12 may be configured with a single processor or multiple processors. Some or all of the functions of the processing device 12 may be realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The processing device 12 may be integrated with at least a part of the image processing circuit 14.

[0030] The communication device 13 is a communication device capable of communicating with various devices, and acquires video data IMG from the terminal device 30 and communicates with the imaging device 20. For example, the communication device 13 is a wired communication device such as a wired LAN (Local Area Network), USB (Universal Serial Bus), or HDMI (High Definition Multimedia Interface), or a wireless communication device such as LPWA (Low Power Wide Area), wireless LAN including Wi-Fi, or Bluetooth. "HDMI," "Wi-Fi," and "Bluetooth" are all registered trademarks.

[0031] The image processing circuit 14 is a circuit that performs necessary processing on the video data IMG from the communication device 13 and inputs the data to the optical device 15. The image processing circuit 14, for example, has a frame memory (not shown), and loads the video data IMG into the frame memory, appropriately performs various processes such as resolution conversion, resizing, and distortion correction, and inputs the data to the optical device 15. Here, the various processes appropriately use the first adjustment point cloud information DP1 or the second adjustment point cloud information DP2 described above. Note that the image processing circuit 14 may also perform processes such as OSD (On Screen Display) processing, which generates image information for menu display or operation guides, etc., and combines it with the video data IMG, as necessary.

[0032] The optical device 15 is a device that projects image light onto the projection surface SC, and includes a light source 15a, a display panel 15b, and an optical system 15c.

[0033] The light source 15a includes a light source such as a halogen lamp, a xenon lamp, an ultra-high pressure mercury lamp, an LED (Light Emitting Diode), or a laser light source, and emits red, green, and blue light, respectively. The display panel 15b is an optical modulator including three light modulation elements corresponding to red, green, and blue. Each light modulation element includes, for example, a transmissive liquid crystal panel, a reflective liquid crystal panel, or a DMD (Digital Mirror Device), and generates image light of each color by modulating light of the corresponding color. The image light of each color generated by the display panel 15b is combined by a color combining optical system to become full-color image light. The optical system 15c is a projection optical system including a projection lens and the like that forms and projects the full-color image light from the display panel 15b onto a projection surface SC.

[0034] The operation device 16 is a device that accepts operations from the user. For example, the operation device 16 includes an operation panel and a remote control receiver, not shown. The operation panel is provided on the exterior housing of the projector 10, and outputs a signal based on an operation from the user. The remote control receiver receives an infrared signal from a remote control, not shown, decodes the infrared signal, and outputs a signal based on the operation of the remote control. The operation device 16 may be provided as needed, or may be omitted.

[0035] In the above-described projector 10, the processing device 12 functions as a projection control unit 12a, an imaging control unit 12b, and a correction unit 12c by executing the program PR1 stored in the storage device 11. Therefore, the processing device 12 includes the projection control unit 12a, the imaging control unit 12b, and the correction unit 12c.

[0036] The projection control unit 12a controls the operations of the image processing circuit 14 and the optical device 15. More specifically, the projection control unit 12a controls the operation of the optical device 15 to project the projection image G onto the projection surface SC during each of a first period and a second period, which will be described later.

[0037] The imaging control unit 12b controls the operation of the imaging device 20. More specifically, the imaging control unit 12b causes the imaging device 20 to capture an image of the projection surface SC during each of a first period and a second period, which will be described later, to acquire first imaging data D1 and second imaging data D2.

[0038] The correction unit 12c corrects the position of the projection image G relative to the projection surface SC based on the first imaging data D1 and the second imaging data D2.

[0039] More specifically, the correction unit 12c generates first normal vector information DV1, first adjustment point cloud information DP1, and second adjustment point cloud information DP2 based on the first imaging data D1, then generates first feature vector information DV3 based on the first adjustment point cloud information DP1 and the second adjustment point cloud information DP2, and then generates first orthogonal vector information DV5 based on the first normal vector information DV1 and the first feature vector information DV3.

[0040] In addition, the correction unit 12c generates second normal vector information DV2, third adjustment point cloud information DP3, and fourth adjustment point cloud information DP4 based on the second imaging data D2, then generates second feature vector information DV4 based on the third adjustment point cloud information DP3 and the fourth adjustment point cloud information DP4, and then generates second orthogonal vector information DV6 based on the second normal vector information DV2 and the second feature vector information DV4.

[0041] Furthermore, the correction unit 12c generates correction parameter information DC based on the first normal vector information DV1, the first feature vector information DV3, the first orthogonal vector information DV5, the second normal vector information DV2, the second feature vector information DV4, and the second orthogonal vector information DV6.

[0042] Furthermore, the corrector 12c generates second adjustment point cloud information DP2 based on the first adjustment point cloud information DP1 and the correction parameter information DC.

[0043] 1-3. Correction method 3 is a flowchart showing the flow of a correction method according to an embodiment. The correction method is performed by the processing device 12 executing the program PR1 using the above-described system 100. As shown in FIG. 3, the correction method includes steps S1 to S9, and corrects the projection image G projected onto the projection surface SC by the projector 10. Here, step S9 includes steps S10 to S14.

[0044] More specifically, first, in step S1, the corrector 12c determines whether or not it is the first period. This determination is made, for example, based on whether or not it is a predetermined timing after the projector 10 has been started and adjustment using an adjustment point PR, which will be described later, has been completed. If it is the predetermined timing, it is determined that it is the first period. This step S1 is repeated until it is determined that it is the first period (step S1: NO).

[0045] If it is determined that it is the first period (step S1: YES), in step S2, the projection control unit 12a causes the image processing circuit 14 and the optical device 15 to project a projection image G-1 (described later) onto the projection surface SC, and the imaging control unit 12b causes the imaging device 20 to capture the projection image G-1 (described later) and the projection surface SC, thereby acquiring a first captured image GG1 (described later). As a result, the imaging control unit 12b acquires first captured data D1.

[0046] After step S2, in step S3, the correction unit 12c determines whether it is the second period. This determination is made, for example, based on whether a predetermined time has elapsed since the completion of step S2, whether the optical device 15 has been adjusted since the completion of step S2, or whether one or both of the position and attitude of the projector 10 have changed by a predetermined amount or more since the completion of step S2. Here, if a predetermined time has elapsed since the completion of step S2, if the optical device 15 has been adjusted since the completion of step S2, or if one or both of the position and attitude of the projector 10 have changed by a predetermined amount or more since the completion of step S2, it is determined that it is the second period. This step S3 is repeated until it is determined that it is the second period (step S3: NO).

[0047] If it is determined that it is the second period (step S3: YES), in step S4, the projection control unit 12a causes the image processing circuit 14 and the optical device 15 to project a projection image G-2 (described later) onto the projection surface SC, and the imaging control unit 12b causes the imaging device 20 to capture the projection image G-2 (described later) and the projection surface SC, thereby acquiring a second captured image GG2 (described later). As a result, the imaging control unit 12b acquires second captured data D2.

[0048] After step S4, in step S5, the correction unit 12c extracts a first feature point P1 and a second feature point P2, which will be described later. This extraction is performed based on a first captured image GG1, which will be described later in detail with reference to Fig. 6. Note that step S5 may be performed after step S2, or may be performed before the above-mentioned step S4.

[0049] After step S5, in step S6, the correction unit 12c extracts a third feature point P3 and a fourth feature point P4, which will be described later. This extraction is performed based on a second captured image GG2, which will be described later in detail with reference to Fig. 6. Note that step S6 may be performed after step S4, or may be performed before the above-mentioned step S5.

[0050] After step S6, in step S7, the correction unit 12c calculates a first normal vector Vn1, which will be described later. This calculation is performed based on a first captured image GG1, which will be described later in detail with reference to Fig. 6. Note that step S7 may be performed after step S2, or may be performed before the above-mentioned step S6.

[0051] After step S7, in step S8, the correction unit 12c calculates a second normal vector Vn2, which will be described later. This calculation is performed based on a second captured image GG2, which will be described later in detail with reference to Fig. 6. Note that step S8 may be performed after step S4, or may be performed before the above-mentioned step S7.

[0052] After step S8, in step S9, the correction unit 12c corrects the position of the projected image G. This correction is performed based on a set of the first feature point P1, the second feature point P2, and the first normal vector Vn1, and a set of the third feature point P3, the fourth feature point P4, and the second normal vector Vn2, as will be described in detail later with reference to FIG.

[0053] Specifically, in step S10, the correction unit 12c calculates a first feature vector Va1, which will be described later. This calculation is performed based on a first feature point P1 and a second feature point P2, as will be described later in detail with reference to FIG.

[0054] After step S10, in step S11, the correction unit 12c calculates a first orthogonal vector Vb1, which will be described later. This calculation is performed based on the first normal vector Vn1 and the first feature vector Va1, as will be described in detail later with reference to FIG.

[0055] After step S11, in step S12, the correction unit 12c calculates a second feature vector Va2, which will be described later. This calculation is performed based on the third feature point P3 and the fourth feature point P4, as will be described in detail later with reference to Fig. 6. Note that step S12 may be performed after step S6, or may be performed before the above-mentioned step S11.

[0056] After step S12, in step S13, the correction unit 12c calculates a second orthogonal vector Vb2, which will be described later. This calculation is performed based on the second normal vector Vn2 and the second feature vector Va2, as will be described in detail later with reference to Fig. 6. Note that step S13 may be executed after step S12.

[0057] After step S13, in step S14, the correction unit 12c calculates correction parameters. As a result, correction parameter information DC indicating the correction parameters is obtained. This calculation is performed based on the first feature point P1, the second feature point P2, the first normal vector Vn1, the first feature vector Va1, the first orthogonal vector Vb1, the third feature point P3, the fourth feature point P4, the second normal vector Vn2, the second feature vector Va2, and the second orthogonal vector Vb2, as will be described in detail later with reference to FIG.

[0058] In step S9, after steps S10 to S14, the correction unit 12c corrects a plurality of adjustment points PR (described later) using an arithmetic expression to which the correction parameters indicated by the correction parameter information DC are applied, thereby correcting the position of the projection image G.

[0059] Fig. 4 is a diagram for explaining acquisition of the first captured image GG1. Fig. 5 is a diagram for explaining the first captured image GG1. Fig. 4 shows an example of a state in which a projection image G-1, which is a projection image G in a first period, is projected onto a projection surface SC.

[0060] The projection image G-1 has an outer shape adjusted using a plurality of adjustment points PR, which will be described later. In the example shown in Fig. 4, the outer shape of the projection image G-1 approximately matches the outer shape of the projection surface SC. Note that the outer shape of the projection image G-1 in the first period is not limited to the example shown in Fig. 4 and may be any shape.

[0061] In step S2, as shown in Fig. 4, the imaging control unit 12b causes the imaging device 20 to capture an image of the projection image G-1 and the projection surface SC during a first period. As a result, step S2 executes acquiring a first captured image GG1. As shown in Fig. 5, the first captured image GG1 of this embodiment captures not only the projection image G-1 and the projection surface SC, but also the first physical marker MK-1 and the second physical marker MK-2.

[0062] Here, the projection image G-1 is not particularly limited as long as it is an image that can measure the attitude of the projection surface SC as seen from the projector 10 or the imaging device 20 based on the first captured image GG1, but preferably it is an image that can measure the three-dimensional shape of the projection surface SC based on the first captured image GG1, and is, for example, a structured light pattern such as a phase shift pattern, a binary code pattern, a dot pattern, a rectangular pattern, a polygonal pattern, a checkered pattern, a gray code pattern, or a random dot pattern.

[0063] Fig. 6 is a diagram for explaining acquisition of the second captured image GG2. Fig. 7 is a diagram for explaining the second captured image GG2. Fig. 6 shows an example of a state in which a projection image G-2, which is a projection image G in a second period, is projected onto the projection surface SC.

[0064] The projection image G-2 has a different outer shape from the aforementioned projection image G-2 on the projection surface SC due to a change in either or both of the position and the attitude of the projector 10. Note that the outer shape of the projection image G-2 in the second period is not limited to the example shown in FIG. 6 and may be any shape.

[0065] In step S4, as shown in Fig. 6, the imaging control unit 12b causes the imaging device 20 to capture an image of the projection image G-2 and the projection surface SC in a second period that is later than the first period. As a result, step S4 executes acquiring a second captured image GG2. As shown in Fig. 7, the second captured image GG2 of this embodiment includes the first physical marker MK-1 and the second physical marker MK-2 in addition to the projection image G-2 and the projection surface SC.

[0066] Here, the projected image G-2 is not particularly limited as long as it is an image that makes it possible to measure the attitude of the projection surface SC as seen from the projector 10 or the imaging device 20 based on the second captured image GG2, but is preferably an image that makes it possible to measure the three-dimensional shape of the projection surface SC based on the second captured image GG2, and is, for example, a structured light pattern such as a phase shift pattern, a binary code pattern, a dot pattern, a rectangular pattern, a polygonal pattern, a checkered pattern, a gray code pattern, or a random dot pattern. Note that the projected image G-2 may be the same as or different from the projected image G-1.

[0067] Fig. 8 is a diagram for explaining the relationship between the first feature point P1, the second feature point P2, the third feature point P3, and the fourth feature point P4. Fig. 8 shows the first feature point P1, the second feature point P2, the third feature point P3, and the fourth feature point P4 in a three-dimensional coordinate system that is an orthogonal coordinate system of the X-axis, the Y-axis, and the Z-axis.

[0068] In step S5, the correction unit 12c extracts a first feature point P1 and a second feature point P2 of the projection surface SC in the first captured image GG1. Similarly, in step S6, the correction unit 12c extracts a third feature point P3 and a fourth feature point P4 of the projection surface SC in the second captured image GG2. The third feature point P3 corresponds to the first feature point P1 of the projection surface SC in the second captured image GG2. The fourth feature point P4 corresponds to the second feature point P2 of the projection surface SC in the second captured image GG2.

[0069] These feature points are extracted by, for example, using a known image recognition technique to detect two predetermined points on the projection surface SC as coordinate values in the two-dimensional imaging coordinate system of the imaging device 20. The detected coordinate values are converted into coordinate values in a three-dimensional coordinate system using a predetermined conversion formula. The three-dimensional coordinate value A of the first feature point P1 is expressed as (Ax, Ay, Az), the three-dimensional coordinate value B of the second feature point P2 is expressed as (Bx, By, Bz), the three-dimensional coordinate value A' of the third feature point P3 is expressed as (A'x, A'y, A'z), and the three-dimensional coordinate value B' of the fourth feature point P4 is expressed as (B'x, B'y, B'z).

[0070] In this embodiment, in step S5, a first physical marker MK-1 and a second physical marker MK-2 are detected as two predetermined points on the projection surface SC. Similarly, in step S6, the first physical marker MK-1 and the second physical marker MK-2 are detected as two predetermined points on the projection surface SC. Here, the first feature point P1 and the third feature point P3 are each extracted based on the light reflected from the first physical marker MK-1. Meanwhile, the second feature point P2 and the fourth feature point P4 are each extracted based on the light reflected from the second physical marker MK-2. In this way, by using physical markers, the accuracy of feature point extraction can be improved regardless of the contrast difference between the projection surface SC and the background, etc.

[0071] In step S7, the correction unit 12c calculates a first normal vector Vn1. The first normal vector Vn1 is a normal vector of the projection surface SC during the first period. In step S7, the correction unit 12c measures the three-dimensional shape of the projection surface SC based on the first captured image GG1 and calculates the first normal vector Vn1 based on the measurement result. Here, the measurement result of the three-dimensional shape is expressed as, for example, three-dimensional plane parameters (a, b, c) of the projection surface SC. The first normal vector Vn1 is obtained by normalizing the normal vector (a, b, c) to a length of 1. Therefore, a, b, and c satisfy the relationship aX+bY+cZ=1 in the coordinate values (X, Y, Z) of the three-dimensional coordinate system.

[0072] Similarly, in step S8, the correction unit 12c calculates a second normal vector Vn2. The second normal vector Vn2 is a normal vector of the projection surface SC during the second period. In step S8, the correction unit 12c measures the three-dimensional shape of the projection surface SC based on the second captured image GG2, and calculates the second normal vector Vn2 based on the measurement result. Here, the measurement result of the three-dimensional shape is expressed as, for example, three-dimensional plane parameters (a', b', c') of the projection surface SC. The second normal vector Vn2 is obtained by normalizing the normal vector (a', b', c') to a length of 1. Therefore, a', b', and c' satisfy the relationship a'X + b'Y + c'Z = 1 in the coordinate values (X, Y, Z) of the three-dimensional coordinate system.

[0073] In step S10, the correction unit 12c calculates a first feature vector Va1. The first feature vector Va1 is a vector directed from the first feature point P1 to the second feature point P2. The first feature vector Va1 is calculated based on the first feature point P1 and the second feature point P2. For example, the first feature vector Va1 can be obtained by normalizing the vector obtained by subtracting the three-dimensional coordinate value A of the first feature point P1 from the three-dimensional coordinate value B of the second feature point P2 to a length of 1.

[0074] Similarly, in step S12, the correction unit 12c calculates a second feature vector Va2. The second feature vector Va2 is a vector directed from the third feature point P3 to the fourth feature point P4. The second feature vector Va2 is calculated based on the third feature point P3 and the fourth feature point P4. For example, the second feature vector Va2 is obtained by normalizing the vector obtained by subtracting the three-dimensional coordinate value A' of the third feature point P3 from the three-dimensional coordinate value B' of the fourth feature point P4 to a length of 1.

[0075] In step S11, the correction unit 12c calculates a first orthogonal vector Vb1. The first orthogonal vector Vb1 is a vector that is orthogonal to both the first normal vector Vn1 and the first feature vector Va1. The first orthogonal vector Vb1 is calculated based on the first normal vector Vn1 and the first feature vector Va1. For example, the first orthogonal vector Vb1 is obtained by calculating the cross product of the first normal vector Vn1 and the first feature vector Va1. Here, the first orthogonal vector Vb1 is a vector normalized to a length of 1.

[0076] Similarly, in step S13, the correction unit 12c calculates a second orthogonal vector Vb2. The second orthogonal vector Vb2 is a vector that is orthogonal to both the second normal vector Vn2 and the second feature vector Va2. The second orthogonal vector Vb2 is calculated based on the second normal vector Vn2 and the second feature vector Va2. For example, the second orthogonal vector Vb2 is obtained by calculating the cross product of the second normal vector Vn2 and the second feature vector Va2. Here, the second orthogonal vector Vb2 is a vector normalized to a length of 1.

[0077] In step S14, the correction unit 12c calculates R and s as correction parameters in the following calculation formula. P´m=sR(Pm-A)+A´

[0078] Here, R is a 3×3 rotation matrix that converts the mutually orthogonal first normal vector Vn1, first feature vector Va1, and first orthogonal vector Vb1 into the second normal vector Vn2, second feature vector Va2, and second orthogonal vector Vb2, respectively.

[0079] When the distance between the first feature point P1 and the second feature point P2 is c and the distance between the third feature point P3 and the fourth feature point P4 is c', s is the ratio c / c'.

[0080] Pm is the three-dimensional coordinate value of the mth image among N pixels in the projection image G-1 on the projection surface SC. P'm is the three-dimensional coordinate value of the mth image among N pixels in the projection image G-2 on the projection surface SC. N is a natural number equal to or greater than 3, and m is a natural number equal to or greater than 1 and equal to or less than N. The three-dimensional coordinate values of the N pixels are, for example, the three-dimensional coordinate values of an adjustment point PR described below.

[0081] Fig. 9 is a diagram for explaining the correction of the position of the projected image G. In Fig. 9, a third feature point P3, a fourth feature point P4, and a plurality of adjustment points PR before correction are shown in a three-dimensional coordinate system.

[0082] Before the first period, the projection image G is adjusted on the projection surface SC using a plurality of adjustment points PR. The plurality of adjustment points PR correspond to a plurality of pixels in the projection image G.

[0083] Each adjustment point PR can be moved by a user operation as needed. When two or more adjustment points PR selected by the user are moved among the multiple adjustment points PR, the portion of the projection image G corresponding to the interval between the multiple adjustment points PR is deformed in accordance with the change in the interval between the multiple adjustment points PR. This makes it possible to adjust the shape of the projection image G to fit the shape of the projection surface SC. By performing such adjustment before step S1, first adjustment point group information DP1 is obtained.

[0084] During the period before correction in the second period, a projection image G-2 based on the coordinate values indicated by the first adjustment point group information DP1 is projected onto the projection surface SC, so that, as shown in Figure 9, the multiple adjustment points PR are shifted from the desired positions relative to the third feature point P3 and the fourth feature point P4.

[0085] In step S9, the correction unit 12c corrects the position of the projection image G relative to the projection surface SC using the aforementioned arithmetic expression. Specifically, after step S14, the correction unit 12c sets the three-dimensional coordinate values indicated by the first adjustment point cloud information DP1 as three-dimensional coordinate values Pm, and calculates the three-dimensional coordinate values indicated by the second adjustment point cloud information DP2 as three-dimensional coordinate values P'm using the aforementioned arithmetic expression. By projecting the projection image G using such second adjustment point cloud information DP2, the position of the projection image G-2 relative to the projection surface SC in the second period is corrected to the position of the projection image G-1 relative to the projection surface SC in the first period.

[0086] As described above, in step S9, the correction unit 12c corrects the position of the projection image G relative to the projection surface SC in the second period to the position of the projection image G relative to the projection surface SC in the first period based on the set of the first feature point P1, the second feature point P2, and the first normal vector Vn1, and the set of the third feature point P3, the fourth feature point P4, and the second normal vector Vn2.

[0087] As described above, step S9 corrects the position of the projected image G-2 on the projection surface SC in the second period to the position of the projected image G-1 on the projection surface SC in the first period based on the difference between the first normal vector Vn1 and the second normal vector Vn2, the difference between the first feature vector Va1 and the second feature vector Va2, and the difference between the first orthogonal vector Vb1 and the second orthogonal vector Vb2. This makes it possible to calculate the matrix R for correcting the projected image G by a relatively simple calculation, and then correct the position of the projected image G.

[0088] As described above, the first normal vector Vn1, the second normal vector Vn2, the first feature vector Va1, the second feature vector Va2, the first orthogonal vector Vb1, and the second orthogonal vector Vb2 are normalized vectors whose magnitude is normalized to 1. This makes it possible to maintain the aspect ratio and the like during restoration.

[0089] The above correction method can calculate a matrix for correcting a projected image based on two feature points in each captured image, thereby reducing the complexity of the matrix calculation.

[0090] In this embodiment, physical markers are used to detect feature points on the projection surface SC, but the number of physical markers that the user must place can be reduced compared to the aspect of Patent Document 1. This reduces the complexity for the user.

[0091] 2. Variations The above-described embodiments can be modified in various ways. Specific modifications that can be applied to the above-described embodiments are exemplified below. Two or more embodiments arbitrarily selected from the following examples can be combined as appropriate within the scope of not mutually contradictory.

[0092] 2-1. Variation 1 In the above-described embodiment, an example is given in which the processing device 12 of the projector 10 executes the program PR1, but this is not limited to this example, and for example, the processing device of a computer or a terminal device 30 that is communicatively connected to the projector 10 and the imaging device 20 may execute the program PR1.

[0093] 2-2. Variation 2 In the above-described embodiment, an example is given in which the first orthogonal vector Vb1 and the second orthogonal vector Vb2 are used to correct the projection position of the projection image G, but this is not limited to this example. For example, instead of the first orthogonal vector Vb1, a vector that is not orthogonal to the first normal vector Vn1 and the first feature vector Va1 but intersects them at a predetermined angle may be used, and instead of the second orthogonal vector Vb2, a vector that is not orthogonal to the second normal vector Vn2 and the second feature vector Va2 but intersects them at the same angle as the predetermined angle may be used.

[0094] 3. Notes A summary of this disclosure is provided below.

[0095] (Supplementary Note 1) A first aspect, which is a preferred example of the correction method of the present disclosure, is a correction method for correcting a projection image projected onto a plane by a projector, the correction method including: acquiring a first captured image obtained by capturing the projection image and the plane by an imaging device during a first period; acquiring a second captured image obtained by capturing the projection image and the plane by the imaging device during a second period after the first period; extracting a first feature point and a second feature point of the plane in the first captured image; and extracting a third feature point of the plane in the second captured image that corresponds to the first feature point. extracting a fourth feature point corresponding to the second feature point; calculating a first normal vector that is a normal vector of the plane in the first period; calculating a second normal vector that is a normal vector of the plane in the second period; and correcting a position of the projected image with respect to the plane in the second period to a position of the projected image with respect to the plane in the first period based on a set of the first feature point, the second feature point, and the first normal vector, and a set of the third feature point, the fourth feature point, and the second normal vector.

[0096] In the above aspect, a matrix for correcting a projected image can be calculated based on two feature points in each captured image, thereby reducing the complexity of the matrix calculation.

[0097] (Supplementary Note 2) In a second aspect which is a preferred example of the first aspect, the method includes calculating a first feature vector which is a vector directed from the first feature point to the second feature point, calculating a first orthogonal vector which is orthogonal to both the first normal vector and the first feature vector, calculating a second feature vector which is a vector directed from the third feature point to the fourth feature point, and calculating a second orthogonal vector which is orthogonal to both the second normal vector and the second feature vector, and correcting the position of the projected image with respect to the plane in the second period to the position of the projected image with respect to the plane in the first period based on the difference between the first normal vector and the second normal vector, the difference between the first feature vector and the second feature vector, and the difference between the first orthogonal vector and the second orthogonal vector. In the above aspect, the matrix for correcting the projected image can be calculated by a relatively simple calculation.

[0098] (Supplementary Note 3) In a third aspect which is a preferred example of the first or second aspect, the first normal vector, the second normal vector, the first feature vector, the second feature vector, the first orthogonal vector, and the second orthogonal vector are normalized vectors whose magnitudes are normalized to 1. In the above aspect, the aspect ratio and the like can be maintained during restoration.

[0099] (Supplementary Note 4) In a fourth aspect, which is a preferred example of any of the first to third aspects, the first feature point and the third feature point are extracted based on reflected light from a first physical marker removably installed on the plane, and the second feature point and the fourth feature point are extracted based on reflected light from a second physical marker removably installed on the plane. In the above aspect, the number of physical markers that a user must install can be reduced, thereby reducing the complexity for the user.

[0100] (Supplementary Note 5) A fifth aspect that is a preferred example of the system of the present disclosure is a system including a projector that projects a projection image onto a plane, and an imaging device that captures the plane and the projection image, wherein the projector acquires a first captured image obtained by capturing the projection image and the plane by the imaging device in a first period, and a second captured image obtained by capturing the projection image and the plane by the imaging device in a second period after the first period, extracts first feature points and second feature points of the plane in the first captured image, and extracting a third feature point corresponding to the feature point and a fourth feature point corresponding to the second feature point; calculating a first normal vector that is a normal vector of the plane in the first period; calculating a second normal vector that is a normal vector of the plane in the second period; and correcting the position of the projected image with respect to the plane in the second period to the position of the projected image with respect to the plane in the first period based on a set of the first feature point, the second feature point, and the first normal vector, and a set of the third feature point, the fourth feature point, and the second normal vector.

[0101] In the above aspect, a matrix for correcting a projected image can be calculated based on two feature points in each captured image, thereby reducing the complexity of the matrix calculation.

[0102] (Supplementary Note 6) A sixth aspect of the program of the present disclosure is a program for correcting a projection image projected onto a plane by a projector, the program comprising: acquiring a first captured image obtained by capturing the projection image and the plane by an imaging device in a first period; and acquiring a second captured image obtained by capturing the projection image and the plane by the imaging device in a second period after the first period; extracting a first feature point and a second feature point of the plane in the first captured image; and extracting a third feature point and a second feature point of the plane corresponding to the first feature point of the plane in the second captured image. and a fourth feature point corresponding to the point, calculating a first normal vector that is a normal vector of the plane in the first period, calculating a second normal vector that is a normal vector of the plane in the second period, and correcting the position of the projected image with respect to the plane in the second period to the position of the projected image with respect to the plane in the first period based on a set of the first feature point, the second feature point, and the first normal vector, and a set of the third feature point, the fourth feature point, and the second normal vector.

[0103] In the above aspect, a matrix for correcting a projected image can be calculated based on two feature points in each captured image, thereby reducing the complexity of the matrix calculation. [Explanation of symbols]

[0104] 10...projector, 11...storage device, 12...processing device, 12a...projection control unit, 12b...imaging control unit, 12c...correction unit, 13...communication device, 14...image processing circuit, 15...optical device, 15a...light source, 15b...display panel, 15c...optical system, 16...operation device, 20...imaging device, 30...terminal device, 100...system, D1...first imaging data, D2...second imaging data, DC...correction parameter information, DP1...first adjusted point cloud information, DP2...second adjusted point cloud information, DP3...third adjusted point cloud information, DP4...fourth adjusted point cloud information, DV1...first normal vector information, DV2...second normal vector information, DV3...first feature vector information, DV4...second feature vector information, DV5...first orthogonal vector information, DV6...second orthogonal vector information, G...projected image, G-1...projected image, G-2...projected image, GG1...first captured image, GG2...second captured image, IMG...video data, MK-1...first physical marker, MK-2...second physical marker, P1...first feature point, P2...second feature point, P3...third feature point, P4...fourth feature point, PR...adjustment point, PR1...program, RC...area, S1...step, S2...step, S3...step, S4...step, S5...step, S6...step, S7...step, S8...step, S9...step, S10...step, S11...step, S12...step, S13...step, S14...step, SC...projection surface (plane), Va1...first feature vector, Va2...second feature vector, Vb1...first orthogonal vector, Vb2...second orthogonal vector, Vn1...first normal vector, Vn2...second normal vector.

Claims

1. A correction method for correcting a projection image projected onto a plane by a projector, comprising: acquiring a first captured image obtained by capturing the projection image and the plane by an imaging device during a first period; acquiring a second captured image by capturing the projection image and the plane by the imaging device during a second period that is later than the first period; extracting a first feature point and a second feature point of the plane in the first captured image; extracting a third feature point corresponding to the first feature point and a fourth feature point corresponding to the second feature point of the plane in the second captured image; calculating a first normal vector that is a normal vector of the plane during the first period; calculating a second normal vector that is a normal vector of the plane during the second period; correcting a position of the projected image with respect to the plane in the second period to a position of the projected image with respect to the plane in the first period based on a set of the first feature point, the second feature point, and the first normal vector, and a set of the third feature point, the fourth feature point, and the second normal vector. Correction method.

2. calculating a first feature vector that is a vector from the first feature point to the second feature point; calculating a first orthogonal vector that is orthogonal to both the first normal vector and the first feature vector; calculating a second feature vector that is a vector extending from the third feature point to the fourth feature point; calculating a second orthogonal vector that is orthogonal to both the second normal vector and the second feature vector; correcting a position of the projected image with respect to the plane in the second period to a position of the projected image with respect to the plane in the first period based on a difference between the first normal vector and the second normal vector, a difference between the first feature vector and the second feature vector, and a difference between the first orthogonal vector and the second orthogonal vector; The correction method according to claim 1 .

3. the first normal vector, the second normal vector, the first feature vector, the second feature vector, the first orthogonal vector, and the second orthogonal vector are normalized vectors whose magnitudes are normalized to 1. The correction method according to claim 2 .

4. the first feature point and the third feature point are extracted based on reflected light from a first physical marker that is detachably installed on the plane; the second feature point and the fourth feature point are extracted based on reflected light from a second physical marker that is removably installed on the plane; The correction method according to claim 1 .

5. a projector that projects an image onto a flat surface; an imaging device that captures the plane and the projected image, The projector includes: acquiring a first captured image obtained by capturing the projection image and the plane by the imaging device during a first period; acquiring a second captured image by capturing the projection image and the plane by the imaging device during a second period that is later than the first period; extracting a first feature point and a second feature point of the plane in the first captured image; extracting a third feature point corresponding to the first feature point and a fourth feature point corresponding to the second feature point of the plane in the second captured image; calculating a first normal vector that is a normal vector of the plane during the first period; calculating a second normal vector that is a normal vector of the plane during the second period; correcting the position of the projected image with respect to the plane in the second period to the position of the projected image with respect to the plane in the first period based on a set of the first feature point, the second feature point, and the first normal vector, and a set of the third feature point, the fourth feature point, and the second normal vector. system.

6. A program for correcting a projection image projected onto a plane by a projector, acquiring a first captured image obtained by capturing the projection image and the plane by an imaging device during a first period; acquiring a second captured image by capturing the projection image and the plane by the imaging device during a second period that is later than the first period; extracting a first feature point and a second feature point of the plane in the first captured image; extracting a third feature point corresponding to the first feature point and a fourth feature point corresponding to the second feature point of the plane in the second captured image; calculating a first normal vector that is a normal vector of the plane during the first period; calculating a second normal vector that is a normal vector of the plane during the second period; correcting a position of the projected image with respect to the plane in the second time period to a position of the projected image with respect to the plane in the first time period based on a set of the first feature point, the second feature point, and the first normal vector, and a set of the third feature point, the fourth feature point, and the second normal vector; program.

Images