Measuring method, measuring system, and information processing apparatus

Abstract

To generate a three-dimensional projective transformation matrix for transforming the coordinates of a plurality of points included in a single plane.SOLUTION: A measuring method includes: on a screen including a first surface that is a plane and a second surface that is a non-plane, generating first coordinate information representing the coordinates in a first coordinate system of a plurality of first points included in a first area of the first surface; generating second coordinate information representing the coordinates in a second coordinate system of the plurality of first points; generating, in the first coordinate system, third coordinate information representing the coordinates of a second point not included in the plane including the plurality of first points; generating, in the second coordinate system, fourth coordinate information representing the coordinates of a third point not included in the plane including the plurality of first points; and based on the first coordinate information, second coordinate information, third coordinate information, and fourth coordinate information, generating a three-dimensional projective transformation matrix for transforming the coordinates of the plurality of first points in the second coordinate system into the coordinates of the plurality of first points in the first coordinate system.SELECTED DRAWING: Figure 14

Description

[Technical field]

[0001] The present invention relates to a measurement method, a measurement system, and an information processing device. [Background technology]

[0002] Matrix operations for coordinate transformation are widely used in technologies for projecting an image onto an object, capturing an image of an object, etc. Specifically, the matrix operations are used for generating panoramic images, correcting the shape and display position of a projected image displayed by a projection device, calibrating a camera, and measuring the three-dimensional shape of an object, etc.

[0003] Patent Document 1 discloses an information processing device that generates a homography transformation matrix for performing homography transformation, which is a transformation from a two-dimensional space to another two-dimensional space, and further generates a three-dimensional projective transformation matrix for performing three-dimensional projective transformation from a three-dimensional space to a three-dimensional space. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-126673 A Summary of the Invention [Problem to be solved by the invention]

[0005] A three-dimensional projective transformation matrix is ​​calculated based on the coordinates of a plurality of points. However, it is known that if the plurality of points exist on the same plane, the three-dimensional projective transformation matrix cannot be estimated correctly due to the nature of the calculation. The information processing device of Patent Document 1 judges the planarity of the plurality of points used in the calculation of the three-dimensional projective transformation matrix, and when it is judged that the plurality of points have planarity, it suspends the process related to the calculation of the three-dimensional projective transformation matrix. For this reason, the information processing device was unable to generate a three-dimensional projective transformation matrix that converts the coordinates of a plurality of points existing on the same plane. [Means for solving the problem]

[0006] One aspect of the measurement method according to the present invention includes a screen including a first surface that is a plane and a second surface that is a non-planar surface, and includes the steps of: generating first coordinate information representing coordinates in a first coordinate system of a plurality of first points included in a first region of the first surface; generating second coordinate information representing coordinates in a second coordinate system of the plurality of first points; generating third coordinate information representing coordinates in the first coordinate system of a second point that is not included in the plane including the plurality of first points; generating fourth coordinate information representing coordinates in the second coordinate system of a third point that is not included in the plane including the plurality of first points; and calculating the coordinates of the plurality of first points in the second coordinate system based on the first coordinate information, the second coordinate information, the third coordinate information, and the fourth coordinate information. generating a three-dimensional projection transformation matrix for transforming the coordinates of a plurality of first points in the screen into coordinates of the plurality of first points in the first coordinate system; generating fifth coordinate information representing coordinates in the first coordinate system of four set points included in the screen based on the three-dimensional projection transformation matrix; and if the coordinates indicated by the fifth coordinate information do not satisfy a predetermined condition, performing a first update to update at least one of the third coordinate information and the fourth coordinate information, performing a second update to update the three-dimensional projection transformation matrix based on a result of the first update, and performing a third update to update the fifth coordinate information based on the three-dimensional projection transformation matrix updated by the second update, wherein a quadrangle having the four set points as vertices has a predetermined shape, and at least one of the four set points is included in the second surface.

[0007] One aspect of the measurement system according to the present invention includes a screen including a first surface which is a plane and a second surface which is a non-planar surface, and includes the steps of: generating first coordinate information representing coordinates in a first coordinate system of a plurality of first points included in a first region of the first surface; generating second coordinate information representing coordinates in a second coordinate system of the plurality of first points; generating third coordinate information representing coordinates in the first coordinate system of a second point not included in the plane including the plurality of first points; generating fourth coordinate information representing coordinates in the second coordinate system of a third point not included in the plane including the plurality of first points; and calculating the coordinates of the plurality of first points in the second coordinate system based on the first coordinate information, the second coordinate information, the third coordinate information, and the fourth coordinate information. the processing device performs the following operations: generating a three-dimensional projective transformation matrix for transforming the coordinates of a plurality of first points included in the screen into coordinates of the plurality of first points included in the screen; generating fifth coordinate information representing coordinates in the first coordinate system of four set points included in the screen based on the three-dimensional projective transformation matrix; if the coordinates indicated by the fifth coordinate information do not satisfy a predetermined condition, performing a first update to update at least one of the third coordinate information and the fourth coordinate information, performing a second update to update the three-dimensional projective transformation matrix based on a result of the first update, and performing a third update to update the fifth coordinate information based on the three-dimensional projective transformation matrix updated by the second update; wherein a quadrangle having the four set points as vertices has a predetermined shape, and at least one of the four set points is included in the second surface.

[0008] One aspect of an information processing device according to the present invention includes a screen including a first surface which is a plane and a second surface which is a non-planar surface, and includes the steps of: generating first coordinate information representing coordinates in a first coordinate system of a plurality of first points included in a first region of the first surface; generating second coordinate information representing coordinates in a second coordinate system of the plurality of first points; generating third coordinate information representing coordinates in the first coordinate system of a second point not included in the plane including the plurality of first points; generating fourth coordinate information representing coordinates in the second coordinate system of a third point not included in the plane including the plurality of first points; and calculating the coordinates of the plurality of first points in the second coordinate system as the coordinates in the first coordinate system based on the first coordinate information, the second coordinate information, the third coordinate information, and the fourth coordinate information. the processing device performs the following operations: generating a three-dimensional projective transformation matrix for transforming the coordinates of a plurality of first points included in the screen into coordinates of the plurality of first points included in the screen; generating fifth coordinate information representing coordinates in the first coordinate system of four set points included in the screen based on the three-dimensional projective transformation matrix; if the coordinates indicated by the fifth coordinate information do not satisfy a predetermined condition, performing a first update to update at least one of the third coordinate information and the fourth coordinate information, performing a second update to update the three-dimensional projective transformation matrix based on a result of the first update, and performing a third update to update the fifth coordinate information based on the three-dimensional projective transformation matrix updated by the second update; wherein a quadrangle having the four set points as vertices has a predetermined shape, and at least one of the four set points is included in the second surface. [Brief description of the drawings]

[0009] [Figure 1] FIG. 13 is a schematic diagram illustrating a state in which a projected image GP1 is displayed. [Diagram 2] FIG. 13 is a schematic diagram illustrating a state in which a projection image GP2 is displayed. [Diagram 3] FIG. 11 is a schematic diagram illustrating a state in which a projected image GP3 is displayed. [Figure 4] FIG. 2 is a block diagram showing the configuration of a measurement system Sys according to the embodiment. [Diagram 5] 1 is a block diagram showing a configuration of a storage device 10 according to an embodiment. [Figure 6]1 is a schematic diagram illustrating an example of an image indicated by the first projection image information 103. FIG. [Figure 7] 1 is a schematic diagram illustrating an example of an image indicated by the first captured image information 106. FIG. [Figure 8] 10 is a schematic diagram illustrating an example of an image indicated by the second captured image information 107. FIG. [Figure 9] 13 is a schematic diagram illustrating an example of an image indicated by the second projection image information 104. FIG. [Figure 10] 13 is a schematic diagram illustrating another example of the image represented by the second projection image information 104. FIG. [Figure 11] FIG. 2 is an explanatory diagram showing a schematic diagram of the positional relationship between a plurality of points in the coordinate system FY. [Figure 12] 1 is an explanatory diagram showing a schematic diagram of the positional relationship between multiple points in a coordinate system FX. [Figure 13] 13 is a schematic diagram for explaining the coordinates of a plurality of points indicated by the fifth coordinate information 115. FIG. [Figure 14] 4 is a flowchart for explaining the operation of the measurement system Sys according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. Note that the dimensions and scale of each part in the drawings may differ from the actual dimensions, and some parts are shown diagrammatically to facilitate understanding. Furthermore, the scope of the present invention is not limited to these embodiments unless otherwise specified in the following description to the effect that the present invention is limited thereto.

[0011] 1. Embodiment In the embodiments, a measurement method, a measurement system, and an information processing device according to the present invention will be described by taking as an example a measurement system that measures the three-dimensional shape of an object using a two-dimensional image such as an image obtained by capturing an image of the object, in other words, acquires the three-dimensional coordinates of a plurality of points located on the surface of the object. The measurement system according to the embodiment generates a three-dimensional projective transformation matrix for converting the three-dimensional coordinates of a plurality of points located on a plane, and measures the shape of the object using the generated three-dimensional projective transformation matrix.

[0012] When acquiring the three-dimensional coordinates of a point located on the surface of an object using a two-dimensional image, "internal parameters" indicating parameters derived from a device used to acquire the three-dimensional coordinates of the point, and "external parameters" indicating parameters derived from the position and orientation of the device are required. In this specification, when the devices used to measure the three-dimensional shape of an object using a two-dimensional image include a device whose internal parameters are unknown, the measurement method is called "projective reconstruction." In addition, when the devices used to measure the three-dimensional shape of an object using a two-dimensional image do not include a device whose internal parameters are unknown, in other words, when the internal parameters of all devices used in the measurement are known, the measurement method is called "Euclidean reconstruction."

[0013] Moreover, the measurement result of the three-dimensional shape of an object by projective restoration, in other words, the three-dimensional coordinates of points located on the surface of the object obtained by projective restoration, is referred to as the "projective restoration result". The three-dimensional coordinates of points indicated by the "projective restoration result" include the uncertainty of projective transformation. For this reason, the shape of the object obtained by projective restoration may differ from the real object. Moreover, the measurement result of the three-dimensional shape of an object by Euclidean restoration, in other words, the three-dimensional coordinates of points located on the surface of the object obtained by Euclidean restoration, is referred to as the "Euclidean restoration result". The three-dimensional coordinates of points indicated by the "Euclidean restoration result" do not include the uncertainty of projective transformation, or the uncertainty of projective transformation can be ignored. In other words, the shape of the object obtained by Euclidean restoration is similar to the real object. In addition, if the three-dimensional coordinates of a point obtained by measuring the three-dimensional shape of an object do not include any uncertainty due to projective transformation, or if the uncertainty due to projective transformation can be ignored, the three-dimensional coordinates of the point are considered to be the "Euclidean restoration result."

[0014] 1.1. Overview of the measurement system Hereinafter, an overview of a measurement system Sys according to an embodiment will be described with reference to Fig. 1 to Fig. 3. Fig. 1 is a schematic diagram illustrating how a projected image GP1 is displayed. Fig. 2 is a schematic diagram illustrating how a projected image GP2 is displayed. Fig. 3 is a schematic diagram illustrating how a projected image GP3 is displayed.

[0015] The measurement system Sys includes a projector 1 and a projector 2. The projector 1 is communicably connected to the projector 2 and transmits and receives various information. The projector 1 includes an imaging device 14 and a projection device 16. The projector 2 includes an imaging device 24 and a projection device 26.

[0016] The projection device 16 and the projection device 26 project an image onto the screen SC1, thereby displaying the projected image on the screen SC1. The internal parameters of the projection device 16 and the projection device 26 are unknown.

[0017] The screen SC1 includes a surface SC11, a surface SC12, and a surface SC13. The surface SC11 is a plane. The surfaces SC12 and SC13 are curved surfaces. More precisely, the surfaces SC12 and SC13 are cylindrical surfaces. At least a part of the projected image displayed by the projection device 16 is displayed on the surface SC11.

[0018] The imaging device 14 captures an image of an area including at least a part of the projected image displayed on the screen SC1 by the projection device 16. Specifically, the imaging device 14 captures an image of an area including at least a part of the projected image displayed on the surface SC11 by the projection device 16. The imaging device 14 also captures an image of an area including at least a part of the projected image displayed on the screen SC1 by the projection device 26. The internal parameters of the imaging device 14 are known.

[0019] The imaging device 24 captures an area including the area R2 of the screen SC1. The area R2 includes a part of the surface SC11 and a part of the surface SC13. The imaging device 24 also captures an area including at least a part of the projected image displayed on the screen SC1 by the projection device 16. Specifically, the imaging device 24 captures an area including at least a part of the projected image displayed on the screen SC1 by the projection device 16. The imaging device 24 also captures an area including at least a part of the projected image displayed on the screen SC1 by the projection device 26. The imaging device 24 has known internal parameters. At least a part of the imaging range of the imaging device 14 and at least a part of the imaging range of the imaging device 24 overlap on the surface SC11. In other words, the overlapping portion between the imaging range of the imaging device 14 and the imaging range of the imaging device 24 is included in the surface SC11.

[0020] The projection device 16 projects an image GF1, which will be described later, onto the screen SC1, thereby displaying a projection image GP1 on the screen SC1. The imaging device 14 captures an image of a range including an area R1 of the screen SC1 where the projection image GP1 is displayed. The area R1 includes a part of the surface SC11 and a part of the surface SC12. A part of the area R1 and a part of the area R2 overlap in an area R3. In other words, the area R1 and the area R2 include the area R3. That is, a part of the projection image GP1 is displayed in the area R3. The area R3 is included in the surface SC11. The imaging device 24 captures an image of a range including an area R2 of the screen SC1 where a part of the projection image GP1 is displayed.

[0021] The projection image GP1 is displayed across the surfaces SC11 and SC12. The projection image GP1 includes a plurality of points. The plurality of points includes a plurality of points DP1. The plurality of points DP1 are included in an area R3 of the surface SC11.

[0022] The projection device 16 displays a projection image GP2 on the screen SC1 by projecting an image GF2, which will be described later, onto the screen SC1. The projection image GP2 is displayed in the area R1 of the screen SC1, similar to the projection image GP1. The projection image GP2 includes points DP21, DP22, DP23, and DP24. Note that the points DP23 and DP24 are displayed on the surface SC12. In other words, the points DP23 and DP24 are included in the surface SC12.

[0023] A user of the measurement system Sys adjusts the shape of a quadrangle having the vertices of the points DP21 to DP24 into a rectangle by performing an operation to adjust the positions of the points DP21 to DP24 included in the projected image GP2. Hereinafter, the operation to adjust the positions of the points DP21 to DP24 included in the projected image GP2 is referred to as a "position adjustment operation." The projector 1 accepts a position adjustment operation from the user to adjust the shape of a quadrangle having the vertices of the points DP21 to DP24 into a rectangle. Note that while the position adjustment operation is being performed, the user may use, for example, a so-called laser marking tool to display a guideline to assist in the position adjustment of the points DP21 to DP24. This allows the user to easily and accurately adjust the positions of the points DP21 to DP24.

[0024] When the user adjusts the shape of a quadrangle having vertices DP21 to DP24 into a rectangle by performing a position adjustment operation, the projection device 16 displays a projection image GP3 on the screen SC1 by projecting an image GF3 (described later) onto the screen SC1. The projection image GP3 is displayed in the area R1 of the screen SC1, similar to the projection images GP1 and GP2. The projector 1 updates the projection image GP2 to the projection image GP3 based on the position adjustment operation received from the user.

[0025] The projected image GP3 includes points DP31, DP32, DP33, and DP34. The projector 1 moves the point DP21 to the point DP31 based on the position adjustment operation. In other words, the point DP31 is a point that represents the result of adjusting the position of the point DP21. Similarly, the projector 1 moves the point DP22 to the point DP32, the point DP23 to the point DP33, and the point DP24 to the point DP34 based on the position adjustment operation. That is, a quadrangle having the points DP31 to DP34 as vertices becomes a rectangle. The points DP31 and DP32 are displayed on the surface SC11. The points DP33 and DP34 are displayed on the surface SC12. In other words, the points DP31 and DP32 are included in the surface SC11. The points DP33 and DP34 are included in the surface SC12.

[0026] The projector 1 acquires a Euclidean reconstruction result of the region R3 based on a captured image acquired by the imaging device 14, whose internal parameters are known, and a captured image acquired by the imaging device 24, whose internal parameters are known. The projector 1 also acquires a projective reconstruction result of the region R1 based on the captured image acquired by the imaging device 14 and an image GF1 projected from the projection device 16, whose internal parameters are unknown. The projective reconstruction result of the region R1 includes the projective reconstruction result of the region R3. The projective reconstruction result of the region R1 indicates the coordinates of points DP31 to DP34.

[0027] The projector 1 generates a three-dimensional projective transformation matrix that converts the projective reconstruction result of the region R3 into the Euclidean reconstruction result of the region R3 based on the Euclidean reconstruction result of the region R3 and the projective reconstruction result of the region R1. As described above, the three-dimensional projective transformation matrix is ​​calculated based on the coordinates of a plurality of points, but is not calculated correctly when the plurality of points exist on the same plane. Since the region R3 is included in the plane SC11, the projector 1 sets a virtual point that is not included in the plane SC11 and its extension, and generates a three-dimensional projective transformation matrix that converts the projective reconstruction result of the region R3 into the Euclidean reconstruction result of the region R3 based on the coordinates of the virtual point, the Euclidean reconstruction result of the region R3, and the projective reconstruction result of the region R1. That is, the projector 1 can generate the three-dimensional projective transformation matrix by setting the coordinates of a plurality of points used in generating the three-dimensional projective transformation matrix that converts the projective reconstruction result of the region R3 into the Euclidean reconstruction result of the region R3 to the coordinates of a plurality of points that do not exist on the same plane.

[0028] Furthermore, the projector 1 transforms the projective reconstruction result of the region R1 using the three-dimensional projective transformation matrix. That is, the projector 1 transforms the coordinates of the points DP31 to DP34 obtained by the projective reconstruction using the three-dimensional projective transformation matrix.

[0029] The projector 1 updates the three-dimensional projective transformation matrix until the coordinates of the points DP31 to DP34 transformed using the three-dimensional projective transformation matrix satisfy a predetermined condition, and transforms the coordinates of the points DP31 to DP34 obtained by projective reconstruction using the updated three-dimensional projective transformation matrix. When the coordinates of the points DP31 to DP34 transformed using the three-dimensional projective transformation matrix satisfy a predetermined condition, the three-dimensional projective transformation matrix can transform the projective reconstruction result of the region R1 into the Euclidean reconstruction result of the region R1.

[0030] That is, the measurement system Sys can acquire the Euclidean restoration result of the region R1 on the screen SC1 by using the imaging device 14 whose internal parameters are known, the imaging device 24 whose internal parameters are known, and the projection device 16 whose internal parameters are unknown, without performing a calibration operation for grasping the internal parameters of the projection device 16. In other words, by using the imaging device 14 whose internal parameters are known, the imaging device 24 whose internal parameters are known, and the projection device 16 whose internal parameters are unknown, the measurement system Sys can acquire the Euclidean restoration result of the region R1 excluding the region R3 that is outside the imaging range of the imaging device 24, in addition to the Euclidean restoration result of the region R3 included in both the imaging range of the imaging device 14 and the imaging range of the imaging device 24.

[0031] 1.2. Measurement system configuration and functions The configuration and functions of the measurement system Sys according to the embodiment will be described below with reference to FIGS.

[0032] 4 is a block diagram showing the configuration of the measurement system Sys according to the embodiment. The measurement system Sys includes the projector 1 and the projector 2, as described above.

[0033] The projector 1 includes a storage device 10 that stores various information, a processing device 12 that controls the operation of the measurement system Sys, an imaging device 14 that acquires a captured image used to measure the three-dimensional shape of an object, a projection device 16 that projects an image onto a projection surface, a communication device 18 that transmits and receives various information to and from the projector 2, and an operation device 30 that accepts input operations from a user. The processing device 12 has functions as a projection control unit 120, an imaging control unit 121, an image analysis unit 122, a three-dimensional coordinate management unit 123, an input management unit 124, and an image editing unit 125. The imaging device 14 includes an imaging lens 142 for collecting light, and an imaging element 140 that generates a captured image by converting the light collected by the imaging lens 142 into an electrical signal. The projection device 16 includes a light source (not shown), an optical modulator 160 that modulates the light emitted from the light source into projection light for displaying a projection image on a projection surface, and a projection lens 162 that projects the projection light modulated by the optical modulator 160 onto the projection surface.

[0034] The projector 2 includes an imaging device 24 that acquires an image used to measure the three-dimensional shape of an object, and a projection device 26 that projects the image onto a projection surface. The imaging device 24 includes an imaging lens 242 for collecting light, and an imaging element 240 that generates a captured image by converting the light collected by the imaging lens 242 into an electrical signal. The projection device 26 includes a light source (not shown), an optical modulator 260 that modulates the light emitted from the light source into projection light for displaying a projection image on the projection surface, and a projection lens 262 that projects the projection light modulated by the optical modulator 260 onto the projection surface.

[0035] The storage device 10 includes, for example, a volatile memory such as a RAM and a non-volatile memory such as a ROM. Here, RAM is an abbreviation for Random Access Memory. Also, ROM is an abbreviation for Read Only Memory.

[0036] 5 is a block diagram showing the configuration of the storage device 10 according to the embodiment. The non-volatile memory of the storage device 10 stores a program 100 that specifies the operation of the projector 1, a three-dimensional projective transformation matrix 101 that converts the projective reconstruction result of the region R3 into the Euclidean reconstruction result of the region R3, projection image information 102 that represents an image projected on the projection surface, captured image information 105 that represents the result of capturing an image of an object to be measured for its three-dimensional shape, three-dimensional coordinate information 110 that represents the result of measuring the three-dimensional shape of the object using the captured image indicated by the captured image information 105, two-dimensional coordinate information 130 that represents the coordinates of points included in various images, and relationship information 116 that represents the correspondence between a plurality of pixels included in the optical modulator 160 and a plurality of pixels included in the image sensor 140.

[0037] The projection image information 102 includes first projection image information 103 representing an image projected when the projection image GP1 is displayed, and second projection image information 104 representing an image projected when the projection image GP2 or the projection image GP3 is displayed.

[0038] The captured image information 105 includes first captured image information 106 representing a captured image acquired by the imaging device 14 and second captured image information 107 representing a captured image acquired by the imaging device 24 .

[0039] The three-dimensional coordinate information 110 includes first coordinate information 111 representing the Euclidean reconstruction result of the region R3, second coordinate information 112 representing the projective reconstruction result of the region R1, third coordinate information 113 and fourth coordinate information 114 representing the coordinates of virtual points not included in the surface SC11 or its extension, and fifth coordinate information 115 representing the result of transforming the coordinates of points DP31 to DP34 obtained by the projective reconstruction using the three-dimensional projective transformation matrix 101. For ease of explanation, the coordinate system used to represent the coordinates of multiple points indicated by the three-dimensional coordinate information 110 has the origin at a point indicating the position of the imaging device 14.

[0040] The two-dimensional coordinate information 130 includes sixth coordinate information 131 representing the coordinates of multiple points included in the image represented by the first captured image information 106, seventh coordinate information 132 representing the coordinates of multiple points included in the image represented by the second captured image information 107, eighth coordinate information 133 representing the coordinates of multiple points included in the image represented by the first projected image information 103, and ninth coordinate information 134 representing the coordinates of multiple points included in the image represented by the second projected image information 104.

[0041] The volatile memory of the storage device 10 is used by the processing device 12 as a work area when the program 100 is executed.

[0042] A part or all of the storage device 10 may be provided in an external storage device or an external server, etc. A part or all of the various information stored in the storage device 10 may be stored in advance in the storage device 10, or may be obtained from an external storage device or an external server, etc.

[0043] 6 is a schematic diagram illustrating an example of an image indicated by the first projection image information 103. In this embodiment, the first projection image information 103 represents an image GF1. The projector 1 causes the projection device 16 to project projection light based on the first projection image information 103, thereby displaying the projection image GP1 on the screen SC1. In other words, the projector 1 causes the projection device 16 to project the image GF1, thereby displaying the projection image GP1 on the screen SC1.

[0044] The image GF1 includes a plurality of points. The plurality of points includes a plurality of points DF1. The plurality of points DF1 correspond one-to-one to a plurality of points DP1 included in the projected image GP1. In FIG. 6, the plurality of points DF1 are points included within an area surrounded by a dashed line.

[0045] 7 is a schematic diagram illustrating an example of an image indicated by the first captured image information 106. In this embodiment, the first captured image information 106 represents a captured image GS1. The imaging device 14 acquires the captured image GS1 by capturing an image of an area including the area R1 of the screen SC1 on which the projection image GP1 is displayed. In other words, the imaging device 14 captures an image of an area including the area R1 of the screen SC1 on which the projection image GP1 is displayed, thereby generating the first captured image information 106 representing the captured image GS1.

[0046] The captured image GS1 includes an image GS11. The image GS11 is an image showing a portion of the screen SC1. The image GS11 is made up of an image GS111 and an image GS112. The image GS111 is an image showing at least a portion of the surface SC11. The image GS112 is an image showing at least a portion of the surface SC12. The image GS11 includes an image GV1. The image GV1 is an image showing a projected image GP1. The image GV1 is made up of an image GV11 and an image GV12. The image GV11 is an image showing a portion of the projected image GP1 displayed on the surface SC11. The image GV11 is included in the image GS111. The image GV12 is an image showing a portion of the projected image GP1 displayed on the surface SC12. The image GV12 is included in the image GS112.

[0047] The image GV1 includes a plurality of points. The plurality of points includes a plurality of points DV1. The plurality of points DV1 correspond one-to-one to a plurality of points DP1 included in the projected image GP1. The plurality of points DV1 also correspond one-to-one to a plurality of points DF1 included in the image GF1. In FIG. 7, the plurality of points DV1 are points included within an area surrounded by a dashed line.

[0048] 8 is a schematic diagram illustrating an example of an image indicated by the second captured image information 107. In this embodiment, the second captured image information 107 represents a captured image GS2. The imaging device 24 acquires the captured image GS2 by capturing an image of a range including the region R2 of the screen SC1 on which a part of the projection image GP1 is displayed. In other words, the imaging device 24 captures an image of a range including the region R2 of the screen SC1 on which a part of the projection image GP1 is displayed, thereby generating the second captured image information 107 representing the captured image GS2.

[0049] The captured image GS2 includes an image GS21. The image GS21 is an image showing a portion of the screen SC1. The image GS21 is made up of an image GS211 and an image GS213. The image GS211 is an image showing at least a portion of the surface SC11. The image GS213 is an image showing at least a portion of the surface SC13. The image GS211 includes an image GV21. The image GV21 is an image showing a portion of the projected image GP1 displayed in the region R3.

[0050] The image GV21 includes a plurality of points DV2. The plurality of points DV2 correspond one-to-one to a plurality of points DP1 included in the projected image GP1. The plurality of points DV2 also correspond one-to-one to a plurality of points DF1 included in the image GF1. The plurality of points DV2 also correspond one-to-one to a plurality of points DV1 included in the captured image GS1. In FIG. 8, the plurality of points DV2 are points included within an area surrounded by a dashed line.

[0051] 9 is a schematic diagram illustrating an example of an image indicated by the second projection image information 104. For example, the second projection image information 104 represents an image GF2. The projector 1 causes the projection device 16 to project projection light based on the second projection image information 104, thereby displaying the projection image GP2 on the screen SC1. In other words, the projector 1 causes the projection device 16 to project the image GF2, thereby displaying the projection image GP2 on the screen SC1.

[0052] Image GF2 includes points DF21, DF22, DF23, and DF24. Point DF21 corresponds to point DP21. Point DF22 corresponds to point DP22. Point DF23 corresponds to point DP23. Point DF24 corresponds to point DP24.

[0053] FIG. 10 is a schematic diagram illustrating another example of an image indicated by the second projection image information 104. Specifically, FIG. 10 is a schematic diagram illustrating an image indicated by the second projection image information 104 when the projector 1 receives a position adjustment operation from the user in a state in which the projection image GP2 is displayed on the screen SC1. In other words, FIG. 10 is a schematic diagram illustrating an image indicated by the second projection image information 104 updated based on the position adjustment operation. The second projection image information 104 updated based on the position adjustment operation represents an image GF3. The projector 1 displays the projection image GP3 on the screen SC1 by having the projection device 16 project the projection light based on the second projection image information 104 updated based on the position adjustment operation. In other words, the projector 1 displays the projection image GP3 on the screen SC1 by having the projection device 16 project the image GF3.

[0054] Image GF3 includes points DF31, DF32, DF33, and DF34. Point DF31 is a point that represents the result of adjusting the position of point DF21 based on the position adjustment operation. Point DF31 corresponds to point DP31. Point DF32 is a point that represents the result of adjusting the position of point DF22 based on the position adjustment operation. Point DF32 corresponds to point DP32. Point DF33 is a point that represents the result of adjusting the position of point DF23 based on the position adjustment operation. Point DF33 corresponds to point DP33. Point DF34 is a point that represents the result of adjusting the position of point DF24 based on the position adjustment operation. Point DF34 corresponds to point DP34.

[0055] 4, the processing device 12 includes one or more CPUs. However, the processing device 12 may include a programmable logic device such as an FPGA instead of or in addition to a CPU. Here, CPU is an abbreviation for Central Processing Unit, and FPGA is an abbreviation for Field-Programmable Gate Array.

[0056] The processing device 12 functions as a projection control unit 120, an imaging control unit 121, an image analysis unit 122, a three-dimensional coordinate management unit 123, an input management unit 124, and an image editing unit 125 shown in FIG. 4 by the CPU or the like possessed by the processing device 12 executing the program 100.

[0057] The projection control unit 120 controls the projection device to project projection light for displaying a projection image onto a projection surface of an object to be measured for three-dimensional shape. Specifically, the projection control unit 120 causes the projection device to project projection light based on projection image information, thereby displaying the projection image on the projection surface.

[0058] In this embodiment, the projection control unit 120 controls the projection device 16 to project projection light for displaying a projection image onto the screen SC1. Specifically, the projection control unit 120 causes the projection device 16 to project projection light based on the first projection image information 103, thereby displaying the projection image GP1 on the screen SC1. In other words, the projection control unit 120 causes the projection device 16 to project the image GF1 indicated by the first projection image information 103, thereby displaying the projection image GP1 on the screen SC1.

[0059] Moreover, the projection control unit 120 causes the projection device 16 to project projection light based on the second projection image information 104, thereby displaying the projection image GP2 on the screen SC1. In other words, the projection control unit 120 causes the projection device 16 to project the image GF2 indicated by the second projection image information 104, thereby displaying the projection image GP2 on the screen SC1.

[0060] Furthermore, the projection control unit 120 causes the projection device 16 to project projection light based on the second projection image information 104 updated based on the position adjustment operation, thereby displaying the projection image GP3 on the screen SC1. In other words, the projection control unit 120 causes the projection device 16 to project the image GF3 indicated by the second projection image information 104 updated based on the position adjustment operation, thereby displaying the projection image GP3 on the screen SC1.

[0061] The imaging control unit 121 controls the imaging device to capture an image of an object to be measured for its three-dimensional shape. The imaging control unit 121 also acquires captured images from the imaging device to be used for measuring the three-dimensional shape of the object. The imaging control unit 121 also stores captured image information 105 representing the acquired captured images in the storage device 10.

[0062] In this embodiment, the imaging control unit 121 controls the imaging device 14 to capture an area including the area R1 of the screen SC1 on which the projection image GP1 is displayed. That is, the imaging control unit 121 controls the imaging device 14 to capture an area including the area R3 of the screen SC1 on which a part of the projection image GP1 is displayed. In other words, the imaging control unit 121 controls the imaging device 14 to capture a plurality of points DP1 included in the projection image GP1. The imaging control unit 121 also acquires a captured image GS1 representing the result of the imaging. The imaging control unit 121 also stores the first captured image information 106 representing the captured image GS1 in the storage device 10.

[0063] The imaging control unit 121 also controls the imaging device 24 to capture an area including the area R2 of the screen SC1 where a part of the projection image GP1 is displayed. That is, the imaging control unit 121 controls the imaging device 24 to capture an area including the area R3 of the screen SC1 where a part of the projection image GP1 is displayed. In other words, the imaging control unit 121 controls the imaging device 24 to capture an image of a plurality of points DP1 included in the projection image GP1. The imaging control unit 121 also obtains a captured image GS2 representing the result of the imaging. The imaging control unit 121 also stores second captured image information 107 representing the captured image GS2 in the storage device 10.

[0064] The image analysis unit 122 detects points included in the image by executing image processing on the image indicated by various image information. That is, the image analysis unit 122 acquires two-dimensional coordinate information 130 representing the coordinates of the detected points. The image analysis unit 122 also stores the acquired two-dimensional coordinate information 130 in the storage device 10.

[0065] In this embodiment, the image analysis unit 122 detects multiple points included in the captured image GS1 by executing image processing on the captured image GS1 indicated by the first captured image information 106. That is, the image analysis unit 122 acquires sixth coordinate information 131 indicating the coordinates of multiple points included in the captured image GS1. Furthermore, the image analysis unit 122 stores the acquired sixth coordinate information 131 in the storage device 10. The sixth coordinate information 131 indicates the coordinates of multiple points DV1.

[0066] Furthermore, the image analysis unit 122 detects a plurality of points included in the captured image GS2 by executing image processing on the captured image GS2 indicated by the second captured image information 107. That is, the image analysis unit 122 acquires seventh coordinate information 132 indicating the coordinates of a plurality of points included in the captured image GS2. Furthermore, the image analysis unit 122 stores the acquired seventh coordinate information 132 in the storage device 10. The seventh coordinate information 132 indicates the coordinates of a plurality of points DV2.

[0067] Furthermore, the image analysis unit 122 detects multiple points included in the image GF1 by executing image processing on the image GF1 indicated by the first projection image information 103. That is, the image analysis unit 122 acquires eighth coordinate information 133 indicating the coordinates of multiple points included in the image GF1. Furthermore, the image analysis unit 122 stores the acquired eighth coordinate information 133 in the storage device 10. The eighth coordinate information 133 indicates the coordinates of multiple points DF1.

[0068] In the function related to the point detection, a known image processing technique may be used. Examples of known image processing techniques related to the point detection include template matching, center of gravity detection, and an algorithm called "AKAZE". In this specification, a detailed technical description of the point detection is omitted.

[0069] Furthermore, the image analysis unit 122 generates relationship information indicating a correspondence relationship between a plurality of pixels of an optical modulator of a projection device that projects an image indicated by the projection image information 102 and a plurality of pixels of an imaging element of an imaging device that generates the captured image information 105, based on the projection image information 102 and the captured image information 105. In other words, the image analysis unit 122 generates the relationship information based on an image indicated by the projection image information 102 and an image indicated by the captured image information 105. Furthermore, the image analysis unit 122 stores the generated relationship information in the storage device 10.

[0070] In this embodiment, the image analysis unit 122 generates, based on the first projection image information 103 and the first captured image information 106, relationship information 116 indicating a correspondence relationship between a plurality of pixels included in the optical modulator 160 of the projection device 16 that projects the image GF1 indicated by the first projection image information 103 and a plurality of pixels included in the image sensor 140 of the image capture device 14 that generates the first captured image information 106. In other words, the image analysis unit 122 generates the relationship information 116 based on the image GF1 indicated by the first projection image information 103 and the captured image GS1 indicated by the first captured image information 106. Specifically, the image analysis unit 122 generates the relationship information 116 based on eighth coordinate information 133 indicating the coordinates of a plurality of points included in the image GF1 and sixth coordinate information 131 indicating the coordinates of a plurality of points included in the captured image GS1. The image analysis unit 122 also stores the generated relationship information 116 in the storage device 10.

[0071] The input management unit 124 acquires operation data indicating the content of an operation accepted from a user by controlling the operation device 30. Furthermore, the input management unit 124 executes various determinations based on the acquired operation data.

[0072] In this embodiment, the input management unit 124 acquires operation data indicating the content of a position adjustment operation received from a user.

[0073] Furthermore, the input management unit 124 determines whether or not operation data indicating the content of an operation for ending the position adjustment of the points DP21 to DP24 has been acquired from the user.

[0074] The image editing unit 125 updates the projection image information 102 based on the position adjustment operation received from the user. In other words, the image editing unit 125 updates the image indicated by the projection image information 102 based on the position adjustment operation received from the user. Also, the image editing unit 125 updates the two-dimensional coordinate information 130 representing the coordinates of points included in the image indicated by the projection image information 102 based on the position adjustment operation received from the user.

[0075] In this embodiment, the image editing unit 125 updates the second projection image information 104 based on the position adjustment operation received from the user. In other words, the image editing unit 125 updates the image indicated by the second projection image information 104 based on the position adjustment operation received from the user. Specifically, the image editing unit 125 updates the image GF2 to the image GF3 based on the position adjustment operation received from the user. That is, the second projection image information 104 updated based on the position adjustment operation represents the image GF3. In addition, the image editing unit 125 updates the ninth coordinate information 134 representing the coordinates of a plurality of points included in the image indicated by the second projection image information 104 based on the position adjustment operation received from the user. The ninth coordinate information 134 updated based on the position adjustment operation represents the coordinates of the points DF31 to DF34 included in the image GF3.

[0076] The three-dimensional coordinate management unit 123 generates three-dimensional coordinate information 110 representing the results of measuring the three-dimensional shape of the object based on the images represented by the various image information. The three-dimensional coordinate management unit 123 also stores the generated three-dimensional coordinate information 110 in the storage device 10.

[0077] In this embodiment, the three-dimensional coordinate management unit 123 generates first coordinate information 111 representing the Euclidean reconstruction result of the region R3 based on a captured image GS1 acquired by the imaging device 14 whose internal parameters are known, and a captured image GS2 acquired by the imaging device 24 whose internal parameters are known. Specifically, the three-dimensional coordinate management unit 123 generates the first coordinate information 111 based on sixth coordinate information 131 representing the coordinates of a plurality of points included in the captured image GS1 and seventh coordinate information 132 representing the coordinates of a plurality of points included in the captured image GS2. In addition, the three-dimensional coordinate management unit 123 stores the generated first coordinate information 111 in the storage device 10.

[0078] Furthermore, the three-dimensional coordinate management unit 123 generates second coordinate information 112 representing the projective reconstruction result of the region R1 based on the captured image GS1 acquired by the imaging device 14, whose internal parameters are known, and the image GF1 projected from the projection device 16, whose internal parameters are unknown. Specifically, the three-dimensional coordinate management unit 123 generates the second coordinate information 112 based on sixth coordinate information 131 representing the coordinates of multiple points included in the captured image GS1 and eighth coordinate information 133 representing the coordinates of multiple points included in the image GF1. Furthermore, the three-dimensional coordinate management unit 123 stores the generated second coordinate information 112 in the storage device 10.

[0079] The coordinates of the multiple points indicated by the second coordinate information 112 include the projective reconstruction results of the points DP31 to DP34. The projective reconstruction results of the points DP31 to DP34 are generated based on the relationship information 116 and the second projection image information 104 updated based on the position adjustment operation. Specifically, the projective reconstruction results of the points DP31 to DP34 are generated based on the relationship information 116 and the ninth coordinate information 134 representing the coordinates of the points DF31 to DF34 included in the image GF3 indicated by the second projection image information 104. That is, in this embodiment, the second coordinate information 112 is generated based on the sixth coordinate information 131, the eighth coordinate information 133, the ninth coordinate information 134, and the relationship information 116. The three-dimensional coordinate management unit 123 generates the second coordinate information 112 based on the sixth coordinate information 131, the eighth coordinate information 133, the ninth coordinate information 134, and the relationship information 116.

[0080] As a method for generating the Euclidean reconstruction result and the projective reconstruction result, for example, a known triangulation method utilizing epipolar geometry may be used. When performing triangulation, known external parameters may be estimated by decomposing an essential matrix. As a method for generating the Euclidean reconstruction result and the projective reconstruction result, known stereo matching may be used. When performing stereo matching, known stereo rectification processing may be performed in advance.

[0081] Furthermore, the three-dimensional coordinate management unit 123 generates information representing the coordinates of a virtual point not included in the plane, based on the coordinates of a plurality of points included in the plane among the coordinates of a plurality of points indicated by the three-dimensional coordinate information 110, and the coordinates of a point indicating the position of the imaging device 14. Specifically, the three-dimensional coordinate management unit 123 generates information representing the coordinates of a virtual point not included in the plane, which is located on a line connecting a point included in the plane and a point indicating the position of the imaging device 14, based on the coordinates of a plurality of points included in the plane among the coordinates of a plurality of points indicated by the three-dimensional coordinate information 110, and the coordinates of a point indicating the position of the imaging device 14.

[0082] In this embodiment, the three-dimensional coordinate management unit 123 generates third coordinate information 113 representing the coordinates of a virtual point not included in the surface SC11 or its extension, based on first coordinate information 111 representing the coordinates of a plurality of points included in the area R3 of the surface SC11 acquired by Euclidean restoration. The three-dimensional coordinate management unit 123 also stores the generated third coordinate information 113 in the storage device 10.

[0083] Furthermore, the three-dimensional coordinate management unit 123 generates fourth coordinate information 114 representing the coordinates of a virtual point not included in the surface SC11 of the screen SC1 or its extension, based on the second coordinate information 112 representing the coordinates of a plurality of points included in the region R1 of the screen SC1 acquired by the projective reconstruction. Furthermore, the three-dimensional coordinate management unit 123 stores the generated fourth coordinate information 114 in the storage device 10.

[0084] 11 is an explanatory diagram showing a schematic diagram of the positional relationship of a plurality of points in the coordinate system FY. The coordinate system FY is a coordinate system used when expressing the coordinates of a plurality of points indicated by the first coordinate information 111 and the coordinates of a point indicated by the third coordinate information 113.

[0085] Each of the multiple points DY1 is located at coordinates indicated by the first coordinate information 111, in other words, coordinates indicated by the Euclidean reconstruction result of the region R3. That is, the multiple points DY1 correspond one-to-one to the multiple points DP1 included in the region R3. Furthermore, the first coordinate information 111 represents the coordinates of the multiple points DP1 in the coordinate system FY. Since the region R3 is included in the surface SC11, which is a plane, in the coordinate system FY, the multiple points DY1 are included in the plane SV1. Furthermore, the multiple points DY1 include the point DY11. That is, the point DY11 is included in the plane SV1.

[0086] Point DY21 is a point indicating the position of imaging device 14 in coordinate system FY. Line L1 is a line connecting point DY11 and point DY21. Point DY31 is located on line L1. Point DY31 is a point located at coordinates indicated by third coordinate information 113. In other words, third coordinate information 113 indicates the coordinates of point DY31. Point DY31 is not included in plane SV1.

[0087] 12 is an explanatory diagram showing a schematic diagram of the positional relationship of a plurality of points in the coordinate system FX. The coordinate system FX is a coordinate system used when expressing the coordinates of a plurality of points indicated by the second coordinate information 112 and the coordinates of a point indicated by the fourth coordinate information 114.

[0088] Each of the multiple points DX1 is located at a coordinate indicated by the projective reconstruction result of the region R3, among the coordinates indicated by the second coordinate information 112. That is, the multiple points DX1 correspond one-to-one to the multiple points DP1 included in the region R3. Also, the second coordinate information 112 represents the coordinates of the multiple points DP1 in the coordinate system FX. In the coordinate system FX, the multiple points DX1 are included in the plane SV2. Also, the multiple points DX1 include the point DX11. That is, the point DX11 is included in the plane SV2. Also, the point DX11 corresponds to the point DY11.

[0089] Point DX21 is a point indicating the position of imaging device 14 in coordinate system FX. Line L2 is a line connecting point DX11 and point DX21. Point DX31 is located on line L2. Point DX31 is a point located at coordinates indicated by fourth coordinate information 114. In other words, fourth coordinate information 114 indicates the coordinates of point DX31. Point DX31 is not included in plane SV2.

[0090] Returning to FIG. 4, the three-dimensional coordinate management unit 123 generates a three-dimensional projective transformation matrix 101 for transforming the coordinates of the multiple points DX1 into the coordinates of the multiple points DY1 based on the first coordinate information 111, the second coordinate information 112, the third coordinate information 113, and the fourth coordinate information 114. The three-dimensional projective transformation matrix 101 transforms the coordinates of the point DX31 indicated by the fourth coordinate information 114 into the coordinates of the point DY31 indicated by the third coordinate information 113. That is, the three-dimensional projective transformation matrix 101 transforms the coordinates of the point in the coordinate system FX into the coordinates of the point in the coordinate system FY. The three-dimensional projective transformation matrix 101 also transforms the projective reconstruction result of the region R3 into the Euclidean reconstruction result of the region R3. The three-dimensional coordinate management unit 123 also stores the generated three-dimensional projective transformation matrix 101 in the storage device 10.

[0091] The three-dimensional projection transformation matrix 101 is a matrix with 4 rows and 4 columns. For example, the coordinates of a point S in the coordinate system FX are expressed as (x S ,y S ,z S ), the coordinates of the points in the coordinate system FY, and the coordinates of the points E corresponding to the points S are (x E ,y E ,z E ), when the three-dimensional projective transformation matrix 101 is a matrix H, the three-dimensional projective transformation that transforms the coordinates of a point S into the coordinates of a point E is expressed by the following equation (1) using homogeneous coordinates.

number

[0092] Furthermore, the three-dimensional coordinate management unit 123 acquires the coordinates of the points DP31 to DP34 in the coordinate system FY by converting the coordinates of the points DP31 to DP34 obtained by the projective reconstruction, that is, the coordinates of the points DP31 to DP34 in the coordinate system FX, using the three-dimensional projective transformation matrix 101. Specifically, the three-dimensional coordinate management unit 123 acquires the coordinates of the points DP31 to DP34 in the coordinate system FY by converting the coordinates of a plurality of points indicated by the second coordinate information 112 including the projective reconstruction results of the points DP31 to DP34, using the three-dimensional projective transformation matrix 101. In other words, the three-dimensional coordinate management unit 123 generates fifth coordinate information 115 representing the coordinates of the points DP31 to DP34 in the coordinate system FY based on the second coordinate information 112 and the three-dimensional projective transformation matrix 101. Furthermore, the three-dimensional coordinate management unit 123 stores the generated fifth coordinate information 115 in the storage device 10.

[0093] 13 is a schematic diagram for explaining the coordinates of a plurality of points indicated by the fifth coordinate information 115. The fifth coordinate information 115 indicates the coordinates of points DY41, DY42, DY43, and DY44.

[0094] Point DY41 is a point located at the coordinates of point DP31 in the coordinate system FY. That is, point DY41 corresponds to point DP31. Point DY41 also corresponds to point DF31.

[0095] Point DY42 is a point located at the coordinates of point DP32 in the coordinate system FY. That is, point DY42 corresponds to point DP32. Point DY42 also corresponds to point DF32.

[0096] Point DY43 is a point located at the coordinates of point DP31 in the coordinate system FY. That is, point DY43 corresponds to point DP33. Point DY43 also corresponds to point DF33.

[0097] Point DY44 is a point located at the coordinates of point DP31 in the coordinate system FY. That is, point DY44 corresponds to point DP34. Point DY44 also corresponds to point DF34.

[0098] Quadrilateral SV3 is a quadrilateral having vertices DY41 to DY44. That is, quadrilateral SV3 corresponds to a quadrilateral having vertices DP31 to DP34. Quadrilateral SV3 has corners T11, T12, T13, and T14.

[0099] Angle T11 is the angle formed by the line segment connecting points DY41 and DY42 and the line segment connecting points DY41 and DY44.

[0100] Angle T12 is the angle formed by the line segment connecting points DY41 and DY42 and the line segment connecting points DY42 and DY43.

[0101] Angle T13 is the angle formed by the line segment connecting points DY42 and DY43 and the line segment connecting points DY43 and DY44.

[0102] Angle T14 is the angle formed by the line segment connecting points DY43 and DY44 and the line segment connecting points DY41 and DY44.

[0103] As described above, the three-dimensional projective transformation matrix 101 can convert the projective reconstruction result of the region R3 into the Euclidean reconstruction result of the region R3. On the other hand, it is not certain whether the three-dimensional projective transformation matrix 101 can convert the projective reconstruction result of the region R1 into the Euclidean reconstruction result of the region R1. For this reason, the processing device 12 judges whether the three-dimensional projective transformation matrix 101 can convert the projective reconstruction result of the region R1 into the Euclidean reconstruction result of the region R1. Hereinafter, the judgment of whether the three-dimensional projective transformation matrix 101 can convert the projective reconstruction result of the region R1 into the Euclidean reconstruction result of the region R1 may be referred to as "evaluation of the three-dimensional projective transformation matrix 101". The evaluation of the three-dimensional projective transformation matrix 101 is performed using the value Q of the following formula (2) when the angle of the corner T11 is θ1, the angle of the corner T12 is θ2, the angle of the corner T13 is θ3, and the angle of the corner T14 is θ4 among the angles of the interior angles of the quadrangle SV3.

number

[0104] In reality, the value Q may not be 0 due to measurement errors and adjustment errors of points DP31 to DP34. Therefore, when the value Q is equal to or less than a predetermined value in the evaluation of the three-dimensional projective transformation matrix 101, it may be considered that the three-dimensional projective transformation matrix 101 can convert the projective reconstruction result of the region R1 into the Euclidean reconstruction result of the region R1. In this embodiment, when the value Q is equal to or less than a predetermined value, it is considered that the three-dimensional projective transformation matrix 101 can convert the projective reconstruction result of the region R1 into the Euclidean reconstruction result of the region R1.

[0105] When the value Q is greater than a predetermined value, the three-dimensional coordinate management unit 123 updates the third coordinate information 113. Specifically, the three-dimensional coordinate management unit 123 updates the third coordinate information 113 by changing the coordinates of the point DY31 indicated by the third coordinate information 113 to other coordinates on the line L1. Hereinafter, updating the third coordinate information 113 may be referred to as a "first update". Furthermore, when the first update is executed, the three-dimensional coordinate management unit 123 updates the three-dimensional projective transformation matrix 101 based on the updated third coordinate information 113, the first coordinate information 111, the second coordinate information 112, and the fourth coordinate information 114. Hereinafter, updating the three-dimensional projective transformation matrix 101 may be referred to as a "second update". Furthermore, when the second update is executed, the three-dimensional coordinate management unit 123 updates the fifth coordinate information 115 based on the updated three-dimensional projective transformation matrix 101 and the second coordinate information 112. Hereinafter, updating the fifth coordinate information 115 may be referred to as a "third update." That is, when the value Q is greater than a predetermined value, the processing device 12 repeatedly executes the first update, the second update, and the third update until the value Q becomes equal to or less than the predetermined value. This allows the processing device 12 to generate a three-dimensional projective transformation matrix 101 that transforms the projective reconstruction result of the region R1 into the Euclidean reconstruction result of the region R1.

[0106] The imaging element 140 and the imaging element 240 are image sensors, such as CCD or CMOS. Here, CCD is an abbreviation for Charge Coupled Device, and CMOS is an abbreviation for Complementary Metal Oxide Semiconductor. The imaging element 140 and the imaging element 240 each include a plurality of pixels. Under the control of the imaging control unit 121, the imaging device 14 captures an area including the region R1 of the screen SC1 on which the projection image GP1 is displayed. The imaging element 140 outputs to the processing device 12 first captured image information 106 representing the result of capturing an area including the region R1 of the screen SC1 on which the projection image GP1 is displayed. In other words, the imaging element 140 outputs to the processing device 12 a captured image GS1 indicated by the first captured image information 106. Under the control of the imaging control unit 121, the imaging device 24 captures an area including the region R2 of the screen SC1 on which a part of the projection image GP1 is displayed. The imaging element 240 outputs second captured image information 107 representing the result of capturing an image of a range including the region R2 of the screen SC1 on which a part of the projection image GP1 is displayed, to the processing device 12. In other words, the imaging element 240 outputs a captured image GS2 indicated by the second captured image information 107 to the processing device 12.

[0107] The optical modulator 160 and the optical modulator 260 include, for example, one or more DMDs or liquid crystal panels. The optical modulator 160 and the optical modulator 260 each include a plurality of pixels. Based on a signal input from the processing device 12, the optical modulator 160 and the optical modulator 260 modulate light emitted from a light source into projection light for displaying a projection image on a projection surface. The light source includes, for example, a halogen lamp, a xenon lamp, an extra-high pressure mercury lamp, an LED, or a laser light source. Here, LED is an abbreviation for Light Emitting Diode, and DMD is an abbreviation for Digital Mirror Device.

[0108] The projection device 16 projects projection light for displaying a projection image on a projection surface under the control of the projection control unit 120. In other words, the projection device 16 projects an image input from the processing device 12 onto the projection surface. In this embodiment, the projection device 16 projects projection light for displaying a projection image on a screen SC1 under the control of the projection control unit 120. Specifically, the projection device 16 projects an image GF1 input from the processing device 12 onto the screen SC1 to display a projection image GP1 on the screen SC1. In addition, the projection device 16 projects an image GF2 input from the processing device 12 onto the screen SC1 to display a projection image GP2 on the screen SC1. In addition, the projection device 16 projects an image GF3 input from the processing device 12 onto the screen SC1 to display a projection image GP3 on the screen SC1.

[0109] Specifically, the intrinsic parameters are parameters related to the optical system of the device used to measure the shape of the object. More specifically, the intrinsic parameters include parameters related to the lens of the device used to measure the shape of the object. For example, in the imaging device 14, the intrinsic parameters include parameters related to the imaging lens 142 and parameters related to the imaging element 140. In the imaging device 24, the intrinsic parameters include parameters related to the imaging lens 242 and parameters related to the imaging element 240. In the projection device 16, the intrinsic parameters include parameters related to the projection lens 162 and parameters related to the optical modulator 160. In the projection device 26, the intrinsic parameters include parameters related to the projection lens 262 and parameters related to the optical modulator 260.

[0110] The communication device 18 includes, for example, an interface board having a connector and an interface circuit, and has a function of receiving various information from an external terminal, an external storage device, an external server, or the like, and a function of transmitting various information to an external terminal, an external storage device, an external server, or the like. The communication device 18 may transmit and receive various information using wired communication, or may transmit and receive various information using wireless communication. When wireless communication is used, the communication device 18 is configured to include an antenna compatible with wireless communication that complies with a predetermined communication standard. In this embodiment, the communication device 18 is communicably connected to the projector 2, and transmits and receives various information to and from the projector 2.

[0111] The operation device 30 accepts an input operation on the projector 1 from a user of the projector 1. The operation device 30 includes, for example, a touch panel or operation buttons provided on the housing of the projector 1. When the operation device 30 includes a touch panel, the operation device 30 outputs data indicating the detected touch position to the processing device 12. When the operation device 30 includes operation buttons, the operation device 30 outputs data identifying the pressed button to the processing device 12. In this manner, the content of the input operation on the projector 1 is transmitted to the processing device 12. In this embodiment, the operation device 30 accepts a position adjustment operation from the user. Furthermore, the operation device 30 outputs operation data indicating the content of the position adjustment operation to the processing device 12.

[0112] 1.3. Operation of the measurement system 14 is a flowchart for explaining the operation of the measurement system Sys according to the embodiment. The series of operations shown in the flowchart is started, for example, when the projectors 1 and 2 are powered on and the projector 1 receives an input operation from a user to start the operation.

[0113] In step S101, the projection control unit 120 causes the projection device 16 to project projection light based on the first projection image information 103, thereby displaying the projection image GP1 on the screen SC1. In other words, the projection control unit 120 causes the projection device 16 to project the image GF1 indicated by the first projection image information 103, thereby displaying the projection image GP1 on the screen SC1.

[0114] In step S102, the imaging control unit 121 controls the imaging device 14 to capture an image of a range including the region R1 of the screen SC1 on which the projection image GP1 is displayed. That is, the imaging control unit 121 controls the imaging device 14 to capture an image of a plurality of points DP1 included in the projection image GP1. The imaging control unit 121 also acquires a captured image GS1 representing the result of the imaging. The imaging control unit 121 also stores the first captured image information 106 representing the captured image GS1 in the storage device 10.

[0115] In step S103, the imaging control unit 121 controls the imaging device 24 to capture an image of a range including the region R2 of the screen SC1 on which a part of the projection image GP1 is displayed. That is, the imaging control unit 121 controls the imaging device 24 to capture an image of a plurality of points DP1 included in the projection image GP1. The imaging control unit 121 also obtains a captured image GS2 representing the result of the imaging. The imaging control unit 121 also stores second captured image information 107 representing the captured image GS2 in the storage device 10.

[0116] In step S104, the image analysis unit 122 detects a plurality of points included in the captured image GS1 by executing image processing on the captured image GS1 indicated by the first captured image information 106. That is, the image analysis unit 122 acquires sixth coordinate information 131 representing the coordinates of a plurality of points included in the captured image GS1. Furthermore, the image analysis unit 122 stores the acquired sixth coordinate information 131 in the storage device 10.

[0117] Furthermore, the image analysis unit 122 detects a plurality of points included in the captured image GS2 by executing image processing on the captured image GS2 indicated by the second captured image information 107. That is, the image analysis unit 122 acquires seventh coordinate information 132 representing the coordinates of a plurality of points included in the captured image GS2. Furthermore, the image analysis unit 122 stores the acquired seventh coordinate information 132 in the storage device 10.

[0118] Furthermore, the image analysis unit 122 detects a plurality of points included in the image GF1 by executing image processing on the image GF1 indicated by the first projection image information 103. That is, the image analysis unit 122 acquires eighth coordinate information 133 representing the coordinates of a plurality of points included in the image GF1. Furthermore, the image analysis unit 122 stores the acquired eighth coordinate information 133 in the storage device 10.

[0119] In step S105, the image analysis unit 122 generates, based on the first projection image information 103 and the first captured image information 106, relationship information 116 indicating a correspondence relationship between a plurality of pixels included in the optical modulator 160 of the projection device 16 that projects the image GF1 indicated by the first projection image information 103 and a plurality of pixels included in the image sensor 140 of the imaging device 14 that generates the first captured image information 106. In other words, the image analysis unit 122 generates the relationship information 116 based on the image GF1 indicated by the first projection image information 103 and the captured image GS1 indicated by the first captured image information 106. Specifically, the image analysis unit 122 generates the relationship information 116 based on eighth coordinate information 133 indicating the coordinates of a plurality of points included in the image GF1 and sixth coordinate information 131 indicating the coordinates of a plurality of points included in the captured image GS1. Moreover, the image analysis unit 122 stores the generated relationship information 116 in the storage device 10.

[0120] In step S106, the projection control unit 120 causes the projection device 16 to project projection light based on the second projection image information 104, thereby displaying the projection image GP2 on the screen SC1. In other words, the projection control unit 120 causes the projection device 16 to project the image GF2 indicated by the second projection image information 104, thereby displaying the projection image GP2 on the screen SC1.

[0121] When the projection image GP2 is displayed on the screen SC1, the user performs a position adjustment operation to adjust the shape of a quadrangle having vertices DP21 to DP24 into a rectangle. The operation device 30 accepts from the user the position adjustment operation for adjusting the shape of a quadrangle having vertices DP21 to DP24 into a rectangle.

[0122] In step S107, the input management unit 124 controls the operation device 30 to obtain operation data indicating the content of the position adjustment operation received from the user.

[0123] In step S108, the image editing unit 125 updates the second projection image information 104 based on the position adjustment operation received from the user. In other words, the image editing unit 125 updates GF2 indicated by the second projection image information 104 to image GF3 based on the position adjustment operation received from the user. The projection device 16 projects the image GF3 input from the processing device 12 onto the screen SC1, thereby updating the projection image displayed on the screen SC1 from the projection image GP2 to the projection image GP3. In addition, the image editing unit 125 updates the ninth coordinate information 134 indicating the coordinates of a plurality of points included in the image indicated by the second projection image information 104 based on the position adjustment operation received from the user.

[0124] In step S109, the input management unit 124 judges whether or not operation data representing the operation to end the position adjustment of the points DP21 to DP24 has been acquired from the user. If operation data representing the operation to end the position adjustment of the points DP21 to DP24 has been acquired, that is, if the answer is YES in step S109, the input management unit 124 advances the process to step S107. If operation data representing the operation to end the position adjustment of the points DP21 to DP24 has not been acquired, that is, if the answer is NO in step S109, the input management unit 124 advances the process to step S107.

[0125] In step S110, the three-dimensional coordinate management unit 123 generates first coordinate information 111 representing the Euclidean reconstruction result of the region R3 based on the captured image GS1 acquired by the imaging device 14 whose internal parameters are known and the captured image GS2 acquired by the imaging device 24 whose internal parameters are known. Specifically, the three-dimensional coordinate management unit 123 generates the first coordinate information 111 based on the sixth coordinate information 131 and the seventh coordinate information 132. In addition, the three-dimensional coordinate management unit 123 stores the generated first coordinate information 111 in the storage device 10.

[0126] In step S111, the three-dimensional coordinate management unit 123 generates second coordinate information 112 representing the projective reconstruction result of the region R1 based on the captured image GS1 acquired by the imaging device 14 whose internal parameters are known and the image GF1 projected from the projection device 16 whose internal parameters are unknown. Specifically, the three-dimensional coordinate management unit 123 generates the second coordinate information 112 based on sixth coordinate information 131 representing the coordinates of multiple points included in the captured image GS1, eighth coordinate information 133 representing the coordinates of multiple points included in the image GF1, ninth coordinate information 134 updated based on the position adjustment operation, and the relationship information 116. In addition, the three-dimensional coordinate management unit 123 stores the generated second coordinate information 112 in the storage device 10.

[0127] In step S112, the three-dimensional coordinate management unit 123 generates third coordinate information 113 representing the coordinates of a point DY31 not included in the plane SV1, based on the first coordinate information 111. In addition, the three-dimensional coordinate management unit 123 stores the generated third coordinate information 113 in the storage device 10.

[0128] In step S113, the three-dimensional coordinate management unit 123 generates fourth coordinate information 114 representing the coordinates of a point DX31 not included in the plane SV2, based on the second coordinate information 112. In addition, the three-dimensional coordinate management unit 123 stores the generated fourth coordinate information 114 in the storage device 10.

[0129] In step S114, the three-dimensional coordinate management unit 123 generates a three-dimensional projective transformation matrix 101 for transforming the coordinates of the multiple points DX1 into the coordinates of the multiple points DY1, based on the first coordinate information 111, the second coordinate information 112, the third coordinate information 113, and the fourth coordinate information 114. In addition, the three-dimensional coordinate management unit 123 stores the generated three-dimensional projective transformation matrix 101 in the storage device 10.

[0130] In other words, the processing device 12 can generate a three-dimensional projection transformation matrix 101 that transforms the coordinates of multiple points existing on the same plane by using first coordinate information 111 that represents the coordinates in the coordinate system FY of multiple points DP1 included in the surface SC11, and second coordinate information 112 that represents the coordinates in the coordinate system FX of the multiple points DP1, as well as third coordinate information 113 and fourth coordinate information 114 that represent the coordinates of virtual points that are not included in the surface SC11 or its extension.

[0131] In step S115, the three-dimensional coordinate management unit 123 generates fifth coordinate information 115 representing the coordinates of the points DP31 to DP34 in the coordinate system FY, based on the second coordinate information 112 and the three-dimensional projective transformation matrix 101. In addition, the three-dimensional coordinate management unit 123 stores the generated fifth coordinate information 115 in the storage device 10.

[0132] In step S116, the three-dimensional coordinate management unit 123 evaluates the three-dimensional projective transformation matrix 101. That is, the three-dimensional coordinate management unit 123 judges whether or not the value Q is equal to or less than a predetermined value. If the value Q is equal to or less than the predetermined value, that is, if YES in step S116, the processing device 12 including the three-dimensional coordinate management unit 123 ends the series of operations shown in the flowchart of Fig. 14. If the value Q is greater than the predetermined value, that is, if NO in step S116, the three-dimensional coordinate management unit 123 advances the process to step S117.

[0133] In step S117, the three-dimensional coordinate management unit 123 executes a first update to update the third coordinate information 113. Specifically, the three-dimensional coordinate management unit 123 executes the first update by changing the coordinates of the point DY31 indicated by the third coordinate information 113 to other coordinates on the straight line L1.

[0134] In step S118, the three-dimensional coordinate management unit 123 performs a second update to update the three-dimensional projection transformation matrix 101 based on the third coordinate information 113 updated by the first update, the first coordinate information 111, the second coordinate information 112, and the fourth coordinate information 114.

[0135] In step S119, the three-dimensional coordinate management unit 123 executes a third update to update the fifth coordinate information 115 based on the three-dimensional projective transformation matrix 101 updated by the second update and the second coordinate information 112.

[0136] The processing device 12 repeatedly executes the first update, the second update, and the third update until the value Q becomes equal to or less than a predetermined value, in other words, until the determination result in step S116 becomes YES. This allows the processing device 12 to generate a three-dimensional projective transformation matrix 101 that transforms the projective reconstruction result of the region R1 into the Euclidean reconstruction result of the region R1.

[0137] As described above, according to the embodiment, the measurement system Sys generates a three-dimensional projection transformation matrix using the coordinates of multiple points existing on the same plane as well as the coordinates of virtual points not existing on the plane. That is, the measurement system Sys can generate a three-dimensional projection transformation matrix that transforms the coordinates of multiple points existing on the same plane.

[0138] According to the embodiment, the measurement system Sys can obtain a Euclidean reconstruction result of an area including a plane by transforming the projective reconstruction result of the area including the plane using a three-dimensional projective transformation matrix that transforms the coordinates of multiple points existing on the same plane. That is, the measurement system Sys can accurately measure the shape of an object even if the object to be measured includes a planar area and a non-planar area.

[0139] As described above, the measurement method according to the embodiment includes the steps of: generating first coordinate information 111 representing the coordinates in the coordinate system FY of multiple points DP1 included in the region R3 of the screen SC1 including the surface SC11, which is a planar surface, and the surface SC12, generating second coordinate information 112 representing the coordinates in the coordinate system FX of the multiple points DP1; generating third coordinate information 113 representing the coordinates in the coordinate system FY of a point DY31 not included in the plane SV1; generating fourth coordinate information 114 representing the coordinates in the coordinate system FX of a point DX31 not included in the plane SV2; and calculating the coordinates of the multiple points DP1 in the coordinate system FX based on the first coordinate information 111, the second coordinate information 112, the third coordinate information 113, and the fourth coordinate information 114. generating, based on the three-dimensional projective transformation matrix 101, fifth coordinate information 115 indicating coordinates in the coordinate system FY of points DP31 to DP34 included in the screen SC1; and, if the coordinates indicated by the fifth coordinate information 115 do not satisfy a predetermined condition, performing a first update to update at least one of the third coordinate information 113 and the fourth coordinate information 114, performing a second update to update the three-dimensional projective transformation matrix 101 based on a result of the first update, and performing a third update to update the fifth coordinate information 115 based on the three-dimensional projective transformation matrix 101 updated by the second update, wherein a quadrangle having the points DP31 to DP34 as vertices has a predetermined shape, and at least one of the points DP31 to DP34 is included in the face SC12.

[0140] Furthermore, in a screen SC1 including a surface SC11 which is a plane and a surface SC12 which is a non-planar surface, the measurement system Sys according to the embodiment performs the following operations: generates first coordinate information 111 which represents coordinates in a coordinate system FY of a plurality of points DP1 included in an area R3 of the surface SC11; generates second coordinate information 112 which represents coordinates in a coordinate system FX of the plurality of points DP1; generates third coordinate information 113 which represents coordinates in the coordinate system FY of a point DY31 which is not included in the plane SV1; generates fourth coordinate information 114 which represents coordinates in the coordinate system FX of a point DX31 which is not included in the plane SV2; and converts the coordinates of the plurality of points DP1 in the coordinate system FX into coordinates of the plurality of points DP1 in the coordinate system FY based on the first coordinate information 111, the second coordinate information 112, the third coordinate information 113, and the fourth coordinate information 114. the processing device 12 executes the following operations: generating a three-dimensional projective transformation matrix 101 for projecting a screen SC1; generating fifth coordinate information 115 representing coordinates in the coordinate system FY of points DP31 to DP34 included in the screen SC1 based on the three-dimensional projective transformation matrix 101; if the coordinates indicated by the fifth coordinate information 115 do not satisfy a predetermined condition, performing a first update to update at least one of the third coordinate information 113 and the fourth coordinate information 114, performing a second update to update the three-dimensional projective transformation matrix 101 based on a result of the first update, and performing a third update to update the fifth coordinate information 115 based on the three-dimensional projective transformation matrix 101 updated by the second update; wherein a quadrangle having the points DP31 to DP34 as vertices has a predetermined shape, and at least one of the points DP31 to DP34 is included in the surface SC12.

[0141] Furthermore, the projector 1 according to the embodiment generates first coordinate information 111 representing the coordinates in the coordinate system FY of a plurality of points DP1 included in an area R3 of the surface SC11, generates second coordinate information 112 representing the coordinates in the coordinate system FX of the plurality of points DP1, generates third coordinate information 113 representing the coordinates in the coordinate system FY of a point DY31 not included in the plane SV1, generates fourth coordinate information 114 representing the coordinates in the coordinate system FX of a point DX31 not included in the plane SV2, and converts the coordinates of the plurality of points DP1 in the coordinate system FX into the coordinates of the plurality of points DP1 in the coordinate system FY based on the first coordinate information 111, the second coordinate information 112, the third coordinate information 113, and the fourth coordinate information 114. the processing device 12 executes the following operations: generating a three-dimensional projective transformation matrix 101 for projecting a screen SC1; generating fifth coordinate information 115 representing coordinates in the coordinate system FY of points DP31 to DP34 included in the screen SC1 based on the three-dimensional projective transformation matrix 101; if the coordinates indicated by the fifth coordinate information 115 do not satisfy a predetermined condition, performing a first update to update at least one of the third coordinate information 113 and the fourth coordinate information 114, performing a second update to update the three-dimensional projective transformation matrix 101 based on a result of the first update, and performing a third update to update the fifth coordinate information 115 based on the three-dimensional projective transformation matrix 101 updated by the second update; wherein a quadrangle having the points DP31 to DP34 as vertices has a predetermined shape, and at least one of the points DP31 to DP34 is included in the surface SC12.

[0142] That is, the measurement system Sys generates a three-dimensional projective transformation matrix 101 using the coordinates of multiple points existing on the same plane as well as the coordinates of virtual points not existing on the plane. This allows the measurement system Sys to generate a three-dimensional projective transformation matrix 101 that transforms the coordinates of multiple points existing on the same plane. Furthermore, the three-dimensional projective transformation matrix 101 can transform the coordinate system of multiple points in an area including a planar portion and a non-planar portion. Therefore, the measurement system Sys can measure the shape of an object even if the object to be measured includes a planar area and a non-planar area.

[0143] In the embodiment, the measurement system Sys is an example of a "measurement system", the projector 1 is an example of an "information processing device", the surface SC11 is an example of a "first surface", the surface SC12 is an example of a "second surface", the screen SC1 is an example of a "screen", the area R3 is an example of a "first area", the multiple points DP1 are an example of "multiple first points", the coordinate system FY is an example of a "first coordinate system", the first coordinate information 111 is an example of "first coordinate information", and the coordinate system FX is an example of a "second coordinate system". ", the second coordinate information 112 is an example of "second coordinate information", the point DY31 is an example of "second point", the third coordinate information 113 is an example of "third coordinate information", the point DX31 is an example of "third point", the fourth coordinate information 114 is an example of "fourth coordinate information", the three-dimensional projective transformation matrix 101 is an example of "three-dimensional projective transformation matrix", the points DP31 to DP34 are examples of "four set points", the fifth coordinate information 115 is an example of "fifth coordinate information", and the processing device 12 is an example of "processing device". Furthermore, "in the first coordinate system, a plane including a plurality of first points" is an example of plane SV1. Furthermore, "in the second coordinate system, a plane including a plurality of first points" is an example of plane SV2. Furthermore, "the predetermined shape" is an example of a rectangle. Furthermore, "the predetermined condition" is an example of the value Q being equal to or less than a predetermined value.

[0144] In addition, the measurement method of the embodiment further includes controlling the imaging device 14 to obtain an captured image GS1 that captures an area including the region R3, and at least one of the first coordinate information 111 and the second coordinate information 112 is generated based on the captured image GS1.

[0145] That is, the measurement system Sys uses an imaging device to measure the shape of an object, thereby enabling the measurement system Sys to accurately measure the shape of the object.

[0146] In the embodiment, the imaging device 14 is an example of an "imaging device", and the captured image GS1 is an example of a "captured image".

[0147] Furthermore, in the measurement method of the embodiment, point DY31 is located on a straight line L1 in the coordinate system FY connecting point DY21 indicating the position of the imaging device 14 and point DY11 included in the plane SV1, and point DX31 is located on a straight line L2 in the coordinate system FX connecting point DX21 indicating the position of the imaging device 14 and point DX11 which is included in the plane SV2 and corresponds to point DY11.

[0148] That is, the measurement system Sys sets the coordinates of a virtual point that does not exist on the plane on a straight line that connects a point indicating the position of the imaging device and a point included in the plane. This allows the measurement system Sys to increase the accuracy of generating the three-dimensional projective transformation matrix 101.

[0149] In the embodiment, point DY11 is an example of a "fourth point", point DX11 is an example of a "fifth point", line L1 is an example of a "first line", and line L2 is an example of a "second line". Also, point DY21 is an example of a "point indicating the position of the imaging device in the first coordinate system". Also, point DX21 is an example of a "point indicating the position of the imaging device in the second coordinate system".

[0150] Furthermore, in the measurement method of the embodiment, when the first update includes updating the third coordinate information 113, updating the third coordinate information 113 includes changing the coordinate of point DY31 to other coordinates on the straight line L1, and when the first update includes updating the fourth coordinate information 114, updating the fourth coordinate information 114 includes changing the coordinate of point DX31 to other coordinates on the straight line L2.

[0151] That is, the measurement system Sys changes the coordinates of virtual points that do not exist on the plane on a line connecting a point indicating the position of the imaging device and a point included in the plane, thereby enabling the measurement system Sys to update the three-dimensional projective transformation matrix 101 with high accuracy and high speed.

[0152] Furthermore, in the measurement method of the embodiment, generating the fifth coordinate information 115 includes displaying a projected image GP3 on the screen SC1 by projecting an image GF3 including points DF31 to DF34 that correspond one-to-one to points DP31 to DP34 from a projection device 16 that projects an image onto the screen SC1, and points DP31 to DP34 are included in the projected image GP3.

[0153] That is, the measurement system Sys can easily display the positions of the points DP31 to DP34, which enables the user to easily adjust the positions of the points DP31 to DP34.

[0154] In the embodiment, the projection device 16 is an example of a "projection device," the points DF31 to DF34 are examples of the "four sixth points," the image GF3 is an example of the "first pattern image," and the projected image GP3 is an example of the "first projected image."

[0155] In addition, the measurement method of the embodiment further includes displaying a projected image GP1 on the screen SC1 by projecting an image GF1 including a plurality of points DF1 that correspond one-to-one to the plurality of points DP1 from a projection device 16 that projects an image onto the screen SC1, and the captured image GS1 includes a plurality of points DV1 that correspond one-to-one to the plurality of points DP1, and either the first coordinate information 111 or the second coordinate information 112 is generated based on the captured image GS1 and the image GF1.

[0156] That is, the measurement system Sys uses an imaging device and a projection device to measure the shape of an object, thereby enabling the measurement system Sys to accurately measure the shape of the object.

[0157] In the embodiment, the multiple points DF1 are an example of a "multiple seventh points", the image GF1 is an example of a "second pattern image", the projected image GP1 is an example of a "second projected image", and the multiple points DV1 are an example of a "multiple eighth points".

[0158] Furthermore, in the measurement method according to the embodiment, generating the first coordinate information 111 includes generating the first coordinate information 111 based on a captured image GS1, which is one of an image input to an imaging device 14 having an imaging lens 142 and an image output from the imaging device 14, and a captured image GS2, which is one of an image input to an imaging device 24 having an imaging lens 242 and an internal parameter related to the imaging lens 242, and an image output from the imaging device 24, and generating the second coordinate information 112 includes generating the second coordinate information 112 based on an image GF1, which is one of an image input to a projection device 16 having a projection lens 162 and an image output from the projection device 16, and the captured image GS1, and at least one of the imaging devices 14, 24, and 16 is an imaging device that captures the screen SC1.

[0159] That is, the first coordinate information 111 represents the Euclidean reconstruction result of the region R3 set on a plane. The second coordinate information 112 represents the projective reconstruction result of the region R3. Therefore, the three-dimensional projective transformation matrix 101 can convert the projective reconstruction result of the region including the region R3 into the Euclidean reconstruction result of the region including the region R3. This allows the measurement system Sys to accurately measure the shape of an object even if the object to be measured includes a planar region and a non-planar region.

[0160] In the embodiment, the imaging lens 142 is an example of a "first lens," the imaging device 14 is an example of a "first device," the captured image GS1 is an example of a "first image," the imaging lens 242 is an example of a "second lens," the imaging device 24 is an example of a "second device," the captured image GS2 is an example of a "second image," the projection lens 162 is an example of a "third lens," the projection device 16 is an example of a "third device," and the image GF1 is an example of a "third image."

[0161] 2. Variations The above embodiment may be modified in various ways. Specific modified aspects are exemplified below. Two or more aspects arbitrarily selected from the following examples may be appropriately combined within a range that does not contradict each other. In the modified examples exemplified below, the elements whose actions and functions are equivalent to those of the above embodiment will be appropriately omitted by using the symbols used in the above explanation.

[0162] 2.1. Variation 1 In the above embodiment, in a measurement system including two imaging devices with known internal parameters and a projection device with unknown internal parameters, a Euclidean restoration result is obtained using the projection device with unknown internal parameters without performing a calibration operation of the projection device with unknown internal parameters, but the present invention is not limited to such an embodiment. For example, the measurement system may include an imaging device with known internal parameters, a projection device with known internal parameters, and an imaging device with unknown internal parameters. In a measurement system including an imaging device with known internal parameters, a projection device with known internal parameters, and an imaging device with unknown internal parameters, a Euclidean restoration result can be obtained using the imaging device with unknown internal parameters without performing a calibration operation of the imaging device with unknown internal parameters.

[0163] Furthermore, the measurement system according to the present invention is not limited to the above-mentioned aspects, and may be, for example, a measurement system including two image capture devices with known internal parameters and an image capture device with unknown internal parameters, or a measurement system including an image capture device with known internal parameters, a projection device with known internal parameters, and a projection device with unknown internal parameters. It is sufficient for the measurement system according to the present invention to include at least one image capture device.

[0164] 2.2. Variation 2 In the above embodiment and modified example, a case where a plurality of points DP1, which is an example of a plurality of first points, and points DP31 to DP34, which are an example of four set points, are displayed at different timings, is illustrated, but the present invention is not limited to such an embodiment. The plurality of first points and the four set points may be displayed simultaneously. This can reduce the time required to switch the projected image, for example.

[0165] 2.3. Variation 3 In the above-described embodiment and modified example, the point DY11 is included in the multiple points DY1, but the point DY11 does not have to be included in the multiple points DY1. The point DY11 only needs to be included in the plane SV1, which is a plane that includes the multiple points DY1.

[0166] 2.4. Variation 4 In the above-described embodiment and modified example, the case where the third coordinate information 113 is updated in the first update is exemplified, but the present invention is not limited to such an aspect. In the first update, the fourth coordinate information 114 may be updated instead of the third coordinate information 113. For example, in the first update, the processing device included in the measurement system according to the present invention may update the fourth coordinate information 114 by changing the coordinate of the point DX31 indicated by the fourth coordinate information 114 to another coordinate on the line L2. In addition, the processing device included in the measurement system according to the present invention may update both the third coordinate information 113 and the fourth coordinate information 114 in the first update.

[0167] 3. Notes The present invention will be summarized below as an appendix.

[0168] Appendix 1 The present invention relates to a screen including a first surface which is a plane and a second surface which is a non-planar surface, and includes the steps of: generating first coordinate information representing coordinates in a first coordinate system of a plurality of first points included in a first region of the first surface; generating second coordinate information representing coordinates in a second coordinate system of the plurality of first points; generating third coordinate information representing coordinates in the first coordinate system of a second point not included in a plane including the plurality of first points; generating fourth coordinate information representing coordinates in the second coordinate system of a third point not included in a plane including the plurality of first points; generating a three-dimensional projection transformation matrix for converting the four set points included in the screen into coordinates of the first coordinate system; generating fifth coordinate information representing coordinates in the first coordinate system of four set points included in the screen based on the three-dimensional projection transformation matrix; and if coordinates indicated by the fifth coordinate information do not satisfy a predetermined condition, performing a first update to update at least one of the third coordinate information and the fourth coordinate information, performing a second update to update the three-dimensional projection transformation matrix based on a result of the first update, and performing a third update to update the fifth coordinate information based on the three-dimensional projection transformation matrix updated by the second update, wherein a quadrangle having the four set points as vertices has a predetermined shape, and at least one of the four set points is included in the second surface.

[0169] That is, the measurement system realizing the measurement method described in Supplementary Note 1 generates a three-dimensional projective transformation matrix using the coordinates of a plurality of points existing on the same plane as well as the coordinates of a virtual point not existing on the plane. In this way, the measurement system realizing the measurement method described in Supplementary Note 1 can generate a three-dimensional projective transformation matrix that transforms the coordinates of a plurality of points existing on the same plane. In addition, the three-dimensional projective transformation matrix generated by the measurement system realizing the measurement method described in Supplementary Note 1 can transform the coordinate system of a plurality of points in an area including a planar portion and a non-planar portion. Therefore, the measurement system realizing the measurement method described in Supplementary Note 1 can measure the shape of an object even if the object to be measured includes a planar area and a non-planar area.

[0170] Appendix 2 The measurement method described in Appendix 1, further comprising controlling an imaging device to obtain an image capturing an area including the first region, wherein at least one of the first coordinate information and the second coordinate information is generated based on the captured image.

[0171] That is, the measurement system that realizes the measurement method described in Supplementary Note 2 measures the shape of an object using an imaging device. As a result, the measurement system that realizes the measurement method described in Supplementary Note 2 can accurately measure the shape of an object.

[0172] Appendix 3 The measurement method described in Appendix 2, wherein the second point is located on a first line connecting, in the first coordinate system, a point indicating the position of the imaging device and a fourth point included in a plane including the multiple first points, and the third point is located on a second line connecting, in the second coordinate system, a point indicating the position of the imaging device and a fifth point corresponding to the fourth point, which is included in a plane including the multiple first points.

[0173] That is, the measurement system that realizes the measurement method described in Supplementary Note 3 sets the coordinates of a virtual point that does not exist on a plane on a straight line that connects a point indicating the position of the imaging device and a point included in the plane. This allows the measurement system that realizes the measurement method described in Supplementary Note 3 to increase the accuracy of generating a three-dimensional projection transformation matrix.

[0174] Appendix 4 The measurement method described in Appendix 3, wherein, when the first update includes updating the third coordinate information, updating the third coordinate information includes changing a coordinate of the second point to another coordinate on the first line, and when the first update includes updating the fourth coordinate information, updating the fourth coordinate information includes changing a coordinate of the third point to another coordinate on the second line.

[0175] That is, the measurement system that realizes the measurement method described in Supplementary Note 4 changes the coordinates of a virtual point that does not exist on a plane on a straight line that connects a point indicating the position of the imaging device and a point included in the plane. This allows the measurement system that realizes the measurement method described in Supplementary Note 4 to update the three-dimensional projective transformation matrix with high accuracy and high speed.

[0176] Appendix 5 The measurement method according to any one of Appendix 1 to Appendix 4, wherein generating the fifth coordinate information includes projecting a first pattern image including four sixth points that correspond one-to-one with the four set points from a projection device that projects an image onto the screen, thereby displaying a first projected image on the screen, wherein the four set points are included in the first projected image.

[0177] That is, the measurement system that realizes the measurement method described in Supplementary Note 5 can easily display the positions of the four set points. This allows the user to easily adjust the positions of the four set points.

[0178] Appendix 6 The measurement method according to any one of Supplementary Note 2 to Supplementary Note 4, further comprising: displaying a second projected image on the screen by projecting a second pattern image including a plurality of seventh points that correspond one-to-one with the plurality of first points from a projection device that projects an image onto the screen, wherein the captured image includes a plurality of eighth points that correspond one-to-one with the plurality of first points, and either one of the first coordinate information and the second coordinate information is generated based on the captured image and the second pattern image.

[0179] That is, the measurement system that realizes the measurement method described in Supplementary Note 6 measures the shape of an object using an imaging device and a projection device. As a result, the measurement system that realizes the measurement method described in Supplementary Note 6 can accurately measure the shape of an object.

[0180] Appendix 7 The measurement method of claim 1, wherein generating the first coordinate information includes generating the first coordinate information based on a first image, the first image being one of an image input to a first device having a first lens and an internal parameter related to the first lens being known and an image output from the first device, and a second image, the second image being one of an image input to a second device having a second lens and an internal parameter related to the second lens being known and an image output from the second device; generating the second coordinate information includes generating the second coordinate information based on a third image, the third image being one of an image input to a third device having a third lens and an internal parameter related to the third lens being unknown and an image output from the third device, and the first image; and at least one of the first device, the second device, and the third device is an imaging device that images the screen.

[0181] That is, the first coordinate information represents a Euclidean reconstruction result of a first region set on a plane. Moreover, the second coordinate information represents a projective reconstruction result of the first region. Therefore, the three-dimensional projective transformation matrix can transform the projective reconstruction result of a region including the first region into a Euclidean reconstruction result of the region including the first region. As a result, the measurement system that realizes the measurement method described in Supplementary Note 7 can accurately measure the shape of an object even if the object to be measured includes a planar region and a non-planar region.

[0182] Appendix 8 A method for generating a screen including a first surface which is a plane and a second surface which is a non-planar surface, the method comprising: generating first coordinate information representing coordinates in a first coordinate system of a plurality of first points included in a first region of the first surface; generating second coordinate information representing coordinates in a second coordinate system of the plurality of first points; generating third coordinate information representing coordinates in the first coordinate system of a second point not included in a plane including the plurality of first points; generating fourth coordinate information representing coordinates in the second coordinate system of a third point not included in a plane including the plurality of first points; and converting the coordinates of the plurality of first points in the second coordinate system into coordinates of the plurality of first points in the first coordinate system based on the first coordinate information, the second coordinate information, the third coordinate information, and the fourth coordinate information. generating a three-dimensional projective transformation matrix for transforming the screen into a first coordinate system; generating fifth coordinate information representing coordinates in the first coordinate system of four set points included in the screen based on the three-dimensional projective transformation matrix; and, if coordinates indicated by the fifth coordinate information do not satisfy a predetermined condition, performing a first update to update at least one of the third coordinate information and the fourth coordinate information, performing a second update to update the three-dimensional projective transformation matrix based on a result of the first update, and performing a third update to update the fifth coordinate information based on the three-dimensional projective transformation matrix updated by the second update, wherein a quadrangle having the four set points as vertices has a predetermined shape, and at least one of the four set points is included in the second surface.

[0183] That is, the measurement system described in Supplementary Note 8 generates a three-dimensional projective transformation matrix using the coordinates of a plurality of points existing on the same plane as well as the coordinates of a virtual point not existing on the plane. This allows the measurement system described in Supplementary Note 8 to generate a three-dimensional projective transformation matrix that transforms the coordinates of a plurality of points existing on the same plane. Furthermore, the three-dimensional projective transformation matrix generated by the measurement system described in Supplementary Note 8 can transform the coordinate system of a plurality of points in an area including a planar portion and a non-planar portion. Therefore, the measurement system described in Supplementary Note 8 can measure the shape of an object even if the object to be measured includes a planar area and a non-planar area.

[0184] Appendix 9 A method for generating a screen including a first surface which is a plane and a second surface which is a non-planar surface, the method comprising: generating first coordinate information representing coordinates in a first coordinate system of a plurality of first points included in a first region of the first surface; generating second coordinate information representing coordinates in a second coordinate system of the plurality of first points; generating third coordinate information representing coordinates in the first coordinate system of a second point not included in a plane including the plurality of first points; generating fourth coordinate information representing coordinates in the second coordinate system of a third point not included in a plane including the plurality of first points; and converting the coordinates of the plurality of first points in the second coordinate system into coordinates of the plurality of first points in the first coordinate system based on the first coordinate information, the second coordinate information, the third coordinate information, and the fourth coordinate information. generating a three-dimensional projective transformation matrix for transforming the screen into a first coordinate system; generating fifth coordinate information representing coordinates in the first coordinate system of four set points included in the screen based on the three-dimensional projective transformation matrix; and, if coordinates indicated by the fifth coordinate information do not satisfy a predetermined condition, performing a first update to update at least one of the third coordinate information and the fourth coordinate information, performing a second update to update the three-dimensional projective transformation matrix based on a result of the first update, and performing a third update to update the fifth coordinate information based on the three-dimensional projective transformation matrix updated by the second update, wherein a quadrangle having the four set points as vertices has a predetermined shape, and at least one of the four set points is included in the second surface.

[0185] That is, the measurement system including the information processing device described in Supplementary Note 9 generates a three-dimensional projective transformation matrix using the coordinates of a plurality of points existing on the same plane as well as the coordinates of a virtual point not existing on the plane. This allows the measurement system including the information processing device described in Supplementary Note 9 to generate a three-dimensional projective transformation matrix that transforms the coordinates of a plurality of points existing on the same plane. In addition, the three-dimensional projective transformation matrix generated by the measurement system including the information processing device described in Supplementary Note 9 can transform the coordinate system of a plurality of points in an area including a planar portion and a non-planar portion. Therefore, the measurement system including the information processing device described in Supplementary Note 9 can measure the shape of an object even if the object to be measured includes a planar area and a non-planar area. [Explanation of symbols]

[0186] 1...projector, 2...projector, 10...storage device, 12...processing device, 14...imaging device, 16...projection device, 18...communication device, 30...operation device, 100...program, 101...three-dimensional projective transformation matrix, 102...projected image information, 105...captured image information, 110...three-dimensional coordinate information, 116...relationship information, 130...two-dimensional coordinate information, 120...projection control unit, 121...imaging control unit, 122...image analysis unit, 123...three-dimensional coordinate management unit, 124...input management unit, 125...image editing unit, 140...imaging element, 142...imaging lens, 160...optical modulator, 162...projection lens, GP1...projected image, GF1...image, GS1...captured image, DP1...point, R1...area, SC1...screen, FY...coordinate system, Sys...measurement system.

Claims

1. In a screen including a first surface that is planar and a second surface that is non-planar, generating first coordinate information representing coordinates in a first coordinate system of a plurality of first points included in a first region of the first surface; generating second coordinate information representing coordinates in a second coordinate system of the plurality of first points; generating third coordinate information representing coordinates of a second point not included in a plane containing the plurality of first points in the first coordinate system; generating fourth coordinate information representing coordinates of a third point not included in a plane containing the plurality of first points in the second coordinate system; generating a three-dimensional projective transformation matrix for transforming coordinates of the plurality of first points in the second coordinate system into coordinates of the plurality of first points in the first coordinate system based on the first coordinate information, the second coordinate information, the third coordinate information, and the fourth coordinate information; generating fifth coordinate information representing coordinates in the first coordinate system of four set points included in the screen based on the three-dimensional projective transformation matrix; when the coordinates indicated by the fifth coordinate information do not satisfy a predetermined condition, performing a first update to update at least one of the third coordinate information and the fourth coordinate information, performing a second update to update the three-dimensional projective transformation matrix based on a result of the first update, and performing a third update to update the fifth coordinate information based on the three-dimensional projective transformation matrix updated by the second update; including a quadrilateral having the four set points as vertices has a predetermined shape; at least one of the four set points is included in the second surface; measurement method.

2. further including acquiring a captured image obtained by capturing a range including the first region by controlling an imaging device, wherein at least one of the first coordinate information and the second coordinate information is generated based on the captured image; The measurement method according to claim 1.

3. The second point is located on a first straight line connecting a point indicating a position of the imaging device in the first coordinate system and a fourth point included in a plane containing the plurality of first points; The third point is located on a second straight line connecting a point indicating a position of the imaging device in the second coordinate system and a point included in a plane containing the plurality of first points and corresponding to the fourth point; The measurement method according to claim 2.

4. When the first update includes updating the third coordinate information, updating the third coordinate information includes changing coordinates of the second point to other coordinates on the first straight line; When the first update includes updating the fourth coordinate information, updating the fourth coordinate information includes changing coordinates of the third point to other coordinates on the second straight line. The measurement method according to claim 3.

5. generating the fifth coordinate information includes projecting a first pattern image including four sixth points that correspond one-to-one to the four set points from a projection device that projects an image onto the screen, thereby displaying a first projection image on the screen; The four set points are included in the first projection image. The measurement method according to any one of claims 1 to 4.

6. a projection device that projects an image onto the screen, and a second pattern image including a plurality of seventh points that correspond one-to-one to the plurality of first points is projected onto the screen, thereby displaying a second projection image on the screen; the captured image includes a plurality of eighth points that correspond one-to-one to the plurality of first points; One of the first coordinate information and the second coordinate information is generated based on the captured image and the second pattern image. The measurement method according to any one of claims 2 to 4.

7. generating the first coordinate information based on a first image, the first image being one of an image input to a first device having a first lens and an internal parameter related to the first lens being known and an image output from the first device, and a second image, the second image being one of an image input to a second device having a second lens and an internal parameter related to the second lens being known and an image output from the second device; generating the second coordinate information based on a third image, the third image being one of an image input to a third device having a third lens and an internal parameter related to the third lens being unknown and an image output from the third device, and the first image; At least one of the first device, the second device, and the third device is an imaging device that images the screen. The measurement method according to claim 1 .

8. In a screen including a first surface that is a plane and a second surface that is a non-planar surface, generating first coordinate information that represents coordinates in a first coordinate system of a plurality of first points included in a first area of ​​the first surface; generating second coordinate information representing coordinates of the first points in a second coordinate system; generating third coordinate information representing coordinates of a second point not included in a plane including the plurality of first points in the first coordinate system; generating fourth coordinate information representing coordinates of a third point not included in a plane including the plurality of first points in the second coordinate system; generating a three-dimensional projection transformation matrix for transforming coordinates of the plurality of first points in the second coordinate system into coordinates of the plurality of first points in the first coordinate system based on the first coordinate information, the second coordinate information, the third coordinate information, and the fourth coordinate information; generating fifth coordinate information representing coordinates in the first coordinate system of four set points included in the screen based on the three-dimensional projection transformation matrix; when the coordinates indicated by the fifth coordinate information do not satisfy a predetermined condition, performing a first update to update at least one of the third coordinate information and the fourth coordinate information, performing a second update to update the three-dimensional projective transformation matrix based on a result of the first update, and performing a third update to update the fifth coordinate information based on the three-dimensional projective transformation matrix updated by the second update; a processor for executing The quadrangle having the four set points as vertices has a predetermined shape, At least one of the four set points is included in the second surface. Measurement system.

9. In a screen including a first surface that is a plane and a second surface that is a non-planar surface, generating first coordinate information that represents coordinates in a first coordinate system of a plurality of first points included in a first area of ​​the first surface; generating second coordinate information representing coordinates of the first points in a second coordinate system; generating third coordinate information representing coordinates of a second point not included in a plane including the plurality of first points in the first coordinate system; generating fourth coordinate information representing coordinates of a third point not included in a plane including the plurality of first points in the second coordinate system; generating a three-dimensional projection transformation matrix for transforming coordinates of the plurality of first points in the second coordinate system into coordinates of the plurality of first points in the first coordinate system based on the first coordinate information, the second coordinate information, the third coordinate information, and the fourth coordinate information; generating fifth coordinate information representing coordinates in the first coordinate system of four set points included in the screen based on the three-dimensional projection transformation matrix; when the coordinates indicated by the fifth coordinate information do not satisfy a predetermined condition, performing a first update to update at least one of the third coordinate information and the fourth coordinate information, performing a second update to update the three-dimensional projective transformation matrix based on a result of the first update, and performing a third update to update the fifth coordinate information based on the three-dimensional projective transformation matrix updated by the second update; a processor for executing The quadrangle having the four set points as vertices has a predetermined shape, At least one of the four set points is included in the second surface. Information processing device.

Images