Position determination method, simulation method, position determination system and simulation system

Abstract

Position determination method, simulation method, position determination system, and simulation system. Provided is a technology capable of assisting in projecting a prescribed image on an object even when a measurer such as a projector and a camera is arranged at an arbitrary position. The position determination method includes the steps of generating relationship information indicating a correspondence relationship between a three-dimensional coordinate system used by a measurer that determines a first coordinate of a three-dimensional portion in which a specific point of an object is present and a projector coordinate system that determines a second coordinate of the specific point in a projected image and a coordinate of a projector, the projected image being projected on the object by the projector, based on the first coordinate and the second coordinate, and determining a position of the projector in the three-dimensional coordinate system based on the coordinate of the projector by using the relationship information.

Description

Technical Field

[0001] This invention relates to a location determination method, a simulation method, a location determination system, and a simulation system. Background Technology

[0002] Patent Document 1 discloses a system for projecting a predetermined image onto an object using a projector and a camera in combination. In this system, the positional relationship between the projector and the camera is predetermined.

[0003] Patent Document 1: U.S. Patent Application Publication No. 2016 / 0343125

[0004] In the system described in Patent Document 1, the positional relationship between the projector and the camera is predetermined, therefore, it may be impossible to place the projector and camera, or other measuring devices, in arbitrary positions. Therefore, a technology is desired that can assist in situations where the projector and camera, or other measuring devices, are placed in arbitrary positions, projecting a predetermined image onto the object. Summary of the Invention

[0005] One aspect of the position determination method of the present invention includes the following steps: generating relational information based on a three-dimensional first coordinate of the portion of the object containing the specific point and a two-dimensional second coordinate of the specific point in the projected image, wherein the projector projects a projected image containing the specific point onto the object, the relational information representing the correspondence between the three-dimensional coordinate system used by the measuring instrument that determines the first coordinate and the projector coordinate system that determines the second coordinate and the coordinates of the projector; and determining the position of the projector in the three-dimensional coordinate system based on the coordinates of the projector by using the relational information.

[0006] One aspect of the simulation method of the present invention includes the following steps: generating first relational information based on a three-dimensional first coordinate of the portion of the object containing the first specific point and a two-dimensional second coordinate of the first specific point in the projected image, wherein the first relational information represents the correspondence between the three-dimensional coordinate system used by the measuring instrument that determines the first coordinate and the projector coordinate system that determines the second coordinate and the coordinates of the projector; generating second relational information based on the coordinates of the second specific point in the object in the three-dimensional coordinate system and the coordinates of a third specific point corresponding to the second specific point in a three-dimensional imaginary space, wherein the second relational information represents the correspondence between the three-dimensional coordinate system and the imaginary space coordinate system that determines the coordinates in the imaginary space; and determining the position of the projector in the imaginary space coordinate system based on the coordinates of the projector by using the first relational information and the second relational information.

[0007] One aspect of the simulation method of the present invention includes the following steps: generating first information based on shooting coordinates and projection coordinates, wherein the shooting coordinates are two-dimensional coordinates of the portion of the object where the first specific point is located in a shooting image generated by a camera when the projector projects a projection image of the object having a first specific point onto the object, and the projection coordinates are two-dimensional coordinates of the first specific point in the projection image; the first information represents the correspondence between the camera coordinate system that determines the shooting coordinates and the projector coordinate system that determines the projection coordinates and the coordinates of the projector; generating second information based on the coordinates of a second specific point in the object in the camera coordinate system and the coordinates of a third specific point corresponding to the second specific point in a three-dimensional imaginary space, wherein the second information represents the correspondence between the camera coordinate system and the imaginary space coordinate system that represents the coordinates in the imaginary space; and determining the position of the projector in the imaginary space coordinate system based on the coordinates of the projector by using the first information and the second information.

[0008] One embodiment of the position determination system of the present invention includes: a generation unit that generates relational information based on a three-dimensional first coordinate of a portion of the object containing the specific point and a two-dimensional second coordinate of the specific point in the projected image, wherein the relational information represents the correspondence between a three-dimensional coordinate system used by a measuring instrument to determine the first coordinate and a projector coordinate system for determining the second coordinate and the coordinates of the projector; and a position determination unit that determines the position of the projector in the three-dimensional coordinate system based on the coordinates of the projector using the relational information.

[0009] One embodiment of the simulation system of the present invention includes: a first generation unit that generates first relational information based on a three-dimensional first coordinate of a portion of the object containing the first specific point and a two-dimensional second coordinate of the first specific point in the projected image, wherein the first relational information represents the correspondence between a three-dimensional coordinate system used by a measuring instrument to determine the first coordinate and a projector coordinate system that determines the second coordinate and the coordinates of the projector; a second generation unit that generates second relational information based on the coordinates of the second specific point in the object in the three-dimensional coordinate system and the coordinates of a third specific point corresponding to the second specific point in a three-dimensional imaginary space, wherein the second relational information represents the correspondence between the three-dimensional coordinate system and an imaginary space coordinate system that determines the coordinates in the imaginary space; and a position determination unit that determines the position of the projector in the imaginary space coordinate system based on the coordinates of the projector using the first relational information and the second relational information.

[0010] One embodiment of the simulation system of the present invention includes: a first generation unit that generates first information based on shooting coordinates and projection coordinates, wherein the shooting coordinates are two-dimensional coordinates of the portion of the object where the first specific point is located in a shooting image generated by a camera shooting the object when a projector projects a projection image having a first specific point onto the object, and the projection coordinates are two-dimensional coordinates of the first specific point in the projection image, and the first information represents the correspondence between a camera coordinate system that determines the shooting coordinates and a projector coordinate system that determines the projection coordinates and the coordinates of the projector; a second generation unit that generates second information based on the coordinates of a second specific point in the object in the camera coordinate system and the coordinates of a third specific point corresponding to the second specific point in a three-dimensional imaginary space, wherein the second information represents the correspondence between the camera coordinate system and an imaginary space coordinate system that represents the coordinates in the imaginary space; and a position determination unit that determines the position of the projector in the imaginary space coordinate system based on the coordinates of the projector by using the first information and the second information. Attached Figure Description

[0011] Figure 1 This is a diagram showing the projection system 1 of the first embodiment.

[0012] Figure 2 This is a diagram showing an example of a deformed image G1.

[0013] Figure 3 This is a diagram showing the original image G2, which serves as the basis for the deformed image G1.

[0014] Figure 4 This is a diagram showing an example of the deformation of the original image G2.

[0015] Figure 5 This is a diagram showing a projection example of the deformed image G1.

[0016] Figure 6 This is a diagram illustrating an example of a simulation of an image in imaginary space d1.

[0017] Figure 7 This is a diagram showing an example of a projector 100.

[0018] Figure 8 This is a diagram showing an example of the projector coordinate system CS1.

[0019] Figure 9 This is a diagram showing an example of the measured image G3.

[0020] Figure 10 This is a diagram showing an example of a measuring instrument 200.

[0021] Figure 11This is a diagram showing an example of the measuring instrument's coordinate system CS2.

[0022] Figure 12 This is a diagram showing an example of an information processing device 300.

[0023] Figure 13 This is a diagram illustrating an example of the imaginary space d1.

[0024] Figure 14 This is a diagram showing an example of the corresponding point f.

[0025] Figure 15 This is a diagram showing one example of multiple measurement images G3.

[0026] Figure 16 It shows pixel 130p and phase A diagram showing the relationships between them.

[0027] Figure 17 This is a diagram used to illustrate the operation of projection system 1.

[0028] Figure 18 This is a diagram used to illustrate vectors v1 and v2.

[0029] Figure 19 This is another example of a measurement image G3.

[0030] Label Explanation

[0031] 1: Projection system; 2: Object; 100: Projector; 110: Image processing unit; 120: Light source; 130: Liquid crystal light valve; 140: Projection lens; 200: Measuring device; 210: First camera; 211: First capturing lens; 212: First image sensor; 220: Second camera; 221: Second capturing lens; 222: Second image sensor; 230: First storage unit; 240: First processing unit; 241: Capture control unit. 341: Control unit; 242: Supply unit; 243: Calculation unit; 300: Information processing device; 310: Operation unit; 320: Display unit; 330: Second storage unit; 340: Second processing unit; 341: Imaginary space generation unit; 342: Projection control unit; 343: Measurement control unit; 344: Acquisition unit; 345: First generation unit; 346: Second generation unit; 347: Position determination unit; 348: Imaginary space control unit; 349: Image generation unit. Detailed Implementation

[0032] A: Implementation Method 1

[0033] A1: Overview of Projection System 1

[0034] Figure 1This is a diagram illustrating the projection system 1 of the first embodiment. The projection system 1 is an example of both a position determination system and a simulation system.

[0035] Projection system 1 projects an image onto object 2. Object 2 is a wall of a room. The wall of the room is an example of an object being displayed. The object being displayed is not limited to the wall of a room; for example, it could be the exterior wall of a building or a product. Object 2 is not limited to the object being displayed; for example, it could be an object under manufacturing or an object under inspection. An object under manufacturing could be, for example, a car under manufacturing, a train under manufacturing, an airplane under manufacturing, an electronic product under manufacturing, or a building under construction. An object under inspection could be, for example, a car under inspection, a train under inspection, an airplane under inspection, an electronic product under inspection, or a building under inspection. The form of object 2 is not limited to... Figure 1 The form shown can be changed appropriately.

[0036] Object 2 has three object points, specifically object point k1, object point k2, and object point k3. When it is not necessary to distinguish between object points k1 and k3, each of them is referred to as "object point k". Object point k is an example of a second specific point. Object 2 may also have four or more object points k.

[0037] The projection system 1 includes a projector 100, a measuring device 200, and an information processing device 300.

[0038] Projector 100 projects a deformed image G1 onto object 2. Figure 2 This is a diagram illustrating an example of a deformed image G1. Deformed image G1 is an image deformed according to the shape of object 2. Figure 3 This is a diagram showing the original image G2 as the basis for the deformed image G1. The original image G2 is an example of the first image. The deformed image G1 is an example of the second image.

[0039] When the original image G2 is projected from the projector 100 onto the object 2, such as Figure 4 As shown, the original image G2 projected onto object 2 is deformed according to the shape of object 2. The deformation of the original image G2 corresponding to the shape of object 2 is caused by the following: the longer the distance from the projector 100 to the projection destination of the image, the larger the image projected from the projector 100. Hereinafter, the deformation produced in the original image G2 projected onto object 2 will be referred to as the "first deformation".

[0040] By causing the original image G2 to produce a second deformation that is canceled out by the first deformation, the deformed image G1 is obtained.

[0041] When a distorted image G1 is projected from the projector 100 onto the object 2, the distorted image G1 projected onto the object 2 undergoes a first deformation. This first deformation cancels out a second deformation in the distorted image G1. Therefore, as... Figure 5 As shown, the deformed image G1 is displayed on object 2 in the same form as the original image G2.

[0042] In the industrial field, when it is desired to show the manufacturing method or inspection method of object 2 on object 2 in a way that can be confirmed using characters or the like, a deformed image G1 can be generated from the original image G2 that shows the manufacturing method or inspection method of object 2 in a way that can be confirmed using characters or the like.

[0043] The measuring device 200 performs a three-dimensional measurement on the object 2. The information processing device 300 generates a deformed image G1 by using the projector 100 and the measuring device 200.

[0044] like Figure 6 As shown, the information processing device 300 simulates the image in the imaginary space d1, thereby generating a deformed image G1.

[0045] First, the information processing device 300 arranges an imaginary object 2v with the same shape as the object 2 in the imaginary space d1.

[0046] Next, the information processing device 300 displays the original image G2 on the imaginary object 2v by pasting an imaginary piece of paper representing the original image G2 onto the imaginary object 2v.

[0047] Next, the information processing device 300 configures a virtual camera 400v in the virtual space d1.

[0048] Specifically, the information processing device 300 configures the virtual camera 400v in such a way that the positional relationship between the virtual camera 400v and the virtual object 2v is consistent with the positional relationship between the projector 100 and the object 2.

[0049] The hypothetical camera 400v has a hypothetical shooting lens 410v, which has the same internal parameters as the projection lens 140 of the projector 100.

[0050] Next, the information processing device 300 generates a hypothetical image G1 obtained by the hypothetical camera 400v capturing the original image G2 displayed on the hypothetical object 2v. Figure 6 The image shows a hypothetical camera with a voltage of 400V and a field of view of 420V.

[0051] When generating the deformed image G1 using this method, it is necessary to determine the position of the virtual camera 400v in the virtual space d1. Even if the projector 100 and the measuring instrument 200 are arranged in arbitrary positions, the information processing device 300 uses the projector 100 and the measuring instrument 200 to determine the position of the virtual camera 400v in the virtual space d1.

[0052] The following section will focus on the methods for determining the position of the imaginary camera 400v in the imaginary space d1 and the method for generating the deformed image G1, while also explaining the structure of the projection system 1.

[0053] A2: Projector 100

[0054] Figure 7 This is a diagram showing an example of a projector 100. The projector 100 includes an image processing unit 110, a light source 120, a liquid crystal light valve 130, and a projection lens 140.

[0055] The image processing unit 110 is configured with, for example, an image processing circuit or other circuitry. The image processing unit 110 receives image data a from the information processing device 300. The image processing unit 110 performs image processing such as gamma correction on the image data a, thereby generating a voltage b based on the image data a.

[0056] The light source 120 is an LED (Light Emitting Diode). The light source 120 is not limited to LEDs; for example, it could also be a xenon lamp or a laser light source.

[0057] The liquid crystal light valve 130 is composed of a liquid crystal panel or the like, which has liquid crystal between a pair of transparent substrates. The liquid crystal light valve 130 has a rectangular pixel area 130a. The pixel area 130a includes a plurality of pixels 130p arranged in a matrix.

[0058] In the liquid crystal light valve 130, a voltage b based on image data a is applied to the liquid crystal per pixel 130p. When the voltage b based on image data a is applied to the liquid crystal per pixel 130p, the pixel 130p is set to a transmittance based on image data a.

[0059] The light emitted from the light source 120 is modulated by the pixel area 130a of the liquid crystal light valve 130. The liquid crystal light valve 130 is an example of a light modulation device. The light modulated by the liquid crystal light valve 130 is directed toward the projection lens 140. The projection lens 140 projects the light modulated by the liquid crystal light valve 130, i.e., the image, onto the object 2.

[0060] Apply the projector coordinate system CS1 to the liquid crystal light valve 130. Figure 8 This diagram illustrates an example of a projector coordinate system CS1. The projector coordinate system CS1 is a two-dimensional coordinate system. The origin o1 of the projector coordinate system CS1 is set at... Figure 8 The top left corner 130c of the pixel region 130a shown. Figure 8 For convenience, the origin o1 is shown in a position different from the top left corner 130c.

[0061] The position of the principal point of the projection lens 140 is determined by the coordinates in the projector coordinate system CS1. The coordinates of the principal point of the projection lens 140 are an example of the coordinates of the projector 100. The coordinates of the projector 100 are not limited to the coordinates of the principal point of the projection lens 140, as long as they are coordinates in the projector coordinate system CS1.

[0062] The projector coordinate system CS1 is defined by the x1 and y1 axes. The x1 and y1 axes are determined by the orientation of the liquid crystal light valve 130. The x1 axis is parallel to the horizontal direction of the liquid crystal light valve 130, i.e., it is parallel to the transverse direction of the liquid crystal light valve 130. The y1 axis is orthogonal to the x1 axis. The y1 axis is parallel to the vertical direction of the liquid crystal light valve 130, i.e., it is parallel to the longitudinal direction of the liquid crystal light valve 130.

[0063] Besides the x1 axis and y1 axis, Figure 8 The z1 axis is also shown. The z1 axis is orthogonal to the x1 axis and the y1 axis, respectively. The z1 axis is along the optical axis of the projection lens 140.

[0064] In addition to the deformed image G1, the projector 100 also projects the measured image G3 onto the object 2.

[0065] Figure 9 This is a diagram illustrating an example of a measurement image G3. Measurement image G3 is used to determine the correspondence between the projector coordinate system CS1 and the measuring coordinate system CS2 of the measuring instrument 200. In other words, measurement image G3 is used to determine the point in the measuring coordinate system CS2 that corresponds to a point in the projector coordinate system CS1.

[0066] Measurement image G3 has a first measurement point e1, a second measurement point e2, and a third measurement point e3. Where it is not necessary to distinguish between the first measurement point e1 to the third measurement point e3, they are each referred to as "measurement point e". Measurement point e is a part of measurement image G3. Measurement point e is an example of a specific point and an example of the first specific point. Measurement image G3 is an example of a projected image with a specific point and an example of a projected image with the first specific point. Measurement image G3 may also have more than four measurement points e.

[0067] When using multiple measurement images G3, in each of the multiple measurement images G3, a measurement point e may also exist in a form that is not distinguishable from other parts of the measurement image G3. An example of multiple measurement images G3 is the multiple phase-shifted images used in the phase-shifting method. Phase-shifted images are described later.

[0068] A3: Measuring Instrument 200

[0069] Figure 10 This diagram illustrates an example of the measuring device 200. The measuring device 200 is a stereo camera. The measuring device 200 is not limited to a stereo camera; it can be any device that performs three-dimensional measurement on the object 2. The measuring device 200 is a separate structure from the projector 100. Alternatively, the measuring device 200 may not be separate from the projector 100 and may be integrated into the projector 100.

[0070] The measuring device 200 includes a first camera 210, a second camera 220, a first storage unit 230, and a first processing unit 240. The positions of the first camera 210 and the second camera 220 are different from each other.

[0071] The first camera 210 includes a first shooting lens 211 and a first image sensor 212.

[0072] The first imaging lens 211 images the optical image of the object 2 onto the first image sensor 212. For example, when the projector 100 projects a measurement image G3 onto the object 2, the first imaging lens 211 images the optical image of the object 2 onto the first image sensor 212.

[0073] The first image sensor 212 is a CCD (Charge Coupled Device) image sensor. The first image sensor 212 is not limited to a CCD image sensor; for example, it could also be a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The first image sensor 212 has a rectangular first imaging area 212a. The first imaging area 212a contains a plurality of cells 212p arranged in a matrix. The first image sensor 212 generates first imaging data c1 based on the optical image imaged by the first imaging lens 211.

[0074] The second camera 220 includes a second shooting lens 221 and a second image sensor 222.

[0075] The second imaging lens 221 images the optical image of the object 2 onto the second image sensor 222. For example, when the projector 100 projects a measurement image G3 onto the object 2, the second imaging lens 221 images the optical image of the object 2 onto the second image sensor 222.

[0076] The second image sensor 222 is a CCD image sensor. The second image sensor 222 is not limited to a CCD image sensor; for example, it could also be a CMOS image sensor. The second image sensor 222 has a rectangular second imaging area 222a. The second imaging area 222a contains a plurality of cells 222p arranged in a matrix. The second image sensor 222 generates second imaging data c2 based on the optical image imaged by the second imaging lens 221.

[0077] The first storage unit 230 is a recording medium readable by the first processing unit 240. The first storage unit 230 includes, for example, non-volatile memory and volatile memory. Non-volatile memory includes, for example, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory). Volatile memory includes, for example, RAM (Random Access Memory). The first storage unit 230 stores a first program P1 executed by the first processing unit 240.

[0078] The first processing unit 240 consists of one or more CPUs (Central Processing Units). One or more CPUs are an example of one or more processors. Processors and CPUs are examples of computers.

[0079] The first processing unit 240 reads the first program P1 from the first storage unit 230. By executing the first program P1, the first processing unit 240 functions as the shooting control unit 241, the supply unit 242, and the calculation unit 243.

[0080] The shooting control unit 241, the supply unit 242, and the computing unit 243 can also be composed of circuits such as DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), and FPGA (Field Programmable Gate Array).

[0081] The shooting control unit 241 controls the shooting performed by the first camera 210 and the shooting performed by the second camera 220.

[0082] The providing unit 242 outputs various information, such as the first captured data c1 and the second captured data c2, to the information processing device 300.

[0083] The calculation unit 243 performs a three-dimensional measurement on the object 2 based on the first image data c1 and the second image data c2. The calculation unit 243 represents the result of the three-dimensional measurement using coordinates in the measuring instrument coordinate system CS2.

[0084] Figure 11 This diagram illustrates an example of the measuring instrument coordinate system CS2. The measuring instrument coordinate system CS2 is a three-dimensional coordinate system. The origin o2 of the measuring instrument coordinate system CS2 is set at the principal point of the first imaging lens 211. The position of the principal point of the first imaging lens 211 is an example of the position of the measuring instrument 200. Figure 11 For convenience, the origin o2 is shown at a position different from the principal point of the first camera lens 211. The origin o2 of the measuring instrument coordinate system CS2 is not limited to the position of the principal point of the first camera lens 211; for example, it could also be the position of the principal point of the second camera lens 221. The position of the principal point of the second camera lens 221 is another example of the position of the measuring instrument 200.

[0085] The measuring instrument coordinate system CS2 is an example of a three-dimensional coordinate system. The measuring instrument coordinate system CS2 is defined by the x2-axis, y2-axis, and z2-axis. The x2-axis, y2-axis, and z2-axis are determined by the orientation of the measuring instrument 200. The x2-axis, y2-axis, and z2-axis are mutually orthogonal.

[0086] The x2 axis is parallel to the horizontal direction of the first image sensor 212, that is, parallel to the lateral direction of the first image sensor 212. When the origin o2 of the measuring coordinate system CS2 is at the same position as the principal point of the second imaging lens 221, the x2 axis is parallel to the horizontal direction of the second image sensor 222, that is, parallel to the lateral direction of the second image sensor 222.

[0087] The y2 axis is parallel to the vertical direction of the first image sensor 212, that is, parallel to the longitudinal direction of the first image sensor 212. When the origin o2 of the measuring instrument coordinate system CS2 is at the same position as the principal point of the second imaging lens 221, the y2 axis is parallel to the vertical direction of the second image sensor 222, that is, parallel to the longitudinal direction of the second image sensor 222.

[0088] The z2 axis is aligned with the optical axis of the first imaging lens 211. When the origin o2 of the measuring instrument coordinate system CS2 is at the same position as the principal point of the second imaging lens 221, the z2 axis is aligned with the optical axis of the second imaging lens 221.

[0089] A4: Information Processing Device 300

[0090] Figure 12This is a diagram illustrating an example of an information processing device 300. The information processing device 300 is a PC (Personal Computer). The information processing device 300 is not limited to a PC; for example, it could also be a tablet computer or a smartphone.

[0091] The information processing device 300 includes an operation unit 310, a display unit 320, a second storage unit 330, and a second processing unit 340.

[0092] The operation unit 310 may be, for example, a keyboard, mouse, operation buttons, operation keys, or a touch panel. The operation unit 310 accepts user input operations.

[0093] Display unit 320 is a display such as a liquid crystal display, plasma display, or organic EL (ElectroLuminescence) display, etc., and is a flat panel display (FPD). Display unit 320 displays various information.

[0094] The second storage unit 330 is a recording medium readable by the second processing unit 340. The second storage unit 330 includes, for example, non-volatile memory and volatile memory. The second storage unit 330 stores a second program P2 executed by the second processing unit 340.

[0095] The second processing unit 340 is composed of one or more CPUs, for example. The second processing unit 340 reads the second program P2 from the second storage unit 330. By executing the second program P2, the second processing unit 340 functions as the imaginary space generation unit 341, the projection control unit 342, the measurement control unit 343, the acquisition unit 344, the first generation unit 345, the second generation unit 346, the position determination unit 347, the imaginary space control unit 348, and the image generation unit 349.

[0096] The imaginary space generation unit 341, projection control unit 342, measurement control unit 343, acquisition unit 344, first generation unit 345, second generation unit 346, position determination unit 347, imaginary space control unit 348, and image generation unit 349 can also be composed of circuits such as DSP, ASIC, PLD, and FPGA.

[0097] The imaginary space generation unit 341 generates a three-dimensional imaginary space d1. Figure 13 This is a diagram illustrating an example of an imaginary space d1. An imaginary space coordinate system CS3 is applied to imaginary space d1. The imaginary space coordinate system CS3 is a three-dimensional coordinate system. The imaginary space coordinate system CS3 is defined by the x3 axis, y3 axis, and z3 axis. The x3 axis, y3 axis, and z3 axis are mutually orthogonal.

[0098] Hypothetical object 2v is located in hypothetical space d1. Hypothetical object 2v corresponds to object 2. The shape of hypothetical object 2v is the same as the shape of object 2.

[0099] The hypothetical object 2v has three hypothetical object points, specifically the first hypothetical object point q1, the second hypothetical object point q2, and the third hypothetical object point q3.

[0100] Without needing to distinguish between the first hypothetical object point q1 and the third hypothetical object point q3, the first hypothetical object point q1 and the third hypothetical object point q3 are respectively referred to as "hypothetical object point q". Hypothetical object point q is an example of the third specific point corresponding to the second specific point.

[0101] The positional relationship between the hypothetical object 2v and the first hypothetical object point q1 is the same as the positional relationship between object 2 and the first object point k1. For example, if the position of the first hypothetical object point q1 in hypothetical object 2v is predetermined, the position of the first object point k1 in object 2 is determined in the same way that the positional relationship between object 2 and the first object point k1 is the same as the positional relationship between hypothetical object 2v and the first hypothetical object point q1.

[0102] In this case, for example, a first physical marker is positioned at the location of a first object point k1 in object 2. The first physical marker has a reflectivity different from that of object 2. Therefore, the first physical marker can be identified in each of the first image data c1 generated by the first camera 210 and the second image data c2 generated by the second camera 220. Therefore, the position of the first physical marker, i.e., the position of the first object point k1, can be determined based on the first image data c1 and the second image data c2. A first hypothetical object point q1 corresponds to the first object point k1.

[0103] The positional relationship between the hypothetical object 2v and the second hypothetical object point q2 is the same as the positional relationship between object 2 and the second hypothetical object point k2. For example, if the position of the second hypothetical object point q2 in the hypothetical object 2v is predetermined, the position of the second object point k2 in object 2 is determined in the same way that the positional relationship between object 2 and the second object point k2 is the same as the positional relationship between the hypothetical object 2v and the second hypothetical object point q2.

[0104] In this case, for example, a second physical marker is positioned at the location of the second object point k2 in object 2. The second physical marker has a reflectivity different from that of object 2 and the first physical marker. Therefore, the second physical marker can be identified in each of the first image data c1 generated by the first camera 210 and the second image data c2 generated by the second camera 220. Therefore, the position of the second physical marker, i.e., the position of the second object point k2, can be determined based on the first image data c1 and the second image data c2. The second hypothetical object point q2 corresponds to the second object point k2.

[0105] The positional relationship between the hypothetical object 2v and the third hypothetical object point q3 is the same as the positional relationship between object 2 and the third hypothetical object point k3. For example, if the position of the third hypothetical object point q3 in hypothetical object 2v is predetermined, the position of the third object point k3 in object 2 is determined in the same way that the positional relationship between object 2 and the third hypothetical object point k3 is the same as the positional relationship between hypothetical object 2v and the third hypothetical object point q3.

[0106] In this case, for example, a third physical marker is positioned at the location of the third object point k3 in object 2. The third physical marker has a reflectivity different from that of object 2, the first physical marker, and the second physical marker. Therefore, the third physical marker can be identified in each of the first image data c1 generated by the first camera 210 and the second image data c2 generated by the second camera 220. Therefore, the position of the third physical marker, i.e., the position of the third object point k3, can be determined based on the first image data c1 and the second image data c2. The third hypothetical object point q3 corresponds to the third object point k3.

[0107] Return to description Figure 12 The projection control unit 342 controls the projector 100. The projection control unit 342 causes the projector 100 to project the measured image G3 and the distorted image G1 respectively. The projection control unit 342 causes the projector 100 to project the measured image G3 by providing the projector 100 with measured image data representing the measured image G3. The measured image data is an example of image data a. The projection control unit 342 causes the projector 100 to project the distorted image G1 by providing the projector 100 with distorted image data representing the distorted image G1. The distorted image data is another example of image data a.

[0108] The measurement and control unit 343 controls the measuring device 200. The measurement and control unit 343 causes the measuring device 200 to take a picture of the object 2. The measurement and control unit 343 causes the measuring device 200 to perform three-dimensional measurement on the object 2.

[0109] The acquisition unit 344 acquires from the measuring unit 200 the three-dimensional coordinates of the portion of the object 2 where the measuring point e of the measuring image G3 is located in the measuring image G3 when the projector 100 projects the measuring image G3 onto the object 2.

[0110] The part where the measurement point e of the measurement image G3 in object 2 is located is called the "corresponding point f". Figure 14 This is a diagram showing an example of the corresponding point f. Figure 14Three corresponding points f are shown, specifically the first corresponding point f1, the second corresponding point f2, and the third corresponding point f3. Each corresponding point f is the point that corresponds to the measurement point e in the measurement image G3. The first corresponding point f1 corresponds to the first measurement point e1. The second corresponding point f2 corresponds to the second measurement point e2. The third corresponding point f3 corresponds to the third measurement point e3.

[0111] The three-dimensional coordinates of the corresponding point f are called "coordinate h". Coordinate h is the coordinate of the corresponding point f in the measuring instrument coordinate system CS2. Coordinate h is measured by measuring instrument 200. Coordinate h is an example of the first coordinate.

[0112] The two-dimensional coordinates of the measured point e in the measured image G3, that is, the coordinates of the measured point e in the projector coordinate system CS1, are called "coordinate i". Coordinate i is stored in advance in the second storage unit 330. Coordinate i is an example of the second coordinate.

[0113] Return to description Figure 12 The first generation unit 345 generates first relationship information j1, representing the correspondence between the measuring instrument coordinate system CS2 and the projector coordinate system CS1. The first generation unit 345 generates the first relationship information j1 ​​based on the three-dimensional coordinates h of the corresponding point f in the measuring instrument coordinate system CS2 and the two-dimensional coordinates i of the measuring point e in the projector coordinate system CS1.

[0114] The second generation unit 346 generates second relationship information j2, which represents the correspondence between the measuring instrument coordinate system CS2 and the imaginary space coordinate system CS3.

[0115] The coordinates of object point k in the measuring instrument coordinate system CS2 of object 2 are called "coordinate m".

[0116] The coordinates of the imaginary point q of the imaginary object 2v in the imaginary space coordinate system CS3 are called "coordinate n". The second generation unit 346 generates second relationship information j2, which represents the correspondence between the measuring instrument coordinate system CS2 and the imaginary space coordinate system CS3, based on the coordinates m and n.

[0117] The position determination unit 347 determines the position of the projector 100 in the imaginary space coordinate system CS3 by using the first relation information j1 ​​and the second relation information j2, based on the coordinates of the projector 100 in the projector coordinate system CS1. The position of the projector 100 in the imaginary space coordinate system CS3 is referred to as the "first imaginary position r1".

[0118] The imaginary space control unit 348 controls the imaginary space d1. The imaginary space control unit 348 sets a second imaginary position r2 in the imaginary space d1. For example, the imaginary space control unit 348 sets the second imaginary position r2 in the imaginary space d1 according to a configuration instruction received from the user by the operation unit 310. The second imaginary position r2 in the imaginary space d1 corresponds to the position of the object 2 in the measuring coordinate system CS2. The imaginary space control unit 348 positions the imaginary object 2v at the second imaginary position r2. The imaginary space control unit 348 displays the original image G2 on the imaginary object 2v. Specifically, the imaginary space control unit 348 displays the original image G2 on the imaginary object 2v by attaching an imaginary piece of paper representing the original image G2 to the imaginary object 2v.

[0119] In the imaginary space control unit 348, a virtual camera 400v is positioned at a first imaginary position r1 in the imaginary space d1, which is the position of the projector 100 in the imaginary space coordinate system CS3. The imaginary space control unit 348 positions the principal point of the imaginary shooting lens 410v of the virtual camera 400v at the first imaginary position r1.

[0120] The internal parameters of the hypothetical imaging lens 410v are equal to the internal parameters of the projection lens 140 of the projector 100. The internal parameters of the hypothetical imaging lens 410v represent characteristics of the hypothetical imaging lens 410v, such as its focal length. The internal parameters of the projection lens 140 represent characteristics of the projection lens 140, such as its focal length. In this embodiment, the internal parameters of the projection lens 140 are known. The internal parameters of the projection lens 140 are, for example, stored in the second storage unit 330.

[0121] The image generation unit 349 generates a hypothetical captured image, G1, obtained by the hypothetical camera 400v capturing the original image G2 displayed on the hypothetical object 2v. The image generation unit 349, for example, generates image data representing the hypothetical captured image as hypothetical image data representing the hypothetical image G1.

[0122] A5: Measurement Image G3

[0123] Figure 15 This is a diagram illustrating an example of multiple measurement images G3. The multiple measurement images G3 include the first phase-shift image G31 to the fourth phase-shift image G34.

[0124] When it is not necessary to distinguish between the first phase-shifted image G31 to the fourth phase-shifted image G34, the first phase-shifted image G31 to the fourth phase-shifted image G34 are referred to as "phase-shifted image G30" respectively.

[0125] The phase-shifted image G30 is a patterned image showing the brightness variation corresponding to a sine wave along the x1 axis of the projector coordinate system CS1. The sine wave is a concept that includes cosine waves. The phase-shifted image G30 is used in the phase-shifting method.

[0126] The phase of the sine wave in the second phase-shifted image G32 leads the phase of the sine wave in the first phase-shifted image G31 by π / 2. The phase of the sine wave in the third phase-shifted image G33 leads the phase of the sine wave in the first phase-shifted image G31 by π. The phase of the sine wave in the fourth phase-shifted image G34 leads the phase of the sine wave in the first phase-shifted image G31 by 3π / 2.

[0127] The phase-shifted image G30 is used to determine the corresponding point f. The corresponding point f is the part of the object 2 where the measurement point e of the phase-shifted image G30 is located in the object 2 when the projector 100 projects the phase-shifted image G30 onto the object 2.

[0128] The phase-shifting method is used to determine the cell 212pt corresponding to the observation point f among the plurality of cells 212p of the first image sensor 212. Furthermore, the phase-shifting method is used to determine the cell 222pt corresponding to the observation point f among the plurality of cells 222p of the second image sensor 222. Cells 212pt and 222pt are determined using the phase of a sine wave in the phase-shifted image G30.

[0129] Figure 16 This shows the phase of the sine wave corresponding to the brightness of pixel 130p when the pixels 130p arranged in the direction along the x1 axis in the liquid crystal light valve 130 are projected by the projector 100 to display the first phase-shifted image G31. A diagram showing the relationships between them.

[0130] like Figure 16 As shown, along the x1 axis, the pixels 130p and phase of the liquid crystal light valve 130 There is a one-to-one correspondence. Therefore, the phase corresponding to the pixel 130pe located at the measurement point e among the multiple pixels 130p of the liquid crystal light valve 130 is uniquely determined. exist Figure 16 In order to simplify the explanation, only pixel 130p located at the first measurement point e1 is shown as pixel 130pe.

[0131] Given the phase of the sine wave corresponding to the brightness observed by any cell 212pi of the first image sensor 212 under the condition of projecting the first phase-shifted image G31, Determined by Equation 1.

[0132]

[0133] I1 is the brightness observed by cell 212pi under the condition of projecting the first phase-shifted image G31.

[0134] I2 is the brightness observed by cell 212pi under the condition of projecting the second phase-shifted image G32.

[0135] I3 is the brightness observed by cell 212pi under the condition of projecting the third phase-shifted image G33.

[0136] I4 is the brightness observed by cell 212pi under the condition of projecting the fourth phase-shifted image G34.

[0137] The cell 212pt determined by the phase-shifting method is the cell corresponding to the observation point f among the multiple cells 212p of the first image sensor 212. Therefore, the phase corresponding to the brightness observed by cell 212pt is... Shows the phase corresponding to the brightness of pixel 130pe. The same value. Therefore, searching cell 212pt means searching for observations and phases from multiple cells 212p. The corresponding brightness unit, phase Showing the phase corresponding to pixel 130pe Same value.

[0138] Therefore, it is possible to determine the coordinates x of the point f corresponding to the x1 axis coordinate of the projector coordinate system CS1 from multiple elements 212p. 1f Corresponding coordinates x 2f The coordinate unit group is 212px.

[0139] Furthermore, although the illustrations are omitted, the multiple measurement images G3 also include phase-shift images G35 through G38. Phase-shift images G35 through G38 are patterned images showing the brightness variations corresponding to the sine wave along the y1 axis of the projector coordinate system CS1. The phase of the sine wave in phase-shift image G36 leads the phase of the sine wave in phase-shift image G35 by π / 2. The phase of the sine wave in phase-shift image G37 leads the phase of the sine wave in phase-shift image G35 by π. The phase of the sine wave in phase-shift image G38 leads the phase of the sine wave in phase-shift image G35 by 3π / 2.

[0140] By capturing the 5th phase-shift image G35 to the 8th phase-shift image G38 using the first camera 210, similar to the case using the 1st phase-shift image G31 to the 4th phase-shift image G34, it is possible to determine from the multiple cells 212p the coordinates of the point f corresponding to the y1 axis coordinates of the projector coordinate system CS1. 1f Corresponding coordinates y 2fThe coordinate unit group 212py.

[0141] Therefore, it is possible to determine the coordinate x from multiple elements 212p. 2f and coordinates y 2f Unit 212p is used as unit 212pt.

[0142] The same method can be used to determine element 222pt.

[0143] A6: Description of the actions

[0144] Figure 17 This diagram illustrates the operation of the projection system 1. Additionally, the imaginary space generation unit 341 generates an imaginary space d1 in advance. The imaginary space control unit 348 sets a second imaginary position r2 in the imaginary space d1 in advance. The imaginary object 2v is then positioned at the second imaginary position r2 in advance by the imaginary space control unit 348.

[0145] The user operates the operation unit 310 and inputs a start instruction into it. After receiving the start instruction, the operation unit 310 provides the start instruction to the second processing unit 340.

[0146] After the second processing unit 340 receives the start instruction, in step S101, the projection control unit 342 sets the first phase-shifted image G31 to the eighth phase-shifted image G38 as unprojected images.

[0147] Next, in step S102, the projection control unit 342 selects an unprojected image from the first phase-shifted image G31 to the eighth phase-shifted image G38. The projection control unit 342 selects the unprojected image in the order of the first phase-shifted image G31, the second phase-shifted image G32, the third phase-shifted image G33, the fourth phase-shifted image G34, the fifth phase-shifted image G35, the sixth phase-shifted image G36, the seventh phase-shifted image G37, and the eighth phase-shifted image G38. The order in which the unprojected image is selected is not limited to the order of the first phase-shifted image G31, the second phase-shifted image G32, the third phase-shifted image G33, the fourth phase-shifted image G34, the fifth phase-shifted image G35, the sixth phase-shifted image G36, the seventh phase-shifted image G37, and the eighth phase-shifted image G38, and can be appropriately changed.

[0148] Next, the projection control unit 342 provides image data a representing the image selected in step S102 to the projector 100. If image data a is stored in the second storage unit 330, the projection control unit 342 reads image data a from the second storage unit 330. The projection control unit 342 provides the image data a read from the second storage unit 330 to the projector 100. Alternatively, the projection control unit 342 may generate image data a according to the second program P2. In this case, the projection control unit 342 provides the generated image data a to the projector 100.

[0149] Next, the projection control unit 342 changes the settings related to the image selected in step S102 from the unprojected image to the projected image.

[0150] Next, in step S103, the projector 100 projects the image represented by the image data a provided by the projection control unit 342 onto the object 2.

[0151] Next, in step S104, the measurement and control unit 343 causes the first camera 210 and the second camera 220 to capture images of the object 2 to which the projected image is projected.

[0152] For example, the measurement control unit 343 provides a shooting instruction, instructing the first camera 210 and the second camera 220 to capture images, to the shooting control unit 241 of the measuring device 200. Based on the shooting instruction, the shooting control unit 241 causes the first camera 210 and the second camera 220 to capture images of the object 2 to be projected. The first camera 210 captures images of the object 2 to be projected, thereby generating first captured image data representing the object 2. This first captured image data is an example of first captured data c1. The second camera 220 captures images of the object 2 to be projected, thereby generating second captured image data representing the object 2. This second captured image data is an example of second captured data c2. The providing unit 242 provides the first captured image data and the second captured image data to the information processing device 300.

[0153] Next, in step S105, if the projection control unit 342 finds an image in the first phase-shifted images G31 to the eighth phase-shifted images G38 that is set to not be projected, the process returns to step S102. Therefore, the first phase-shifted images G31 to the eighth phase-shifted images G38 are projected individually onto the object 2 by the projector 100. Furthermore, the first captured image data and the second captured image data for each of the first phase-shifted images G31 to the eighth phase-shifted images G38 are provided to the information processing device 300.

[0154] In step S105, if there are no images set as unprojected in the first phase-shifted images G31 to the eighth phase-shifted images G38, in step S106, the acquisition unit 344 acquires the three-dimensional coordinates (i.e., coordinates h) of the corresponding point f in the measuring coordinate system CS2 for each measuring point e. The corresponding point f is the portion of the phase-shifted image G30 in the object 2 where the projector 100 projects the phase-shifted image G30 onto the object 2.

[0155] In step S106, firstly, the acquisition unit 344 determines unit 212pt and unit 222pt according to each measurement point e. Unit 212pt is the unit corresponding to the observation point f among the plurality of units 212p of the first image sensor 212. Unit 222pt is the unit corresponding to the observation point f among the plurality of units 222p of the second image sensor 222.

[0156] The acquisition unit 344 determines a unit 212pt for each measurement point e based on the coordinates i of the measurement point e in the projector coordinate system CS1 and the first captured image data of each of the first phase-shifted images G31 to G38. Additionally, the second storage unit 330 stores the coordinates i; therefore, the acquisition unit 344 acquires the coordinates i from the second storage unit 330. The acquisition unit 344 determines the unit 212pt for each measurement point e using the phase-shifting method.

[0157] The acquisition unit 344 determines a unit 222pt for each measurement point e based on the coordinates i of the measurement point e in the projector coordinate system CS1 and the second captured image data of each of the first phase-shifted images G31 to the eighth phase-shifted images G38. The acquisition unit 344 determines the unit 222pt for each measurement point e by using the phase-shifting method.

[0158] After determining unit 212pt and unit 222pt according to each measurement point e, the acquisition unit 344 uses the measurement control unit 343 to cause the measuring device 200 to perform three-dimensional measurement related to each corresponding point f.

[0159] For example, the acquisition unit 344 provides the measurement instructions of unit 212pt and unit 222pt from the measurement control unit 343 to the calculation unit 243 of the measuring instrument 200 according to each measurement point e.

[0160] The calculation unit 243 calculates the three-dimensional coordinates of the corresponding point f in the measuring coordinate system CS2 according to the position of the unit 212pt in the first image sensor 212 and the position of the unit 222pt in the second image sensor 222 for each measuring point e.

[0161] For example, the calculation unit 243 calculates the distance from the corresponding point f to the measuring device 200 based on the position of the unit 212pt and the position of the unit 222pt for each measuring point e, according to the principle of triangulation.

[0162] The calculation unit 243 uses the x2-axis-based coordinates of unit 212pt as the x2-axis-based coordinates of the corresponding point f in the three-dimensional coordinates of the measuring instrument coordinate system CS2. The calculation unit 243 uses the y2-axis-based coordinates of unit 212pt as the y2-axis-based coordinates of the corresponding point f in the three-dimensional coordinates of the measuring instrument coordinate system CS2. The calculation unit 243 uses the value based on the distance from the corresponding point f to the measuring instrument 200 as the z2-axis-based coordinates of the corresponding point f in the three-dimensional coordinates of the measuring instrument coordinate system CS2.

[0163] Next, the providing unit 242 provides the information processing device 300 with the three-dimensional coordinates of the corresponding point f in the measuring instrument coordinate system CS2 for each measuring point e.

[0164] The acquisition unit 344 obtains the three-dimensional coordinates of the corresponding point f in the measuring instrument coordinate system CS2 from the supply unit 242 for each measuring point e, as coordinate h.

[0165] Next, in step S107, the first generation unit 345 generates first relationship information j1, which represents the correspondence between the measuring instrument coordinate system CS2 and the projector coordinate system CS1.

[0166] For example, the first generation unit 345 generates the first relational information j1 ​​based on the three-dimensional coordinates h of the corresponding point f in the measuring instrument coordinate system CS2 and the two-dimensional coordinates i of the measuring point e in the projector coordinate system CS1.

[0167] The first generation unit 345 obtains the three-dimensional coordinates, i.e., coordinate h, from the acquisition unit 344. The two-dimensional coordinates, i.e., coordinate i, are stored in the second storage unit 330. Therefore, the first generation unit 345 obtains the two-dimensional coordinates, i.e., coordinate i, from the second storage unit 330.

[0168] The first generation unit 345 uses the "pair" of each measurement point e in three-dimensional coordinates (i.e., coordinate h) and two-dimensional coordinates (i.e., coordinate i) to solve the PnP (Perspective n Points) problem, thereby generating the first relational information j1. Hereinafter, the "pair" of coordinates h and coordinate i will be referred to as the "first coordinate pair".

[0169] For example, the first generation unit 345 substitutes the first coordinate pair into Equation 2 for each measurement point e, thereby solving the PnP problem. Equation 2 is also called the perspective projection transformation formula.

[0170]

[0171] in

[0172] It is the internal parameter matrix A of the projection lens 140.

[0173] It is the rotation matrix R.

[0174] It is the parallel shift matrix T.

[0175] (X, Y, Z) represents the three-dimensional coordinates of the measuring instrument coordinate system CS2, for example, coordinate h.

[0176] (u, v) represents the two-dimensional coordinates of the projector coordinate system CS1, such as coordinate i.

[0177] s is a scaling factor used to implement "1" in (u, v, 1). s = Z holds true.

[0178] (c x c y ) are the coordinates of the principal point of the projection lens 140.

[0179] f x and f y The focal length of the projection lens 140 is expressed using a value of 130p as one unit.

[0180] In other words, f x and f y The focal length of the projection lens 140 is expressed in pixels.

[0181] f x The focal length of the projection lens 140 is expressed in units of length along the x1 axis based on pixel 130p.

[0182] f y The focal length of the projection lens 140 is expressed in units of length along the y1 axis based on pixel 130p.

[0183] Equation 2 is equivalent to Equation 3.

[0184]

[0185] x'=x / z

[0186] y'=y / z

[0187] x”=x'(1+k1r 2 +k2r 4 +k3r 6 )+2p1x'y'+p2(r 2 +2x' 2 )

[0188] y”=y'(1+k1r 2 +k2r 4 +k3r 6 )+p1(r 2 +2y' 2 )+2p2x'y'

[0189] in,

[0190] r 2 =x' 2 +y' 2

[0191] u = f x *x”+c x

[0192] v = f y *y”+c y

[0193] k1, k2, and k3 are the distortion coefficients in the radial direction of the projection lens 140.

[0194] p1 and p2 are the distortion coefficients in the circumferential direction of the projection lens 140.

[0195] Each distortion coefficient belongs to the internal parameters of the projection lens 140.

[0196] The first generation unit 345 substitutes the first coordinate pair into Equation 2 for each measurement point e, thereby generating multiple equations. The first generation unit 345 determines the rotation matrix R, the parallel translation matrix T, and the internal parameter matrix A by solving the multiple equations.

[0197] In this embodiment, the internal parameters of the projection lens 140 are known. That is, the internal parameter matrix A is known. Therefore, the first generation unit 345 can determine the rotation matrix R and the parallel translation matrix T by solving multiple equations. In this case, the first generation unit 345 can determine the rotation matrix R and the parallel translation matrix T by using at least three first coordinate pairs.

[0198] If more than 6 first coordinate pairs are used, the rotation matrix R and the parallel translation matrix T can be determined as a solution in the first generation part 345.

[0199] The coordinates represented by the first coordinate pair may contain errors. Therefore, the more first coordinate pairs substituted into Equation 2, the higher the accuracy of the rotation matrix R and the parallel translation matrix T. Thus, it is preferable that the first generation unit 345 determines the rotation matrix R and the parallel translation matrix T by using more first coordinate pairs. The first generation unit 345 generates a group of the rotation matrix R, the parallel translation matrix T, the intrinsic parameter matrix A, and the scaling factor s as first relational information j1.

[0200] Next, in step S108, the second generation unit 346 generates second relationship information j2, which represents the correspondence between the measuring instrument coordinate system CS2 and the imaginary space coordinate system CS3.

[0201] The second generation unit 346 generates the second relation information j2 based on the coordinates (i.e., coordinate m) of the object point k of the object 2 in the measuring instrument coordinate system CS2 and the coordinates (i.e., coordinate n) of the imaginary object point q of the imaginary object 2v in the imaginary space coordinate system CS3.

[0202] The second generation unit 346 obtains the coordinates (i.e., coordinates m) of the object point k of the object 2 in the measuring instrument coordinate system CS2 from the measuring instrument 200.

[0203] For example, the second generation unit 346 uses the measurement control unit 343 to cause the measuring device 200 to perform three-dimensional measurement on each object point k in the object 2.

[0204] For example, the second generation unit 346 provides the three-dimensional measurement instructions related to each object point k from the measurement control unit 343 to the calculation unit 243 of the measuring unit 200.

[0205] The calculation unit 243 calculates the three-dimensional coordinates of each object point k in the measuring instrument coordinate system CS2 based on the instructions of the three-dimensional measurement related to each object point k.

[0206] The calculation unit 243 uses the x2-axis-based coordinates of the observed object point k in cell 212p of the first image sensor 212 as the x2-axis-based coordinates of object point k in the three-dimensional coordinates of the measuring instrument coordinate system CS2. The calculation unit 243 uses the y2-axis-based coordinates of the observed object point k in cell 212p of the first image sensor 212 as the y2-axis-based coordinates of object point k in the three-dimensional coordinates of the measuring instrument coordinate system CS2. The calculation unit 243 uses the principle of triangulation to calculate the distance from object point k to the measuring instrument 200. The calculation unit 243 uses the value based on the distance from object point k to the measuring instrument 200 as the z2-axis-based coordinates of object point k in the three-dimensional coordinates of the measuring instrument coordinate system CS2.

[0207] Next, the providing unit 242 provides the information processing device 300 with the three-dimensional coordinates of each object point k in the measuring coordinate system CS2.

[0208] The second generation unit 346 obtains the three-dimensional coordinates of the object point k in the measuring instrument coordinate system CS2 from the providing unit 242 as coordinate m for each object point k.

[0209] The virtual space control unit 348 manages the coordinates (i.e., coordinates n) of the virtual object point q in the virtual space coordinate system CS3 for each virtual object point q possessed by the virtual object 2v. Therefore, the second generation unit 346 obtains the coordinates (i.e., coordinates n) of the virtual object point q in the virtual space coordinate system CS3 from the virtual space control unit 348 according to each virtual object point q possessed by the virtual object 2v.

[0210] The second generation unit 346 generates second relational information j2 using each pair of object point k, where the coordinates m of object point k are the coordinates n of the imaginary object point q corresponding to object point k. Hereinafter, the pair of coordinates m and coordinates n is referred to as the "second coordinate pair".

[0211] The second generation unit 346 determines a rotation matrix R1 that aligns the orientation of the measuring coordinate system CS2 with that of the imaginary space coordinate system CS3, and a parallel translation matrix T1 that aligns the origin o2 of the measuring coordinate system CS2 with the origin o3 of the imaginary space coordinate system CS3. The rotation matrix R1 is a 3x3 matrix. The parallel translation matrix T1 is a 1x3 matrix.

[0212] Equation 4 shows the relationship between the measuring instrument coordinate system CS2, the imaginary space coordinate system CS3, the rotation matrix R1, and the parallel translation matrix T1.

[0213]

[0214] (X,Y,Z) represents the three-dimensional coordinates of the measuring instrument coordinate system CS2, for example, coordinate m.

[0215] (X V ,Y V Z V ) represents the three-dimensional coordinates of the imaginary spatial coordinate system CS3, for example, coordinate n.

[0216] The second generation unit 346 substitutes the second coordinate pair into Equation 4 for each object point k, thereby generating multiple equations. The second generation unit 346 determines the rotation matrix R1 and the parallel translation matrix T1 by solving these multiple equations. The second generation unit 346 generates a group of rotation matrix R1 and parallel translation matrix T1 as the second relational information j2.

[0217] Next, in step S109, the position determination unit 347 determines the position of the projector 100 in the imaginary space coordinate system CS3, i.e., the first imaginary position r1, by using the first relation information j1 ​​and the second relation information j2, based on the coordinates of the projector 100 in the projector coordinate system CS1.

[0218] First, the position determination unit 347 determines the position of the projector 100 in the measuring instrument coordinate system CS2 by using the first relationship information j1 ​​and based on the coordinates of the projector 100 in the projector coordinate system CS1.

[0219] For example, the position determination unit 347 converts the two-dimensional coordinates of the projector 100 in the projector coordinate system CS1 into three-dimensional coordinates of the projector 100 in the measuring instrument coordinate system CS2 by using Equation 2 with the first relational information j1 ​​applied. The position determination unit 347 determines the position represented by the three-dimensional coordinates of the projector 100 in the measuring instrument coordinate system CS2 as the position of the projector 100 in the measuring instrument coordinate system CS2.

[0220] Next, the position determination unit 347 determines the position of the projector 100 in the imaginary space coordinate system CS3 by using the second relational information j2 and the three-dimensional coordinates representing the position of the projector 100 in the measuring coordinate system CS2.

[0221] For example, the position determination unit 347 uses Equation 4, which applies the second relational information j2, to convert the three-dimensional coordinates of the projector 100 in the measuring coordinate system CS2 into the three-dimensional coordinates of the projector 100 in the imaginary space coordinate system CS3. The position determination unit 347 determines the position represented by the three-dimensional coordinates of the projector 100 in the imaginary space coordinate system CS3 as the position of the projector 100 in the imaginary space coordinate system CS3, i.e., the first imaginary position r1.

[0222] Next, in step S110, the virtual space control unit 348 configures the virtual camera 400v at the first virtual position r1.

[0223] In step S110, the imaginary space control unit 348 positions the principal point of the imaginary shooting lens 410v of the imaginary camera 400v at the first imaginary position r1.

[0224] Next, in step S111, the imaginary space control unit 348 displays the original image G2 without distortion on the imaginary object 2v. The imaginary space control unit 348 displays the original image G2 on the imaginary object 2v by pasting an imaginary piece of paper representing the original image G2 onto the imaginary object 2v.

[0225] Next, in step S112, the image generation unit 349 generates a hypothetical captured image, obtained by the hypothetical camera 400v capturing the original image G2 displayed on the hypothetical object 2v, as a deformed image G1. The hypothetical captured image becomes an image that undergoes a second deformation of the original image G2, which cancels out the first deformation generated in the original image G2 projected onto the object 2. The image generation unit 349 generates image data representing the hypothetical captured image as deformed image data representing the deformed image G1.

[0226] Next, in step S113, the projection control unit 342 provides the projector 100 with the deformed image data representing the deformed image G1, thereby causing the projector 100 to project the deformed image G1 onto the object 2.

[0227] When a deformed image G1 is projected onto object 2, a first deformation occurs in the deformed image G1 shown on object 2. This first deformation cancels out a second deformation in the deformed image G1. Therefore, as... Figure 5 As shown, object 2 is shown as deformed image G1 in the same form as the original image G2.

[0228] A7: Summary of the first implementation method

[0229] The first generation unit 345 generates first relational information j1 ​​based on the three-dimensional coordinates h of the portion of the object 2 where the measurement point e is located (i.e., the corresponding point f) and the two-dimensional coordinates i of the measurement point e in the measurement image G3, when the projector 100 projects the measurement image G3 onto the object 2. The first relational information j1 ​​represents the correspondence between the measuring coordinate system CS2 used by the measuring instrument 200 to determine the coordinates h and the projector coordinate system CS1 used to determine the coordinates i and the coordinates of the projector 100. The position determination unit 347 determines the position of the projector 100 in the measuring coordinate system CS2 based on the coordinates of the projector 100 using the first relational information j1.

[0230] According to this method, even if the projector 100 and the measuring device 200 are positioned arbitrarily, the position of the projector 100 can be determined in the measuring device coordinate system CS2 used by the measuring device 200. The position of the projector 100 in the measuring device coordinate system CS2 is the information required to generate the deformed image G1. Therefore, even if the projector 100 and the measuring device 200 are positioned arbitrarily, it is possible to assist in projecting a prescribed image, such as the deformed image G1, onto the object 2.

[0231] The first generation unit 345 generates first relational information j1 ​​based on the three-dimensional coordinates h of the portion of the object 2 where the measurement point e is located (i.e., the corresponding point f) and the two-dimensional coordinates i of the measurement point e in the measurement image G3, when the projector 100 projects the measurement image G3 onto the object 2. The first relational information j1 ​​represents the correspondence between the measuring coordinate system CS2 used by the measuring instrument 200 to determine the coordinates h and the projector coordinate system CS1 used to determine the coordinates i and the coordinates of the projector 100. The second generation unit 346 generates second relational information j2 based on the coordinates m of the object point k in the measuring coordinate system CS2 and the coordinates n of the imaginary object point q corresponding to the object point k in the three-dimensional imaginary space d1. The second relational information j2 represents the correspondence between the measuring coordinate system CS2 and the imaginary space coordinate system CS3 used to determine the coordinates in the imaginary space d1. The position determination unit 347 determines the position of the projector 100 in the imaginary space coordinate system CS3 by using the first relation information j1 ​​and the second relation information j2.

[0232] According to this method, even if the projector 100 and the measuring device 200 are positioned arbitrarily, the position of the projector 100 can be determined in the imaginary spatial coordinate system CS3. Therefore, based on the position of the projector 100 in the imaginary spatial coordinate system CS3, a prescribed image such as the deformed image G1 can be generated. Therefore, even if the projector 100 and the measuring device 200 are positioned arbitrarily, it is possible to assist in projecting a prescribed image such as the deformed image G1 onto the object 2.

[0233] The acquisition unit 344 acquires the coordinates h from the measuring device 200, which has a first camera 210 and a second camera 220. The measuring device 200 determines the coordinates h based on a first image generated by the first camera 210 capturing an image of the corresponding point f in the object 2 where the measuring point e is located, and a second image generated by the second camera 220 capturing an image of the corresponding point f in the object 2 where the measuring point e is located. In this way, a stereo camera can be used as the measuring device 200. Therefore, the coordinates h are easily acquired.

[0234] The imaginary space control unit 348 sets a second imaginary position r2 in the imaginary space d1 corresponding to the position of the object 2. The imaginary space control unit 348 positions an imaginary object 2v corresponding to the object 2 at the second imaginary position r2 in the imaginary space d1. The imaginary space control unit 348 displays the original image G2 on the imaginary object 2v. The imaginary space control unit 348 positions a virtual camera 400v in the imaginary space d1 at the position of the projector 100 in the imaginary space coordinate system CS3. The image generation unit 349 generates a deformed image G1 obtained by the virtual camera 400v capturing the original image G2 displayed on the imaginary object 2v. According to this method, when the projector 100 projects onto the object 2, a deformed image G1 that is displayed in the same form as the original image G2 can be generated.

[0235] The projection control unit 342 causes the projector 100 to project a distorted image G1 onto the object 2. In this manner, an image can be projected onto the object 2 in the same form as the original image G2.

[0236] The measured image G3 is a pattern image representing the brightness change corresponding to a sine wave. According to this method, the phase-shifted image used in the so-called phase-shifting method can be used as the measured image G3.

[0237] B: Variation Example

[0238] Below, variations of the embodiments illustrated above are given. It is also possible to appropriately combine two or more embodiments selected arbitrarily from the following examples, provided they do not contradict each other.

[0239] B1: First Variation

[0240] In the first embodiment, the first generation unit 345 can also determine the rotation matrix R and the parallel translation matrix T by using Equation 5 instead of Equation 2.

[0241] v2·(T×Rv1)=0…Equation 5

[0242] Equation 5 is known as the epipolar equation.

[0243] Figure 18 This is a diagram illustrating vectors v1 and v2 shown in Equation 5. In the first variation, the principal point of the projection lens 140 is not set on the two-dimensional plane 130s defined by the projector coordinate system CS1, but rather at position C, which is separated from the two-dimensional plane 130s by the focal length of the projection lens 140 along the z1 axis. Figure 18 For the sake of simplicity, the positions of the two-dimensional plane 130s and the first image sensor 212 are adjusted. In reality, the two-dimensional plane 130s exists at a position symmetrical to point C. The first image sensor 212 exists at a position symmetrical to the origin o2.

[0244] The cross product of two vectors contained in the epipolar plane K represents the vector perpendicular to the epipolar plane K. Therefore, the dot product of the cross product of two vectors contained in the epipolar plane K and the dot product of other vectors contained in the epipolar plane K is "0". Equation 5 is determined based on this relationship.

[0245] The first generation unit 345 determines multiple pairs of vectors v1 and v2 based on multiple first coordinate pairs. These pairs of vectors v1 and v2 are referred to as "vector pairs". Using these multiple vector pairs, the first generation unit 345 solves the epipolar equation of Equation 5 using methods such as nonlinear least squares, thereby determining the rotation matrix R and the parallel translation matrix T.

[0246] Equation 5 can constrain the direction of the vector defined by the parallel translation matrix T. However, Equation 5 cannot constrain the magnitude of the vector defined by the parallel translation matrix T. Therefore, in order to define the magnitude of the vector defined by the parallel translation matrix T, the first generation unit 345 uses Equation 6.

[0247]

[0248] In Equation 6, (x,y,z) is generated by converting the two-dimensional coordinates of the projector coordinate system CS1 in the first coordinate pair into three-dimensional coordinates. The first generation unit 345 converts the two-dimensional coordinates of the projector coordinate system CS1 in the first coordinate pair into three-dimensional coordinates by using the focal length of the projection lens 140 as z in (x,y,z) in Equation 6.

[0249] According to the first variation, by using the epipolar equation, the rotation matrix R and the parallel translation matrix T representing the relationship between the projector coordinate system CS1 and the measuring instrument coordinate system CS2 can be determined.

[0250] B2: Second Variation

[0251] In the first embodiment and the first variation, a zoom lens may also be used as the projection lens 140 of the projector 100.

[0252] In this case, the internal parameters of the projection lens 140 change according to the zoom state in the zoom lens. Therefore, when the first generation unit 345 uses Equation 2, it is necessary to determine the internal parameter matrix A based on the rotation matrix R and the parallel translation matrix T. When calculating the rotation matrix R, the parallel translation matrix T, and the internal parameter matrix A using Equation 2, at least 10 first coordinate pairs are required. Therefore, the first generation unit 345 determines the rotation matrix R, the parallel translation matrix T, and the internal parameter matrix A by using more than 10 first coordinate pairs.

[0253] The coordinates represented by the first coordinate pair may contain errors. Therefore, the more first coordinate pairs substituted into Equation 2, the higher the accuracy of the rotation matrix R, the parallel translation matrix T, and the intrinsic parameter matrix A. Thus, it is preferable that the first generation unit 345 determines the rotation matrix R, the parallel translation matrix T, and the intrinsic parameter matrix A by using more first coordinate pairs. The first generation unit 345 generates a group of the rotation matrix R, the parallel translation matrix T, the intrinsic parameter matrix A, and the scaling factor s as first relational information j1.

[0254] According to the second variation, a zoom lens can be used as the projection lens 140.

[0255] B3: Third Variation

[0256] In the first embodiment and the first variation, a two-dimensional coordinate system, namely the shooting coordinate system, can also be applied to the first image sensor 212 of the first camera 210.

[0257] In this case, the first generation unit 345 can also generate first information indicating the correspondence between the projector coordinate system CS1 and the shooting coordinate system. The second generation unit 346 can also generate second information indicating the correspondence between the shooting coordinate system and the imaginary space coordinate system CS3. The position determination unit 347 can also determine the position of the projector 100 in the imaginary space coordinate system CS3, i.e., the first imaginary position r1, based on the coordinates of the projector 100 in the projector coordinate system CS1 by using the first information and the second information.

[0258] For example, the first generation unit 345 generates first information representing the correspondence between the projector coordinate system CS1 and the shooting coordinate system by using multiple pairs of two-dimensional coordinates of the measurement point e in the projector coordinate system CS1 and the corresponding point f in the shooting coordinate system. The first information is, for example, a projective transformation matrix based on the projector coordinate system CS1 and the shooting coordinate system.

[0259] The second generation unit 346 generates second information representing the correspondence between the shooting coordinate system and the imaginary space coordinate system CS3 by using multiple pairs of two-dimensional coordinates of the object point k in the shooting coordinate system and three-dimensional coordinates of the imaginary object point q in the imaginary space coordinate system CS3. The second generation unit 346 determines the unknowns in Equation 2 of the aforementioned perspective projection transformation, specifically the rotation matrix R and the parallel translation matrix T, by using multiple pairs of two-dimensional coordinates of the object point k in the shooting coordinate system and three-dimensional coordinates of the imaginary object point q in the imaginary space coordinate system CS3. In this case, the second generation unit 346 applies the internal parameters of the first shooting lens 211 to the internal parameter matrix A.

[0260] First, the position determination unit 347 determines the position of the projector 100 in the shooting coordinate system by using the first information and based on the coordinates of the projector 100 in the projector coordinate system CS1.

[0261] Next, the position determination unit 347 uses the second information to determine the position of the projector 100 in the imaginary space coordinate system CS3 based on the two-dimensional coordinates representing the position of the projector 100 in the shooting coordinate system.

[0262] The third variation has the following characteristics.

[0263] The first generation unit 345 generates first information based on the two-dimensional coordinates (i.e., shooting coordinates) of the portion of the image where the measuring point e is located (i.e., the corresponding point f) in the image generated by the first camera 210 capturing the object 2 when the projector 100 projects the measuring image G3 onto the object 2, and the two-dimensional coordinates (i.e., projection coordinates) of the measuring point e in the measuring image G3. The first information represents the correspondence between the camera coordinate system that determines the shooting coordinates and the projector coordinate system CS1 that determines the projection coordinates and the coordinates of the projector 100. The second generation unit 346 generates second information based on the coordinates of the object point k in the camera coordinate system and the coordinates of the imaginary object point q corresponding to the object point k in the three-dimensional imaginary space d1. The second information represents the correspondence between the camera coordinate system and the imaginary space coordinate system CS3 that represents the coordinates in the imaginary space d1. The position determination unit 347 determines the position of the projector 100 in the imaginary space coordinate system CS3 based on the coordinates of the projector 100 by using the first information and the second information.

[0264] According to the third variation, even if the projector 100 and the first camera 210 are positioned arbitrarily, the position of the projector 100 can be determined in the imaginary spatial coordinate system CS3. Therefore, a deformed image G1 or other prescribed image can be generated based on the position of the projector 100 in the imaginary spatial coordinate system CS3. Therefore, even if the projector 100 and the first camera 210 are positioned arbitrarily, it is possible to assist in projecting a deformed image G1 or other prescribed image onto the object 2. Furthermore, the second camera 220 can be omitted. Furthermore, the generation of the first relational information j1 ​​and the second relational information j2 can be omitted.

[0265] B4: Fourth Variation

[0266] In the first embodiment and the first to second modifications, the origin o2 of the measuring coordinate system CS2 is set at the position of the principal point of the first shooting lens 211. Therefore, the coordinates of the first camera 210 in the measuring coordinate system CS2 are known. Therefore, the position determination unit 347 can determine the coordinates of the first camera 210 in the imaginary space coordinate system CS3 by using the second relational information j2.

[0267] The second generation unit 346 can also use a hypothetical image obtained from the position of the first camera 210 in the hypothetical space coordinate system CS3, through a view body having the internal parameters of the first camera 210, to observe the hypothetical object 2v, and update the second relational information j2.

[0268] For example, the second generation unit 346 updates the second relational information j2 based on the object image obtained by the first camera 210 taking a picture of the object 2 and the hypothetical image obtained by observing the hypothetical object 2v from the position of the first camera 210 in the hypothetical space coordinate system CS3 through a view having the internal parameters of the first camera 210.

[0269] For example, the second generation unit 346 optimizes the second relational information j2 to minimize the sum of squares of the errors between the position of the first object point k1 shown in the object captured image and the position of the first imaginary object point q1 shown in the imaginary captured image. The point used to determine the error is not limited to the first object point k1; for example, it could be the second object point k2, or a point in the object 2 located in an area where no image is projected from the projector 100. Furthermore, in the fourth variation, this is based on the premise that the internal parameters of the first camera 210 are known. When applying the method of the fourth variation to the third variation, the second generation unit 346 optimizes the second information j2 as described above, instead of the second relational information j2.

[0270] According to the fourth variation, the accuracy of the second relational information j2 and the accuracy of the second information can be improved.

[0271] B5: Fifth Variation

[0272] In the fourth variation, the hypothetical captured image represents an image of the area within the shooting range of the hypothetical camera 400v contained in the hypothetical object 2v. Here, the internal parameters of the hypothetical shooting lens 410v of the hypothetical camera 400v are equal to the internal parameters of the projection lens 140 of the projector 100. Therefore, the shooting range of the hypothetical camera 400v is equal to the projection range of the projected image when the hypothetical projector, having the same internal parameters as the projector 100, is configured at the first hypothetical position r1. The three-dimensional coordinates of multiple points contained in the area of ​​the hypothetical object 2v that becomes the shooting range of the hypothetical camera 400v are known in the hypothetical space coordinate system CS. The second generation unit 346 can also use the known three-dimensional coordinates to update the second relational information j2.

[0273] Suppose that object 2 has multiple sample points that correspond one-to-one with multiple points contained within the shooting range of the imaginary camera 400v in imaginary object 2v. In this case, the second generation unit 346 causes the measuring device 200 to measure the three-dimensional coordinates of the multiple sample points in the measuring device coordinate system CS2. Next, the second generation unit 346 obtains the three-dimensional coordinates of the multiple sample points in the measuring device coordinate system CS2 from the measuring device 200. Next, the second generation unit 346 optimizes the second relationship information j2 according to each sample point to minimize the sum of the squares of the differences between the three-dimensional coordinates of the sample point in the measuring device coordinate system CS2 and the three-dimensional coordinates of the point corresponding to that sample point in the imaginary space coordinate system CS3. In addition, this method matches the shape of the point group using three-dimensional coordinates, and therefore is used when the area in imaginary object 2v that is the shooting range of the imaginary camera 400v is not a completely planar surface.

[0274] According to the fifth variation, the accuracy of the second relational information j2 can be improved.

[0275] B6: Sixth Variation

[0276] In the first embodiment and the first to fifth modifications, the position of the object point k may not be the position of a physical marker, but rather the position of a feature point of the object 2. A feature point could be, for example, a bolt hole or a protrusion. In this case, the hypothetical object point q is set based on the position of the feature point of the object 2.

[0277] According to the sixth variation, the feature points of object 2 can be used as object point k.

[0278] B7: 7th Variation

[0279] In the first embodiment and the first to sixth modifications, the measurement image G3 is not limited to a phase-shifted image and can be appropriately modified. For example, the measurement image G3 may also be an image showing a mark at the measurement point e.

[0280] Figure 19 This is another example of a measurement image G3. Figure 19 Image G35 shows point G35a at the location of measurement point e. The center position of point G35a can be easily calculated using methods such as centroid detection. Therefore, it is preferable that the center of point G35a is located at the location of each measurement point e. Furthermore, the marker is not limited to point G35a; for example, it can be a polygonal marker or a marker with a shape where two lines intersect.

[0281] According to the seventh variation, compared with the structure using phase-shifted images, the number of measurement images G3 can be reduced.

[0282] B8: Eighth Variation

[0283] In the first embodiment and the first to seventh modifications, the information processing device 300 may also be incorporated into the projector 100 or the measuring device 200.

[0284] B9: 9th Variation

[0285] In the first embodiment and the first to eighth modifications, a liquid crystal light valve 130 is used as an example of a light modulation device. However, the light modulation device is not limited to a liquid crystal light valve and can be modified appropriately. For example, the light modulation device may also be a structure that uses a single digital mirror device. In addition, in addition to a liquid crystal panel and a DMD, any structure capable of modulating the light emitted by the light source 120 can also be used as a light modulation device.

Claims

1. A method for determining a location, comprising the following steps: Based on the first three-dimensional coordinates of the portion of the object containing the specific point and the second two-dimensional coordinates of the specific point in the projected image, when a projector projects a projected image of a specific point onto an object, relational information is generated. This relational information represents the correspondence between the three-dimensional coordinate system used by the measuring instrument that determines the first coordinate and the projector coordinate system that determines the second coordinate and the coordinates of the projector. The relationship information is determined as the positional relationship between the measuring instrument and the projector in the three-dimensional coordinate system.

2. A simulation method comprising the following steps: Based on the three-dimensional first coordinates of the portion of the object containing the first specific point and the two-dimensional second coordinates of the first specific point in the projected image, when the projector projects a projected image of the first specific point onto the object, first relationship information is generated. This first relationship information represents the correspondence between the three-dimensional coordinate system used by the measuring instrument that determines the first coordinates and the projector coordinate system that determines the second coordinates and the coordinates of the projector. Based on the coordinates of a second specific point in the object in the three-dimensional coordinate system and the coordinates of a third specific point corresponding to the second specific point in the three-dimensional imaginary space, a second relationship information is generated. This second relationship information represents the correspondence between the three-dimensional coordinate system and the imaginary space coordinate system that determines the coordinates in the imaginary space. The first relationship information, transformed using the second relationship information, is used to determine the positional relationship between the measuring instrument and the projector in the hypothetical spatial coordinate system.

3. The simulation method according to claim 2, wherein, The simulation method further includes the step of obtaining the first coordinates from a measuring device having a first camera and a second camera. The measuring device determines the first coordinate based on a first captured image generated by the first camera capturing the portion of the object containing the first specific point, and a second captured image generated by the second camera capturing the portion of the object containing the first specific point.

4. A simulation method comprising the following steps: Based on the shooting coordinates and projection coordinates, first information is generated. The shooting coordinates are the two-dimensional coordinates of the portion of the object containing the first specific point in the image generated by the camera when the projector projects an image of the object containing the first specific point onto the object. The projection coordinates are the two-dimensional coordinates of the first specific point in the projection image. The first information represents the correspondence between the camera coordinate system that determines the shooting coordinates and the projector coordinate system that determines the projection coordinates and the coordinates of the projector. Based on the coordinates of a second specific point in the object in the camera coordinate system and the coordinates of a third specific point corresponding to the second specific point in a three-dimensional imaginary space, second information is generated. This second information represents the correspondence between the camera coordinate system and the imaginary space coordinate system representing the coordinates in the imaginary space. By using the first information and the second information, the positional relationship between the camera and the projector in the hypothetical spatial coordinate system is determined.

5. The simulation method according to any one of claims 2 to 4, wherein, The simulation method further includes the following steps: In the imaginary space, an imaginary position corresponding to the position of the object is set. The imaginary object corresponding to the stated object is placed at the imaginary position in the imaginary space. The first image is displayed on the hypothetical object. In the hypothetical space, a hypothetical camera is positioned at the location of the projector in the hypothetical space coordinate system. A second image is generated by the imaginary camera capturing the first image displayed on the imaginary object.

6. The simulation method according to claim 5, wherein, The simulation method further includes the step of projecting the second image onto the object using the projector.

7. The simulation method according to any one of claims 2 to 4, wherein, The projected image is a patterned image showing the brightness changes corresponding to a sine wave.

8. The simulation method according to any one of claims 2 to 4, wherein, The projected image is an image showing the mark at the location of the first specific point.

9. A location determination system, comprising: The generation unit generates relational information based on the three-dimensional first coordinates of the portion of the object containing the specific point and the two-dimensional second coordinates of the specific point in the projected image, when a projector projects a projected image containing a specific point onto the object. This relational information represents the correspondence between the three-dimensional coordinate system used by the measuring instrument determining the first coordinates and the projector coordinate system used to determine the second coordinates and the coordinates of the projector. The position determination unit determines the relationship information as the positional relationship between the measuring instrument and the projector in the three-dimensional coordinate system.

10. A simulation system comprising: The first generation unit generates first relational information based on the three-dimensional first coordinates of the portion of the object containing the first specific point and the two-dimensional second coordinates of the first specific point in the projected image when the projector projects a projected image having the first specific point onto the object. The first relational information represents the correspondence between the three-dimensional coordinate system used by the measuring instrument that determines the first coordinates and the projector coordinate system that determines the second coordinates and the coordinates of the projector. The second generation unit generates second relationship information based on the coordinates of a second specific point in the object in the three-dimensional coordinate system and the coordinates of a third specific point corresponding to the second specific point in a three-dimensional imaginary space. This second relationship information represents the correspondence between the three-dimensional coordinate system and the imaginary space coordinate system that determines the coordinates in the imaginary space. The position determination unit determines the positional relationship between the measuring instrument and the projector in the imaginary spatial coordinate system by using the first relationship information transformed using the second relationship information.

11. A simulation system comprising: The first generation unit generates first information based on the shooting coordinates and the projection coordinates. The shooting coordinates are two-dimensional coordinates of the portion of the object where the first specific point is located in the shooting image generated by the camera when the projector projects a projection image of the object with the first specific point onto the object. The projection coordinates are two-dimensional coordinates of the first specific point in the projection image. The first information indicates the correspondence between the camera coordinate system that determines the shooting coordinates and the projector coordinate system that determines the projection coordinates and the coordinates of the projector. The second generation unit generates second information based on the coordinates of a second specific point in the object in the camera coordinate system and the coordinates of a third specific point corresponding to the second specific point in a three-dimensional imaginary space. This second information represents the correspondence between the camera coordinate system and the imaginary space coordinate system representing the coordinates in the imaginary space. The position determination unit determines the positional relationship between the camera and the projector in the hypothetical spatial coordinate system by using the first information and the second information.

Images