Method, projector, and system for determining a projective transformation matrix
By projecting multiple images with adjusted feature point positions, the method addresses the issue of three-dimensional obstructions, ensuring accurate projective transformation matrix determination and image alignment.
Patent Information
- Application Number
- JP2020183521
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-11-02
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2040-11-02
AI Technical Summary
Existing methods for determining a projective transformation matrix fail when a three-dimensional object obstructs the projection, as feature points in the projected image may not align with the same flat surface, preventing accurate correlation between projected and captured images.
A method where a projector projects multiple images with varying positional relationships of feature points, allowing a camera to capture these changes, enabling the determination of a projective transformation matrix by aligning feature points on a flat surface despite three-dimensional obstructions.
Ensures accurate determination of the projective transformation matrix even when three-dimensional objects interfere, by adjusting feature point positions to ensure all points are projected onto a flat surface, facilitating precise image alignment.
Smart Images

Figure 0007593061000003 
Figure 0007593061000004 
Figure 0007593061000005
Abstract
Description
[Technical field]
[0001] The present invention relates to a method, a projector, and a system for determining a homography matrix. [Background technology]
[0002] Patent Document 1 discloses a method for determining a projective transformation matrix. The projective transformation matrix described in Patent Document 1 associates a position in a projected image projected from a projector onto a screen with a position in a captured image obtained by a camera capturing an image of the screen onto which the projected image is projected. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2012-249009 A Summary of the Invention [Problem to be solved by the invention]
[0004] In order to determine the projective transformation matrix, the projected image must have at least four feature points, and for each feature point, a correspondence between the coordinates of the feature point in the projected image and the coordinates of the feature point in the captured image must be known. Furthermore, each feature point in the projected image must be projected onto the same flat display surface.
[0005] However, in a situation where a projection image is projected onto a flat display surface such as a wall for the purpose of signage in a store or the like, if there is a three-dimensional object such as a bronze statue in front of the display surface, there is a risk that any of the feature points in the projection image will be projected onto the three-dimensional object rather than onto the display surface. In this case, since the feature points in the projection image are not projected onto the flat display surface, there is a risk that it will be impossible to determine a projective transformation matrix that associates positions in the projection image with positions in the captured image. [Means for solving the problem]
[0006] One aspect of the method for determining a projective transformation matrix according to the present invention includes receiving a change instruction to change a positional relationship between the four feature points when a projector projects a first projected image indicating four feature points onto a first area, causing the projector to project a second projected image indicating the four feature points onto the first area in response to the change instruction, and determining a projective transformation matrix that mutually corresponds between a projector coordinate system that represents the coordinates in the projected image projected by the projector and a camera coordinate system that represents the coordinates in the captured image generated by the camera, based on coordinates of the four feature points in the second projected image and coordinates of the four feature points in a first captured image generated by a camera capturing an image of the first area onto which the second projected image is projected, wherein the positional relationship of the four feature points in the second projected image is different from the positional relationship of the four feature points in the first projected image.
[0007] One aspect of the projector according to the present invention is A projection unit that projects an image; the projection control unit projects a second projection image indicating the four feature points onto the first area in response to the change instruction; and a determination unit determines a projection transformation matrix that mutually corresponds between a projection coordinate system that indicates the coordinates in the projection image projected by the projection unit and a camera coordinate system that indicates the coordinates in the captured image generated by the camera, based on coordinates of the four feature points in the second projection image and coordinates of the four feature points in a first captured image generated by a camera capturing an image of the first area onto which the second projection image is projected, wherein the positional relationship of the four feature points in the second projection image is different from the positional relationship of the four feature points in the first projection image.
[0008] One aspect of the determination system of the present invention includes a projector that projects an image, a camera that performs imaging, and a control device that controls the projector and the camera, wherein the control device includes a receiving unit that receives a change instruction to change the positional relationship of the four feature points when the projector projects a first projection image indicating the four feature points into a first area, a projection control unit that causes the projector to project a second projection image indicating the four feature points into the first area in response to the change instruction, and a determination unit that determines a projection transformation matrix that mutually corresponds between a projector coordinate system that represents coordinates in a projection image projected by the projector and a camera coordinate system that represents coordinates in a captured image generated by the camera, based on coordinates of the four feature points in the second projection image and coordinates of the four feature points in a first captured image generated by the camera capturing the first area into which the second projection image is projected, and the positional relationship of the four feature points in the second projection image is different from the positional relationship of the four feature points in the first projection image. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a diagram illustrating a determination system 1 according to a first embodiment. [Diagram 2] FIG. 2 is a diagram showing a first projected image c1, which is an example of a projected image. [Diagram 3] FIG. 13 is a diagram showing a second projected image c2, which is another example of a projected image. [Figure 4] 1 is a diagram illustrating an example of a projector 100 and an example of a camera 200. FIG. [Diagram 5] FIG. 2 is a diagram showing an example of a liquid crystal light valve 142. [Figure 6] FIG. 2 illustrates an example of an image sensor 220. [Figure 7] FIG. 2 is a diagram showing an example of a captured image f1. [Figure 8] FIG. 2 is a diagram for explaining the operation of the decision system 1. [Figure 9] FIG. 13 is a diagram showing a projection example of a first projected image c1. [Figure 10] FIG. 2 illustrates an example of an operation unit 110. [Figure 11] FIG. 11 is a diagram showing a projector 100a according to a third modified example. [Figure 12] FIG. 13 is a diagram illustrating a determination system 1 according to a fourth modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] A: First embodiment A1: Decision System 1 1 is a diagram showing a determination system 1 according to the first embodiment. The determination system 1 includes a projector 100 and a camera 200.
[0011] The projector 100 projects a projection image c onto a projection target 300. A projector coordinate system J1 is applied to the projection image c. The projector coordinate system J1 represents coordinates in the projection image c. The projector coordinate system J1 is an example of a projection coordinate system.
[0012] The projection object 300 includes a flat display surface 400 and a three-dimensional object 500. The display surface 400 is a wall. The display surface 400 is not limited to a wall, and may be, for example, a screen, a table top, a floor, a ceiling, a whiteboard, or a door. The three-dimensional object 500 is a bronze statue. The three-dimensional object 500 is not limited to a bronze statue, and may be, for example, a doll, a pillar, or a shelf. The area of the projection object 300 onto which the projection image c is projected is an example of a first area.
[0013] The camera 200 generates a captured image by capturing an image of an area of the projection target 300 including an area onto which the projection image c is projected. A camera coordinate system J2 is applied to the captured image. The camera coordinate system J2 represents coordinates in the captured image. The camera 200 generates captured image data b indicating the captured image. The camera 200 provides the captured image to the projector 100 by providing the captured image data b to the projector 100.
[0014] The projector 100 uses the projected image c and the captured image to determine a projective transformation matrix H that associates the projector coordinate system J1 with the camera coordinate system J2.
[0015] A2: Projected image c 2 is a diagram showing a first projected image c1 which is an example of the projected image c. The first projected image c1 shows four white dots d1 to d4 on a black background. The positional relationship of the dots d1 to d4 in the first projected image c1 is an example of a first positional relationship.
[0016] In the first projected image c1, the color of the background is not limited to black, but may be any color different from the colors of the dots d1 to d4. The color of the dots d1 to d4 is not limited to white, but may be any color different from the color of the background. The positional relationship of the four dots d1 to d4 in the first projected image c1 is not limited to the positional relationship shown in FIG. 2, but may be changed as appropriate. Each of the dots d1 to d4 is an example of a feature point. The feature point is not limited to a dot, but may be, for example, a cross mark, a polygon, an ellipse, an intersection of multiple lines, or an intersection of multiple shapes. The number of feature points is not limited to four, but may be five or more. The four dots d1 to d4 are an example of four feature points, or more specifically, an example of at least four feature points.
[0017] 3 is a diagram showing a second projected image c2, which is another example of the projected image c. The second projected image c2 shows four white dots d1 to d4 on a black background, similar to the first projected image c1. The positional relationship of the dots d1 to d4 in the second projected image c2 is an example of a second positional relationship.
[0018] In the second projected image c2, the color of the background is not limited to black, but may be any color different from the colors of the dots d1 to d4. In the second projected image c2, the color of the dots d1 to d4 is not limited to white, but may be any color different from the color of the background. The number of dots shown by the second projected image c2 is equal to the number of dots shown by the first projected image c1.
[0019] The positional relationship of the four dots d1 to d4 in the second projected image c2 is different from the positional relationship of the four dots d1 to d4 in the first projected image c1. The positional relationship of the four dots d1 to d4 is different means that the position of one or more dots among the four dots d1 to d4 is different between the first projected image c1 and the second projected image c2. For example, as shown in FIG. 2 and FIG. 3, a state in which only the position of the dot d3 among the four dots d1 to d4 is different between the first projected image c1 and the second projected image c2 means that the positional relationship of the four dots d1 to d4 in the second projected image c2 is different from the positional relationship of the four dots d1 to d4 in the first projected image c1. The positional relationship of the four dots d1 to d4 in the second projected image c2 is not limited to the positional relationship shown in FIG. 3 and can be changed as appropriate as long as it is different from the positional relationship of the four dots d1 to d4 in the first projected image c1.
[0020] The projector 100 is capable of projecting the first projection image c1 and the second projection image c2 onto the projection object 300 separately.
[0021] A3: Projector 100 and camera 200 4 is a diagram showing an example of the projector 100 and an example of the camera 200. The projector 100 includes an operation unit 110, a storage unit 120, a processing unit 130, and a projection unit 140.
[0022] The operation unit 110 has at least one of an operation button and a touch panel. A user operates the operation unit 110 to input various instructions to the operation unit 110. The operation unit 110 receives the instructions input by the user. The operation unit 110 may receive instructions from the user via a remote control (not shown). The operation unit 110 is an example of an instruction input unit.
[0023] The storage unit 120 is a recording medium readable by the processing unit 130. The storage unit 120 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM), and an Electrically Erasable Programmable Read Only Memory (EEPROM). The volatile memory is, for example, a Random Access Memory (RAM).
[0024] The storage unit 120 stores a program P executed by the processing unit 130 and various data used by the processing unit 130.
[0025] The processing unit 130 is configured with one or more central processing units (CPUs). The one or more CPUs are an example of one or more processors. Each of the processor and the CPU is an example of a computer.
[0026] The processing unit 130 reads the program P from the storage unit 120. The processing unit 130 executes the program P to realize various functions. For example, the processing unit 130 provides image data a indicating a projection image c to the projection unit 140, thereby causing the projection unit 140 to project the projection image. The image data a is represented by a voltage. The processing unit 130 receives a captured image, more specifically, captured image data b, from the camera 200. The processing unit 130 uses the projection image c and the captured image to determine a projection transformation matrix H that associates the projector coordinate system J1 with the camera coordinate system J2.
[0027] Some or all of the functions of the processing unit 130 may be realized by a circuit such as a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA). The processing unit 130 is an example of a control device.
[0028] The projection unit 140 projects the projection image c onto the projection target 300. The projection unit 140 includes a light source 141, a liquid crystal light valve 142, and a projection lens 143.
[0029] The light source 141 is a light emitting diode (LED). The light source 141 is not limited to an LED, and may be, for example, a xenon lamp, an extra-high pressure mercury lamp, or a laser light source.
[0030] The liquid crystal light valve 142 is composed of a liquid crystal panel in which liquid crystal exists between a pair of transparent substrates. The liquid crystal light valve 142 has a rectangular pixel area 142a as shown in Fig. 5. The pixel area 142a includes a plurality of pixels 142p arranged in a matrix.
[0031] A projector coordinate system J1 is applied to the liquid crystal light valve 142. The projector coordinate system J1 is defined by an x-axis and a y-axis.
[0032] The x-axis and y-axis are determined according to the orientation of the liquid crystal light valve 142. The x-axis is parallel to the horizontal direction of the liquid crystal light valve 142, i.e., parallel to the lateral direction of the liquid crystal light valve 142. The y-axis is perpendicular to the x-axis. The y-axis is parallel to the vertical direction of the liquid crystal light valve 142, i.e., parallel to the longitudinal direction of the liquid crystal light valve 142.
[0033] In the liquid crystal light valve 142, a voltage representing the image data a is applied to the liquid crystal for each pixel 142p. The pixel 142p is set to a light transmittance based on the image data a.
[0034] 4, light emitted from a light source 141 is modulated by a pixel region 142a of a liquid crystal light valve 142. The liquid crystal light valve 142 is an example of a light modulation device. The light modulated by the liquid crystal light valve 142 travels toward a projection lens 143. The projection lens 143 projects the light modulated by the liquid crystal light valve 142, i.e., a projection image c, onto a projection object 300.
[0035] Note that a configuration including at least the projection unit 140, for example, a configuration in which the operation unit 110, the storage unit 120, and the processing unit 130 are omitted from the projector 100, functions as a projector. For this reason, the projection unit 140 may also be referred to as a projector.
[0036] The camera 200 is a separate component from the projector 100. The camera 200 may be incorporated into the projector 100. The camera 200 includes an imaging lens 210 and an image sensor 220.
[0037] The imaging lens 210 forms an optical image including a projection image c on the projection object 300 on the image sensor 220. The optical image including the projection image c is a captured image.
[0038] The image sensor 220 is a CCD (Charge Coupled Device) image sensor. The image sensor 220 is not limited to a CCD image sensor, and may be, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The image sensor 220 generates imaging data b indicating a captured image based on an optical image formed by the imaging lens 210. As shown in FIG. 6, the image sensor 220 has a rectangular imaging area 220a. The imaging area 220a includes a plurality of light receiving cells 220p arranged in a matrix.
[0039] A camera coordinate system J2 is applied to the image sensor 220. The camera coordinate system J2 is defined by a u axis and a v axis.
[0040] The u-axis and v-axis are determined according to the orientation of the image sensor 220. The u-axis is parallel to the horizontal direction of the image sensor 220, i.e., parallel to the lateral direction of the image sensor 220. The v-axis is perpendicular to the u-axis. The v-axis is parallel to the vertical direction of the image sensor 220, i.e., parallel to the longitudinal direction of the image sensor 220.
[0041] A4: An example of the processing unit 130 4, the processing unit 130 executes a program P to function as a projection control unit 131, a receiving unit 132, an imaging control unit 133, a determining unit 134, and an operation control unit 135. Note that the operation control unit 135 may be omitted.
[0042] The projection control unit 131 controls the projection unit 140. The projection control unit 131 causes the projection unit 140 to project the projection image c onto the projection target 300. The projection control unit 131 causes the projection unit 140 to project the projection image c onto the projection target 300 by providing image data a indicating the projection image c to the projection unit 140. For example, the projection control unit 131 causes the projection unit 140 to project the first projection image c1 onto the projection target 300 by providing image data a1 indicating a first projection image c1 to the projection unit 140. In addition, the projection control unit 131 causes the projection unit 140 to project the second projection image c2 onto the projection target 300 by providing image data a2 indicating a second projection image c2 to the projection unit 140.
[0043] The receiving unit 132 receives various instructions from the operation unit 110. For example, the receiving unit 132 receives a change instruction e1 to change the positional relationship of the dots d1 to d4 in a situation where the projection unit 140 projects the first projected image c1 onto the projection target 300. For example, the change instruction e1 is an instruction to change the positional relationship of the dots d1 to d4 in the first projected image c1 from a first positional relationship to a second positional relationship.
[0044] When the receiving unit 132 receives the change instruction e1 in a situation where the projection unit 140 projects the first projection image c1 onto the projection target 300, the projection control unit 131, in response to the change instruction e1, causes the projection unit 140 to project the second projection image c2 onto the projection target 300 instead of the first projection image c1. When the receiving unit 132 receives the change instruction e1 in a situation where the projection unit 140 projects the second projection image c2 onto the projection target 300, the projection control unit 131, in response to the change instruction e1, causes the projection unit 140 to project the first projection image c1 onto the projection target 300 instead of the second projection image c2.
[0045] The imaging control unit 133 controls the camera 200. The imaging control unit 133 makes the camera 200 capture the projection image c projected onto the projection target 300, thereby causing the camera 200 to generate a captured image. For example, the imaging control unit 133 makes the camera 200 capture an area of the projection target 300 including an area onto which the second projection image c2 is projected, thereby causing the camera 200 to generate a captured image f1. The captured image f1 is an example of the first captured image. FIG. 7 is a diagram showing an example of the captured image f1. The captured image f1 shown in FIG. 7 shows a state in which none of the dots d1 to d4 are projected onto the three-dimensional object 500 but are projected onto the display surface 400.
[0046] In FIG. 4, the determination unit 134 determines the projective transformation matrix H based on the coordinates of the dots d1 to d4 in the second projected image c2 and the coordinates of the dots d1 to d4 in the captured image f1.
[0047] The operation control unit 135 controls the operation of the projector 100 by using the projective transformation matrix H. For example, when the captured image shows a pointer such as a finger pointing at the display surface 400, the operation control unit 135 converts the coordinates of the pointer in the camera coordinate system J2 into the coordinates of the pointer in the projector coordinate system J1 by using the projective transformation matrix H. Next, the operation control unit 135 generates image data showing an image having a picture such as a line at the coordinates of the pointer in the projector coordinate system J1. The projection control unit 131 provides the image data generated by the operation control unit 135 to the projection unit 140, thereby causing the projection unit 140 to project an image having a picture such as a line at the coordinates of the pointer in the projector coordinate system J1.
[0048] A5: Projection transformation matrix H The projective transformation matrix H is specified based on eight coefficients. Therefore, the determination unit 134 specifies the eight coefficients to determine the projective transformation matrix H. This point will be described below.
[0049] The coordinates (x, y) on the plane represented by the projector coordinate system J1 and the coordinates (u, v) obtained by projecting the coordinates (x, y) onto the plane represented by the camera coordinate system J2 have the relationship shown in the following equations 1 and 2.
number
number
[0050] In Equation 1 and Equation 2, there are nine coefficients: α0, α1, α2, β0, β1, β2, γ0, γ1, and γ2. However, because both Equation 1 and Equation 2 are fractions, one of the nine coefficients can be determined to an arbitrary value. This results in eight independent unknown coefficients. Therefore, the homography matrix H can be determined based on the eight independent unknown coefficients.
[0051] The determination unit 134 generates simultaneous equations with eight unknowns by substituting the coordinates of dots d1 to d4 in the projector coordinate system J1 and the coordinates of dots d1 to d4 in the camera coordinate system J2 into formulas 1 and 2. The determination unit 134 determines eight independent unknown coefficients by solving the simultaneous equations with eight unknowns. Based on the eight determined coefficients, the determination unit 134 determines a projection transformation matrix H that performs projection transformation on coordinates (x, y) in the projector coordinate system J1 to coordinates (u, v) in the camera coordinate system J2.
[0052] The determination unit 134 calculates a projection transformation matrix H -1 The operation control unit 135 determines the projective transformation matrix H based on the projective transformation matrix H. -1 By using the above formula, the coordinates of the pointer in the camera coordinate system J2 are converted into the coordinates of the pointer in the projector coordinate system J1.
[0053] The determination unit 134 determines the projective transformation matrix H -1 In this case, the determination unit 134 may determine the projective transformation matrix H -1 The projection transformation matrix H is determined based on the above.
[0054] Projection transformation matrix H and projective transformation matrix H -1 Both of these are projection transformation matrices that associate the projector coordinate system J1 with the camera coordinate system J2.
[0055] Equations 1 and 2 used to determine the projective transformation matrix H indicate the relationship between coordinates (x, y) on a plane represented by the projector coordinate system J1 and coordinates (u, v) obtained by projecting the coordinates (x, y) onto a plane represented by the camera coordinate system J2. Therefore, when the determination unit 134 determines the projective transformation matrix H using equations 1 and 2, it is necessary that the coordinates of the dots d1 to d4 in the projector coordinate system J1 exist on the same plane, and that the coordinates of the dots d1 to d4 in the camera coordinate system J2 exist on the same plane.
[0056] The coordinates of the dots d1 to d4 in the projector coordinate system J1 exist on the projected image c, and therefore exist on the same plane. The coordinates of the dots d1 to d4 in the camera coordinate system J2 exist on the same plane when all of the four dots d1 to d4 in the projected image c are projected onto the flat display surface 400. When a part of the four dots d1 to d4 in the projected image c is projected onto the flat display surface 400, and the remaining four dots d1 to d4 in the projected image c are projected onto the three-dimensional object 500, the coordinates of the dots d1 to d4 in the camera coordinate system J2 do not exist on the same plane. For this reason, when a part of the four dots d1 to d4 in the projected image c is projected onto the flat display surface 400, and the remaining four dots d1 to d4 in the projected image c are projected onto the three-dimensional object 500, it is difficult to determine a valid projective transformation matrix H.
[0057] In this embodiment, when some of the four dots d1 to d4 in the first projected image c1 are projected onto the flat display surface 400 and the remaining four dots d1 to d4 in the first projected image c1 are projected onto the three-dimensional object 500, the first projected image c1 can be changed to a second projected image c2 in which the positional relationship of the four dots d1 to d4 is different from that of the first projected image c1 in response to a change instruction e1. Therefore, compared to a configuration in which the first projected image c1 cannot be switched, it is possible to easily create a situation in which all of the four dots d1 to d4 in the projected image c are projected onto the flat display surface 400.
[0058] A6: Description of operation 8 is a diagram for explaining the operation of the determination system 1. A user operates the operation unit 110 to input a start instruction to the operation unit 110. Upon receiving the start instruction, the operation unit 110 provides the start instruction to the processing unit 130.
[0059] When the processing unit 130 receives the start instruction, the projection control unit 131 causes the projection unit 140 to project the first projection image c1 onto the projection object 300 in step S101.
[0060] In step S101, the projection control unit 131 provides the projection unit 140 with image data a1 indicating a first projection image c1. If the image data a1 is stored in the storage unit 120, the projection control unit 131 reads the image data a1 from the storage unit 120. The projection control unit 131 provides the image data a1 read from the storage unit 120 to the projection unit 140. The projection control unit 131 may generate the image data a1 based on a program P. In this case, the projection control unit 131 provides the generated image data a1 to the projection unit 140. Upon receiving the image data a1, the projection unit 140 projects the first projection image c1 indicated by the image data a1 onto the projection target 300.
[0061] Here, as shown in FIG. 9, when only the dot d3 among the four dots d1 to d4 shown in the first projected image c1 is projected onto the three-dimensional object 500, not onto the display surface 400, the determination unit 134 cannot determine a valid projective transformation matrix H.
[0062] Therefore, the user operates the operation unit 110 to input a change instruction e1 to change the positional relationship of the dots d1 to d4 to the operation unit 110. The change instruction e1 is, for example, an instruction to change the position of at least one dot among the four dots d1 to d4 in the first projected image c1 projected onto the three-dimensional object 500. The at least one dot among the four dots d1 to d4 in the first projected image c1 is, for example, the dot d3.
[0063] An example of input of a change instruction e1 in a case where the operation unit 110 has a selection button 111 and a direction button 112 as shown in FIG. 10 will be described.
[0064] In this case, the modification instruction e1 includes a selection instruction for selecting a dot, a direction instruction for specifying a direction to move the dot, and a distance instruction for specifying a distance to move the dot. The selection button 111 is used to input the selection instruction. The direction button 112 is used to input the direction instruction and the distance instruction.
[0065] The selection buttons 111 include a first selection button 111a corresponding to the dot d1, a second selection button 111b corresponding to the dot d2, a third selection button 111c corresponding to the dot d3, and a fourth selection button 111d corresponding to the dot d4.
[0066] Pressing the first selection button 111a is an operation for inputting a selection instruction to select dot d1. Pressing is an example of an operation. Pressing the second selection button 111b is an operation for inputting a selection instruction to select dot d2. Pressing the third selection button 111c is an operation for inputting a selection instruction to select dot d3. Pressing the fourth selection button 111d is an operation for inputting a selection instruction to select dot d4.
[0067] The direction buttons 112 include an up direction button 112a corresponding to the up direction, a down direction button 112b corresponding to the down direction, a right direction button 112c corresponding to the right direction, and a left direction button 112d corresponding to the left direction.
[0068] Pressing the up button 112a is an operation for inputting a directional instruction for specifying an upward movement of the dots. Pressing the down button 112b is an operation for inputting a directional instruction for specifying a downward movement of the dots. Pressing the right button 112c is an operation for inputting a directional instruction for specifying a rightward movement of the dots. Pressing the left button 112d is an operation for inputting a directional instruction for specifying a leftward movement of the dots.
[0069] A single press of each of the up button 112a, the down button 112b, the right button 112c, and the left button 112d is an operation for inputting a distance instruction that specifies a moving distance of five pixels in the liquid crystal light valve 142. Five pixels is an example of the predetermined number of pixels. The predetermined number of pixels is not limited to five pixels, and may be six pixels or more, or may be one pixel or more and less than five pixels.
[0070] In order to move the position of dot d3 projected onto three-dimensional object 500 as shown in FIG. 9 to the right region located to the right of three-dimensional object 500 on display surface 400 as shown in FIG. 7, the user first inputs a selection instruction to select dot d3 into operation unit 110 by pressing third selection button 111c.
[0071] Next, the user presses right button 112c one or more times to input to operation unit 110 a direction instruction specifying movement of the dot to the right and a distance instruction specifying the movement distance of the dot to the right region.
[0072] When the operation unit 110 receives a change instruction e1 including a selection instruction, a direction instruction, and a distance instruction from the user, it provides the change instruction e1 to the processing unit 130.
[0073] Next, the receiving unit 132 of the processing unit 130 receives a change instruction e1 from the operation unit 110 in step S102.
[0074] When the receiving unit 132 receives the change instruction e1, in step S103, the projection control unit 131 determines the positional relationship of the four dots d1 to d4 in the second projected image c2 by changing the positional relationship of the four dots d1 to d4 in the first projected image c1 in accordance with the change instruction e1.
[0075] For example, the projection control unit 131 moves only the coordinate of the dot d3 among the coordinates of the dots d1 to d4 in the first projected image c1 on the liquid crystal light valve 142 by the distance indicated by the distance instruction to the right in the projector coordinate system J1, that is, in the direction parallel to the x-axis in which the x-coordinate value increases. The projection control unit 131 determines the positional relationship of the four dots d1 to d4 in the second projected image c2 by moving the coordinate of the dot d3.
[0076] Next, in step S104, the projection control section 131 causes the projection section 140 to project onto the projection object 300 a second projection image c2 indicating the four dots d1 to d4 having the positional relationship determined in step S103.
[0077] In step S104, the projection control unit 131 first generates image data a2 indicating the second projection image c2. Then, the projection control unit 131 provides the image data a2 to the projection unit 140. Upon receiving the image data a2, the projection unit 140 projects the second projection image c2 indicated by the image data a2 onto the projection target 300.
[0078] When the user confirms that all of the four dots d1 to d4 in the second projected image c2 are projected onto the display surface 400, the user operates the operation unit 110 to input an image capturing instruction to the operation unit 110. Upon receiving the image capturing instruction, the operation unit 110 provides the image capturing instruction to the processing unit 130.
[0079] When the processing unit 130 receives the image capturing instruction, in step S105, the image capturing control unit 133 causes the camera 200 to capture an image of an area including an area in which the second projection image c2 is projected on the projection object 300, thereby causing the camera 200 to generate a captured image f1. The camera 200 provides the processing unit 130 with image capturing data b indicating the captured image f1.
[0080] Next, in step S106, the determination unit 134 specifies the coordinates in the projector coordinate system J1 of the dots d1 to d4 shown in the second projected image c2 and the coordinates in the camera coordinate system J2 of the dots d1 to d4 shown in the captured image f1.
[0081] For example, the determination unit 134 analyzes image data a2 representing the second projected image c2 to determine the coordinates in the projector coordinate system J1 of the dots d1 to d4 shown in the second projected image c2. The determination unit 134 analyzes imaging data b representing the captured image f1 to determine the coordinates in the camera coordinate system J2 of the dots d1 to d4 shown in the captured image f1.
[0082] Next, in step S107, the determination unit 134 determines a projection transformation matrix H based on the coordinates in the projector coordinate system J1 of the dots d1 to d4 shown in the second projected image c2 and the coordinates in the camera coordinate system J2 of the dots d1 to d4 shown in the captured image f1.
[0083] For example, the determination unit 134 generates simultaneous equations with eight unknowns by substituting the coordinates in the projector coordinate system J1 of the dots d1 to d4 shown in the second projected image c2 and the coordinates in the camera coordinate system J2 of the dots d1 to d4 shown in the captured image f1 into the above-mentioned formulas 1 and 2. The determination unit 134 determines eight independent unknown coefficients by solving the simultaneous equations with eight unknowns. Based on the determined eight coefficients, the determination unit 134 determines a projection transformation matrix H that performs projection transformation on the coordinates in the projector coordinate system J1 to the coordinates in the camera coordinate system J2.
[0084] The determination unit 134 uses the above-mentioned formulas 1 and 2 to determine the projective transformation matrix H -1 may be determined.
[0085] For example, when the user inputs an image capturing instruction to the operation unit 110 in a situation where all of the dots d1 to d4 in the first projected image c1 are projected onto the display surface 400, the following operation is executed. Steps S102 to S104 are omitted. In step S106, the determination unit 134 specifies the coordinates in the projector coordinate system J1 of the dots d1 to d4 shown in the first projected image c1 and the coordinates in the camera coordinate system J2 of the dots d1 to d4 shown in the captured image showing the first projected image c1. In step S107, the determination unit 134 determines the projective transformation matrix H based on the coordinates in the projector coordinate system J1 of the dots d1 to d4 shown in the first projected image c1 and the coordinates in the camera coordinate system J2 of the dots d1 to d4 shown in the captured image showing the first projected image c1.
[0086] A7: Summary of the first embodiment In response to the change instruction e1, the projection control unit 131 causes the projection unit 140 to project the second projection image c2, instead of the first projection image c1, onto the projection target 300. The positional relationship of the dots d1 to d4 in the second projection image c2 is different from the positional relationship of the dots d1 to d4 in the first projection image c1. Therefore, even if any of the dots d1 to d4 in the first projection image c1 is not projected onto the display surface 400, there is a possibility that all of the dots d1 to d4 in the second projection image c2 are projected onto the display surface 400. The determination unit 134 determines the projective transformation matrix H based on the coordinates in the projector coordinate system J1 of the dots d1 to d4 shown in the second projection image c2 and the coordinates in the camera coordinate system J2 of the dots d1 to d4 shown in the captured image f1. Therefore, even if any of the dots d1 to d4 in the first projected image c1 is not projected onto the display surface 400, the determination unit 134 can determine a valid projective transformation matrix H.
[0087] When an instruction to change the position of at least one of the four dots d1 to d4 in the first projected image c1 is used as the change instruction e1, the user can give an intuitively recognizable instruction to change the position of a dot that is not on the same plane to the same plane where other dots exist.
[0088] The determination unit 134 determines the positional relationship of the four dots d1 to d4 in the second projected image c2 by changing the positional relationship of the four dots d1 to d4 in the first projected image c1 in response to the change instruction e1. Therefore, the positional relationship of the four dots d1 to d4 in the second projected image c2 can be determined to be the positional relationship intended by the user.
[0089] The receiving unit 132 receives the change instruction e1 via the operation unit 110 to which an instruction is input by the user. Therefore, by inputting the change instruction e1 to the operation unit 110, the user can change the positional relationship of the four dots d1 to d4.
[0090] B: Variation The following are examples of modified aspects of the above-described embodiment. Two or more aspects selected from the following examples may be combined as appropriate to the extent that they are not mutually inconsistent.
[0091] B1: First modified example In the first embodiment, the operation unit 110 is not limited to a configuration having the selection button 111 and the direction button 112, but can be modified as appropriate.
[0092] Instead of the selection button 111, the operation unit 110 may have a selection switch having a function similar to that of the selection button 111. Instead of the direction button 112, the operation unit 110 may have a direction switch having a function similar to that of the direction button 112.
[0093] Operation unit 110 may include a touch panel that realizes selection buttons 111 and direction buttons 112 through a GUI (Graphical User Interface). In this case, physical selection buttons 111 and physical direction buttons 112 may be omitted.
[0094] The operation unit 110 may include a touch panel that does not show either the selection button 111 or the direction button 112. In this case, the touch panel of the operation unit 110 displays the first projected image c1. A user touches positions of dots d1 to d4 to be displayed in the second projected image c2 in the first projected image c1 displayed by the touch panel, thereby inputting a change instruction e1 indicating the positions of the dots d1 to d4 to be displayed in the second projected image c2 to the operation unit 110. The operation unit 110 provides the change instruction e1 to the processing unit 130. The receiving unit 132 receives the change instruction e1 from the operation unit 110.
[0095] When the operation unit 110 includes a touch panel that displays the first projected image c1, the user may drag a dot of the first projected image c1 on the touch panel to a position where the dot is to be displayed in the second projected image c2 by a flick operation on the touch panel, and drop the dot at that position. A change instruction e1 for changing the original position of the dragged dot to the dropped position is input to the operation unit 110 by the user's flick operation. The operation unit 110 provides the change instruction e1 to the processing unit 130. The receiving unit 132 receives the change instruction e1 from the operation unit 110.
[0096] The operation unit 110 may include a microphone and a voice recognition unit. The voice recognition unit is, for example, a processor having a voice recognition function, or a dedicated circuit having a voice recognition function.
[0097] In this case, the user first selects a change dot from among the dots d1 to d4. Then, the user inputs the name of the change dot by voice into the microphone. The names of the dots d1 to d4 are assumed to be shown in the first projected image c1, for example.
[0098] Next, the user inputs into the microphone either up, down, left or right as the change direction by voice. One utterance of either up, down, left or right means a moving distance of a specific number of pixels in the liquid crystal light valve 142. The specific number of pixels is, for example, five pixels. The specific number of pixels is not limited to five pixels, and may be six pixels or more, or may be less than five pixels. The voice recognition unit converts the voice input into the microphone into a character string.
[0099] A change instruction e1 for moving the position of the change dot in the change direction by the number of pixels according to the number of utterances in either the up, down, left, or right direction is input to the operation unit 110 by voice input by the user. The operation unit 110 provides the change instruction e1 to the processing unit 130. The receiving unit 132 receives the change instruction e1 from the operation unit 110.
[0100] The operation unit 110 may include a camera and a gesture recognition unit. The gesture recognition unit is, for example, a processor having a gesture recognition function or a dedicated circuit having gesture recognition.
[0101] In this case, the user first selects a change dot from among dots d1 to d4. Then, the user executes a gesture corresponding to the change dot toward the camera of operation unit 110. It is assumed that gestures corresponding one-to-one to each of dots d1 to d4 are set in advance.
[0102] The camera of the operation unit 110 captures an image of a gesture corresponding to the change dot, thereby generating first gesture imaging data indicating the gesture.
[0103] Next, the user selects either up, down, left, or right as the change direction. Next, the user performs a gesture corresponding to the change direction toward the camera of the operation unit 110. It is assumed that gestures corresponding one-to-one to each of up, down, left, and right are set in advance. One gesture corresponding to the change direction means a moving distance of a preset number of pixels in the liquid crystal light valve 142. The preset number of pixels is, for example, five pixels. The preset number of pixels is not limited to five pixels, and may be six pixels or more, or may be less than five pixels.
[0104] The camera of operation unit 110 captures an image of a gesture corresponding to the change direction, thereby generating second gesture imaging data indicating the gesture.
[0105] The gesture recognition unit identifies, based on the first gesture imaging data, a change dot corresponding to the gesture indicated by the first gesture imaging data, and identifies, based on the second gesture imaging data, a change direction and a movement distance corresponding to the gesture indicated by the second gesture imaging data.
[0106] Therefore, a change instruction e1 for moving the position of the change dot in the change direction by a number of pixels according to the number of gestures corresponding to the change direction is input to the operation unit 110 by a gesture input by the user. The operation unit 110 provides the change instruction e1 to the processing unit 130. The receiving unit 132 receives the change instruction e1 from the operation unit 110.
[0107] According to the first modified example, the receiving unit 132 can receive various change instructions e1.
[0108] B2: Second variant In the first embodiment and the first modified example, the change instruction e1 may be an instruction to select one candidate image as the second projected image c2 from among multiple candidate images showing dots d1 to d4 having a positional relationship different from the positional relationship of the dots d1 to d4 in the first projected image c1.
[0109] The number of dots in the first projected image c1 may be four or more, and the number of dots shown in each of the multiple candidate images is the same as the number of dots in the first projected image c1.
[0110] In the second modified example, a change instruction e1 for selecting one candidate image as the second projected image c2 is input to a component of the operation unit 110, such as an operation button, a touch panel, a microphone, or a camera. The operation unit 110 provides the change instruction e1 to the processing unit 130. The receiving unit 132 receives the change instruction e1 from the operation unit 110.
[0111] When the receiving unit 132 receives an input instruction to select one candidate image as the second projected image c2, the projection control unit 131 provides image data indicating the one candidate image selected as the second projected image c2 to the projection unit 140 as image data a2 indicating the second projected image c2.
[0112] When image data indicating one candidate image selected as the second projected image c2 is stored in the storage unit 120, the projection control unit 131 reads the image data from the storage unit 120. The projection control unit 131 provides the image data read from the storage unit 120 to the projection unit 140. The projection control unit 131 may generate image data indicating one candidate image selected as the second projected image c2 based on the program P. In this case, the projection control unit 131 provides the generated image data to the projection unit 140.
[0113] According to the second modified example, the user can change the first projected image c1 to the second projected image c2 without selecting or moving a dot, thereby reducing the operational burden on the user.
[0114] B3: Third variant In the first embodiment and the first modified example, the determination system 1 may prompt the user to project the dots d1 to d4 on the same plane.
[0115] For example, the projection control unit 131 superimposes a guide image, which prompts the user to project the dots d1 to d4 on the same plane, on the first projected image c1 as an OSD (On Screen Display) image. The guide image is, for example, a display saying "Change the dot positions so that the dots are located on the same plane." The guide image is not limited to the above display and can be changed as appropriate. The guide image is an example of an image that prompts the user to project at least four feature points on the same plane.
[0116] In the case where the projector 100a has the speaker 150 as shown in FIG. 11, the operation control unit 135 may output, from the speaker 150, a voice guide that urges the user to project the dots d1 to d4 on the same plane in a situation where the first projection image c1 is being projected. The voice guide is, for example, a voice saying "Please change the dot positions so that the dots are located on the same plane." The voice guide is not limited to the above voice and can be changed as appropriate. The voice guide is an example of a voice that urges the user to project at least four feature points on the same plane.
[0117] According to the third modified example, it becomes easier for the user to project the dots d1 to d4 onto the same plane.
[0118] B4: Fourth variant In the first embodiment and the first to third modified examples, the determination system 1 may include a projector 100b, a camera 200, and a control device 600, as shown in FIG.
[0119] The projector 100b does not include the operation unit 110, the storage unit 120, and the processing unit 130, but includes the projection unit 140. The control device 600 is, for example, a PC (Personal Computer), a tablet, or a smartphone. The control device 600 includes the operation unit 110, the storage unit 120, and the processing unit 130. The camera 200 may be an object different from either the projector 100b or the control device 600, or may be mounted on either the projector 100b or the control device 600. The control device 600 provides image data a to the projector 100b, and therefore may also be referred to as an image supply device.
[0120] According to the fourth modified example, it is possible to reduce the processing load on the projector compared to the first embodiment and the first to third modified examples.
[0121] B5: Fifth variant In the first embodiment and the first to fourth modified examples, the liquid crystal light valve 142 is used as an example of the light modulation device, but the light modulation device is not limited to the liquid crystal light valve and can be appropriately changed. For example, the light modulation device may be configured in a manner using one digital mirror device. Also, other than the liquid crystal panel and the DMD, any configuration capable of modulating the light emitted by the light source 141 can be adopted as the light modulation device. [Explanation of symbols]
[0122] 1...Decision system, 100, 100a, 100b...projector, 110...operation unit, 120...memory unit, 130...processing unit, 131...projection control unit, 132...receiving unit, 133...imaging control unit, 134...determination unit, 135...operation control unit, 140...projection unit, 141...light source, 142...liquid crystal light valve, 143...projection lens, 150...speaker, 200...camera, 210...imaging lens, 220...image sensor, 300...projected object, 400...display surface, 500...three-dimensional object, 600...control device.
Claims
1. A method for determining a homography matrix, comprising the steps of: receiving a change instruction to change a positional relationship between the at least one feature point and the other feature points when at least one of the four feature points is projected out of the same plane in a situation where a first projection image indicating four feature points is projected by a projector onto a first area; causing the projector to project a second projection image, which indicates the four feature points after the change, onto the first area in response to the change instruction; determining a projection transformation matrix that associates a projector coordinate system that represents coordinates in the projection image projected by the projector with a camera coordinate system that represents coordinates in the captured image generated by the camera, based on coordinates of the four feature points in the second projection image and coordinates of the four feature points in a first captured image generated by a camera capturing an image of the first area onto which the second projection image is projected; Including, a positional relationship of the four feature points after the change in the second projection image is different from a positional relationship of the four feature points in the first projection image; How to decide.
2. the modification instruction is an instruction to modify a position of one of the four feature points in the first projected image; The method of claim 1 .
3. receiving the change instruction to change a positional relationship of the four feature points in the first projected image from a first positional relationship to a second positional relationship, and setting the positional relationship of the four feature points in the second projected image to the second positional relationship; The method of claim 2 , further comprising:
4. the change instruction is an instruction to select, as the second projection image, one candidate image from among a plurality of candidate images showing the four feature points having a positional relationship different from a positional relationship of the four feature points in the first projection image; The method of claim 1 .
5. receiving the change instruction via an instruction input unit into which an instruction is input by a user; The method according to any one of claims 1 to 4.
6. Prompting the four feature points to be projected onto the same plane; The method according to claim 1 , further comprising:
7. The first projected image further indicates an image that prompts the four feature points to be projected on the same plane. The method of claim 6.
8. prompting the user with audio that the four feature points are projected onto the same plane; The method of claim 6.
9. A projection unit that projects an image; a receiving unit that receives a change instruction for changing a positional relationship between the at least one feature point and another feature point when at least one of the four feature points is projected out of a same plane in a situation where the projection unit projects a first projection image indicating four feature points onto a first area; a projection control unit that, in response to the change instruction, causes the projection unit to project a second projection image indicating the four feature points after the change onto the first area; a determination unit that determines a projection transformation matrix that mutually corresponds a projection coordinate system that represents coordinates in the projection image projected by the projection unit and a camera coordinate system that represents coordinates in the captured image generated by the camera, based on coordinates of the four feature points in the second projection image and coordinates of the four feature points in a first captured image generated by a camera capturing an image of the first area onto which the second projection image is projected; Including, a positional relationship of the four feature points after the change in the second projection image is different from a positional relationship of the four feature points in the first projection image; projector.
10. A projector that projects an image, a camera that captures an image, and a control device that controls the projector and the camera, The control device includes: a receiving unit that receives a change instruction for changing a positional relationship between the at least one feature point and another feature point when at least one of the four feature points is projected out of a same plane in a situation where the projector projects a first projection image indicating four feature points onto a first area; a projection control unit that causes the projector to project a second projection image indicating the four feature points after the change onto the first area in response to the change instruction; a determination unit that determines a projection transformation matrix that mutually corresponds a projector coordinate system that represents coordinates in a projection image projected by the projector and a camera coordinate system that represents coordinates in a captured image generated by the camera, based on coordinates of the four feature points in the second projection image and coordinates of the four feature points in a first captured image generated by the camera capturing an image of the first area onto which the second projection image is projected; a positional relationship of the four feature points after the change in the second projection image is different from a positional relationship of the four feature points in the first projection image; Decision system.
Citation Information
Patent Citations
Optical projection control apparatus, optical projection control method, and program
JP2012249009A
Parameter generation apparatus, control method therefor, and program
JP2019125955A
Projection control device, projection device, projection control method and program
JP2019168546A
Projection control device, projection control method, and program
JP2020178221A
