Calculation method, program, and information processing device

Abstract

To improve calculation accuracy when a normal vector of a projection surface is calculated based on a captured image of a pattern projected on a planar projection surface from a projector.SOLUTION: A calculation method is provided that includes calculation processing SA150 that estimates a normal vector of a projection surface by calculating, for each captured image, a transformation matrix that associates a first corresponding point with a second corresponding point using a first corresponding point set in advance in a pattern projected on a projection surface by a projector, a second corresponding point corresponding to the first corresponding point in each of two or more captured images obtained by imaging the projection surface on which the pattern is projected by a camera from two or more positions, an estimation value of an internal parameter of the camera, and an internal parameter of the projector, and comparing the normal vector of the projection surface calculated from each of the transformation matrix calculated for each captured image.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to a calculation method, a program, and an information processing device. [Background technology]

[0002] An example of an image display device is a projector that displays an image by projecting the image onto a projection surface such as a projection screen. Depending on the positional relationship between the projection surface and the image display device, distortions such as trapezoidal distortion may occur in the image displayed on an object by the image display device. Various technologies for correcting this distortion have been proposed, one example of which is the technology disclosed in Patent Document 1. Patent Document 1 discloses a projector that calculates a normal vector of the projection surface based on a transformation matrix that performs projective transformation of an image in an optical modulator onto an image captured by an imaging unit, and performs distortion correction based on this normal vector. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-187515 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the technique disclosed in Patent Document 1, there are cases where a plurality of normal vectors are calculated as solutions from one transformation matrix, and the accuracy of calculating normal vectors is insufficient. [Means for solving the problem]

[0005] A calculation method according to one embodiment of the present disclosure includes projecting a pattern detectable by a camera from a projector onto a planar projection surface; calculating, based on a first captured image obtained by capturing an image of the pattern with the camera from a first position, a first transformation matrix for converting the coordinate system of the first captured image from one side to another of the projector's coordinate system; calculating, based on a second captured image obtained by capturing an image of the pattern with the camera from a second position different from the first position, a second transformation matrix for converting the coordinate system of the second captured image and the projector from one side to another of the coordinate system; and estimating the normal vector of the projection surface based on a first normal vector of the projection surface calculated based on the first transformation matrix, a second normal vector of the projection surface calculated based on the second transformation matrix, internal parameters of the camera, and internal parameters of the projector.

[0006] A program according to one embodiment of the present disclosure causes a computer to perform the following operations: project a pattern detectable by a camera from a projector onto a planar projection surface onto which an image is projected from the projector; detect the pattern from each of two or more captured images obtained by capturing images of the projection surface with the camera from two or more positions; calculate a transformation matrix that associates the first corresponding points with the second corresponding points using first corresponding points that are preset in the pattern, second corresponding points that correspond to the first corresponding points in each of the two or more captured images, internal parameters of the camera, and internal parameters of the projector; and calculate a normal vector of the projection surface based on the transformation matrix.

[0007] The information processing device disclosed herein includes a communication device that communicates with a projector that projects an image onto a planar projection surface, a camera that captures the image projected onto the projection surface from the projector, and a processing device, wherein the processing device performs the following operations: projecting a pattern that can be detected by the camera onto the projection surface from the projector; detecting the pattern from each of two or more captured images obtained by capturing images of the projection surface with the camera from two or more positions; calculating a transformation matrix that associates the first corresponding points with the second corresponding points using first corresponding points that are preset in the pattern, second corresponding points that correspond to the first corresponding points in each of the two or more captured images, internal parameters of the camera, and internal parameters of the projector; and calculating a normal vector of the projection surface based on the transformation matrix. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of the configuration of a display system 1 including an information processing device 10A according to a first embodiment of the present disclosure. [Figure 2] 10 is a diagram showing an example of an adjustment pattern projected from the projector 20 onto the projection surface SC. FIG. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of an information processing device 10A. [Figure 4] FIG. 10 is a diagram showing an example of a UI screen G20. [Figure 5] 10 is a flowchart showing the flow of processing in a calculation method executed by the processing device 110 in accordance with the program PRA. [Figure 6] FIG. 10 is a diagram for explaining the processing content of calculation processing SA150. [Figure 7] FIG. 10 is an explanatory diagram of a homography decomposition process. [Figure 8] FIG. 10 is a diagram illustrating an example of calculation of a normal vector when there is no error in the internal parameters of the camera. [Figure 9] FIG. 10 is a diagram illustrating an example of calculation of a normal vector when there is an error in the internal parameters of the camera. [Figure 10]FIG. 10 is a diagram illustrating an example of the configuration of an information processing device 10B according to a second embodiment of the present disclosure. [Figure 11] 10 is a flowchart showing the flow of processing in a calculation method executed by the processing unit 110 in accordance with the program PRB. [Figure 12] 10 is a diagram for explaining the processing content of the calculation processing SB150. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0009] The embodiments described below are subject to various technically preferable limitations, but the embodiments of the present disclosure are not limited to the following embodiments. 1. Embodiment 1 is a diagram showing an example configuration of a display system 1 including an information processing device 10A according to a first embodiment of the present disclosure. In addition to the information processing device 10A, the display system 1 includes a projector 20 that communicates with the information processing device 10A via a network NW. Specific examples of the network NW include a wired or wireless LAN (Local Area Network), a wired or wireless USB, or Bluetooth (registered trademark).

[0010] The projector 20 projects a projection image GA onto a projection surface SC. In this embodiment, the projection surface SC is, for example, one of the interior walls of a room in which a user of the display system 1 resides. In FIG. 1, the arrow y represents the vertical direction, the arrow x represents one of two directions perpendicular to the vertical direction (hereinafter referred to as the horizontal direction), and the dot z represents the other of the two directions perpendicular to the vertical direction (hereinafter referred to as the depth direction). The projector 20 includes an optical system including a liquid crystal panel or other optical modulator that generates image light according to input image data and a projection lens that guides the image light generated by the optical modulator to the projection surface SC. The projector 20 also includes an image processing circuit that corrects distortion of the projected image according to a geometric correction value provided by the information processing device 10A. The optical modulator, optical system, and image processing circuit are not shown in FIG. 1.

[0011] The interior wall surface of a room is not flat like a projection screen, and distortion may occur in the projected image GA displayed on the projection surface SC by the projector 20 due to depressions or protrusions in the projection surface SC. Furthermore, when the projection lens of the projector 20 is not directly facing the wall surface, distortion such as a trapezoidal shape may occur in the projected image GA displayed on the wall surface by the projector 20. In this embodiment, as in the conventional case, an adjustment pattern can be used to correct distortion in the projected image GA displayed on the projection surface SC. Although details will be described later, in this embodiment, a chessboard pattern shown in FIG. 2 is used to adjust the outer shape of the projected image GA displayed on the projection surface SC to a rectangle.

[0012] The information processing device 10A is, for example, a smartphone. In this embodiment, the information processing device 10A is a smartphone, but it may also be a tablet terminal. In short, the information processing device 10A may be a portable device that can be carried by a user and moved relative to the projector 20 and the projection surface SC. FIG. 3 is a diagram showing an example configuration of the information processing device 10A. As shown in FIG. 3, the information processing device 10A includes a processing device 110, a communication device 120, a camera 130, a display 140, an input device 150, a sensor 160, and a storage device 170. In the information processing device 10A, the communication device 120, the camera 130, the display 140, the input device 150, the sensor 160, and the storage device 170 are each connected to the processing device 110 via a bus.

[0013] The processing device 110 is one or more processors. The processing device 110 is, for example, a CPU (Central Processing Unit). The processing device 110 operates in accordance with a program PRA stored in the storage device 170, thereby functioning as the control center of the information processing device 10A.

[0014] The communication device 120 is a device that performs wireless communication with other devices, and includes, for example, an interface circuit. A specific example of another device that communicates with the communication device 120 is the projector 20.

[0015] The camera 130 is, for example, a video camera. The camera 130 includes an imaging lens (not shown) and an imaging element (not shown) on which light from the imaging lens forms an image. The imaging element is, for example, a CMOS (Complementary Metal-Oxide-Semiconductor) sensor, but is not particularly limited to this. The camera 130 is a device for capturing a projection image projected onto an object from the projector 20. The camera 130 captures images within its imaging field of view as needed, and transfers image data representing the captured image to the processing device 110.

[0016] The display 140 includes a panel display such as a liquid crystal display, a plasma display, or an organic EL display, and a drive circuit for the panel display. The display 140 displays various images under the control of the processing device 110. The input device 150 is a transparent pressure-sensitive sensor provided to cover the display area of ​​the display 140. The input device 150 may include multiple controls. The input device 150 receives a user's operation and outputs operation content data indicating the received operation to the processing device 110. In this way, the user's operation on the input device 150 is transmitted to the processing device 110.

[0017] The sensor 160 is, for example, a three-axis acceleration sensor. The sensor 160 detects acceleration occurring in the information processing device 10A in each of three mutually orthogonal axial directions, and outputs acceleration data representing the acceleration in each axial direction to the processing device 110. By analyzing this acceleration data, it is possible to detect the direction of gravitational acceleration relative to the information processing device 10A, i.e., the vertical direction.

[0018] The storage device 170 is a recording medium readable by the processing device 110. The storage device 170 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM). The volatile memory is, for example, a random access memory (RAM). Various programs are stored in the nonvolatile memory. Examples of the various programs stored in the nonvolatile memory include a kernel program and a program PRA. The kernel program is not shown in FIG. 3. The kernel program is a program that causes the processing device 110 to implement an operating system (OS). When the information processing device 10A is powered on, the processing device 110 reads the kernel program from the nonvolatile memory to the volatile memory and begins executing the read kernel program. When an instruction to start executing another program is received via an operation on the input device 150, the processing device 110, operating according to the kernel program, begins executing the other program.

[0019] For example, when an instruction to start execution of program PRA is given by operating input device 150, processing device 110 reads program PRA from non-volatile memory to volatile memory and starts execution of program PRA read to volatile memory. Processing device 110 operating in accordance with program PRA functions as first control unit 111, second control unit 112, third control unit 113, and calculation unit 114A shown in FIG. 3. In other words, first control unit 111, second control unit 112, third control unit 113, and calculation unit 114A shown in FIG. 3 are each software modules realized by operating processing device 110 in accordance with program PRA. The roles of first control unit 111, second control unit 112, third control unit 113, and calculation unit 114A shown in FIG. 3 are as follows.

[0020] The first control unit 111 displays on the display 140 an image of a UI screen that prompts the user to execute the calculation method of the present disclosure. A specific example of a UI screen that is displayed on the display 140 under the control of the first control unit 111 is a UI screen that prompts the user to use the camera 130 to capture images of the projection surface SC on which an adjustment pattern is projected from the projector 20, from two or more different positions. For example, the UI screen prompts the user to capture an image of the adjustment pattern with the camera 130 when the camera 130 is located at a first position, and to capture an image of the adjustment pattern with the camera 130 when the camera 130 is located at a second position different from the first position. The captured image obtained when the camera 130 is located at the first position is an example of a first captured image, and the captured image obtained when the camera 130 is located at the second position is an example of a second captured image. Each of the two or more positions is the position of the camera 130 relative to the projector 20, which serves as a reference object.

[0021] 4 is a diagram showing an example of a UI screen G20 in this embodiment. The UI screen G20 includes an image G21 of text that describes how to adjust the contour of the projection image, an image G22 that schematically illustrates capturing an adjustment pattern using the information processing device 10A, and an image G23 of a button that instructs the user to perform the adjustment. While the UI screen G20 is displayed on the display 140, the user can select the image G23 by operating the input device 150. When the image G23 is selected by operating the input device 150, the second control unit 112 (described later) controls the projector 20 to project the adjustment pattern, and the third control unit 113 controls the camera 130 to capture the adjustment pattern.

[0022] The second control unit 112 communicates with the projector 20 via the communication device 120, thereby causing the projector 20 to project the adjustment pattern described above.

[0023] The third control unit 113 controls the camera 130 to capture an image including the adjustment pattern projected onto the projection surface SC at a regular interval, such as every few milliseconds. The user is prompted by the image G21 to capture the adjustment pattern while moving left and right. As the user moves left and right while holding the camera 130 toward the projection surface SC, the camera 130 captures images of the adjustment pattern displayed on the projection surface SC at multiple different imaging positions, and multiple captured images can be captured at each imaging position. The multiple imaging positions include a first position and a second position.

[0024] The calculation unit 114A analyzes each of the multiple captured images captured by the camera 130 to detect an adjustment pattern from each captured image. Next, the calculation unit 114A estimates a normal vector of the projection surface SC based on each adjustment pattern detected from the multiple captured images, and performs a calculation process to calculate a geometric correction value for correcting the shape of the projection image to a rectangle based on the estimated normal vector. The normal vector of the projection surface SC is a vector in the depth direction z, i.e., a direction perpendicular to the projection surface SC. The details of this calculation process will be made clear later.

[0025] When a user viewing the UI screen instructs the processor 110 to execute distortion correction, the processor 110, operating in accordance with the program PRA, executes a calculation method embodying the features of the present disclosure. Figure 9 is a flowchart showing the flow of processing in this calculation method. As shown in Figure 9, this control method includes a projection process SA110, an imaging process SA120, a detection process SA130, a determination process SA140, and a calculation process SA150.

[0026] In the projection process SA110, the processing device 110 functions as a second control unit. In the projection process SA110, the processing device 110 causes the projector 20 to project the adjustment pattern described above. In the imaging process SA120, the processing device 110 functions as a third control unit 113. In the imaging process SA120, the processing device 110 causes the camera 130 to capture an image of the adjustment pattern projected onto the projection surface SC. While the imaging process SA120 is being performed, if a user holding the information processing device 10A moves left and right relative to the projection surface SC while pointing the camera 130 of the information processing device 10A toward the projection surface SC, a plurality of captured images are captured by the camera 130 at each of a plurality of different imaging positions.

[0027] In the detection process SA130, the determination process SA140, and the calculation process SA150, the processing device 110 functions as a calculation unit 114A. In the detection process SA130, the processing device 110 analyzes each of a plurality of captured images captured by the camera 130 to detect an adjustment pattern from each captured image. More specifically, in the detection process SA140, the processing device 110 detects a plurality of first corresponding points, which are predetermined points in the adjustment pattern drawn on the optical modulator, for each captured image. Note that the first corresponding points may be natural feature points.

[0028] In determination process SA140, the processing device 110 determines whether or not an adjustment pattern was detected in two or more captured images in detection process SA130. If an adjustment pattern was detected in two or more captured images in detection process SA130, the determination result of determination process SA140 is "Yes." On the other hand, if the number of captured images in which an adjustment pattern was detected in detection process SA130 is less than two, the determination result of determination process SA140 is "No." If the determination result of determination process SA140 is "No," the processing device 110 re-executes the processes from projection process SA110 onwards. On the other hand, if the determination result of determination process SA140 is "Yes," the processing device 110 executes calculation process SA150 and ends this calculation method.

[0029] In the calculation process SA150, the processing device 110 functions as a calculation unit 114A. FIG. 6 is a diagram for explaining the processing content of the calculation process SA150. The rectangles in FIG. 6 represent processes executed by the calculation unit 114A, and the rectangles with rounded corners represent data that is input or output in each process. As shown in FIG. 6, the calculation process SA150 includes normalization process SA151, normalization process SA152, projective transformation calculation process SA153, homography decomposition process SA154, discrimination process SA155, estimation process SA156, acquisition process SA157, and geometric correction value calculation process SA158. Note that in the calculation process SA150, processes that are not necessary for estimating the normal vector of the projection surface SC, such as the acquisition process SA157 and the geometric correction value calculation process SA158, may be omitted.

[0030] In the normalization process SA151, the calculation unit 114A converts the coordinates of the first corresponding point in the projector coordinate system, which indicates the position on the projection image projected from the projector 20, into normalized coordinates that do not include components corresponding to the internal parameters, using the internal parameters of the projector 20. The projector coordinate system can also be said to be a two-dimensional coordinate system for expressing the coordinates of each pixel of the light modulator. The projector coordinate system is an example of a projector's coordinate system. Therefore, the coordinates of the first corresponding point are the coordinates of the pixel in the light modulator. The internal parameters of the projector are data that represent the angle of view, lens center, and lens distortion components of the projection lens of the projector, and specifically, the focal length f of the projection lens, the lens center (C u ,C v ), and lens coefficients k1 and k2. The calculation unit 114A may acquire the internal parameters from the projector 20 by communicating with the projector 20, or may acquire the internal parameters that the user inputs by operating the input device 150 from values ​​published in a catalog or the like.

[0031] The coordinates of the corresponding points in the projector coordinate system (u n ,v n ) and the normalized coordinates of the corresponding points (x n , y n) have the relationship shown in the following Expressions 1 and 2. The calculation unit 114A converts the coordinates of the first corresponding point in the projector coordinate system indicating the position on the image projected by the projector 20 into normalized coordinates using the following Expressions 1 and 2.

number

number

[0032] In normalization processing SA152, calculation unit 114A converts the coordinates of the second corresponding point in a camera coordinate system representing a position on an image captured by camera 130 into normalized coordinates that do not include components corresponding to the internal parameters, using the internal parameters of camera 130. The camera coordinate system can also be considered a two-dimensional coordinate system representing the coordinates of each pixel of the image sensor, i.e., the coordinates of each pixel in the captured image. Therefore, the coordinates of the second corresponding point are the coordinates of a pixel in the captured image. The camera coordinate system representing a position on an image captured by camera 130 positioned at a first position is an example of a coordinate system for the first captured image, and the camera coordinate system representing a position on an image captured by camera 130 positioned at a second position is an example of a coordinate system for the second captured image. The internal parameters of the camera are data representing the angle of view, lens center, and lens distortion components of the imaging lens of the camera. Specifically, these parameters include the focal length, lens center, and lens coefficient of the imaging lens. In this embodiment, information processing device 10A is a smartphone, and it is common for a user to not know the internal parameters of the camera of the smartphone, even if the smartphone is his or her own. Therefore, calculation unit 114A calculates the normalized coordinates of the second corresponding point by performing calculations using the above-mentioned Equations 1 and 2 using estimated values ​​of the internal parameters of camera 130 (for example, the internal parameters of the camera provided in a typical smartphone).

[0033] In the projective transformation calculation process SA153, the calculation unit 114A calculates a projective transformation matrix, which is a transformation matrix that transforms the normalized coordinates of the first corresponding point into the normalized coordinates of the second corresponding point, based on the normalized coordinates of the first corresponding point and the normalized coordinates of the second corresponding point. That is, the calculation unit 114A executes the following operations: calculating a first transformation matrix for transforming from one of the coordinate system of the first captured image and the coordinate system of the projector to the other, based on a first captured image obtained by capturing an image of the adjustment pattern with the camera 130 from a first position; and calculating a second transformation matrix for transforming from one of the coordinate system of the second captured image and the coordinate system of the projector to the other, based on a second captured image obtained by capturing an image of the adjustment pattern with the camera 130 from a second position different from the first position. The projective transformation matrix is ​​an example of the first transformation matrix or the second transformation matrix. An existing algorithm may be used as appropriate to calculate the projective transformation matrix.

[0034] In the homography decomposition process SA154, the calculation unit 114A performs homography decomposition shown in FIG. 7 on each of a plurality of homography matrices calculated based on each of a plurality of captured images, i.e., the first transformation matrix and the second transformation matrix, to generate a data set including a normal vector of the projection surface SC and data indicating the relative position of the camera 130 with respect to the projection surface SC (extrinsic parameters of the camera 130). The extrinsic parameters of the camera 130 refer to the translation matrix and rotation matrix of the camera 130. As shown in FIG. 7, two sets of data sets are obtained from one homography matrix. In other words, if the number of captured images in which adjustment patterns have been successfully detected is N (N is an integer equal to or greater than 2), the calculation unit 114A calculates 2N sets of data sets of the normal vector of the projection surface SC and the extrinsic parameters of the camera 130 from these N captured images.

[0035] In this embodiment, the projection surface SC is stationary or nearly stationary relative to the projector 20, so the direction and magnitude of the normal vector of the projection surface SC are nearly constant. Therefore, if the internal parameters of the camera 130 are truly correct, as shown in FIG. 8 , the 2N data sets include N data sets whose normal vectors match, and these normal vectors correctly represent the normal direction of the projection surface SC. In FIG. 8 , the data sets whose foreground and background colors are inverted are data sets whose normal vectors match. However, in this embodiment, the internal parameters of the camera 130 are estimated values ​​and contain errors relative to the true values. Therefore, as shown in FIG. 9 , the 2N data sets include N data sets whose normal vectors are similar to each other, but these normal vectors do not completely match. In FIG. 9 , the hatched data sets are data sets whose normal vectors are similar to each other. Therefore, in this embodiment, the calculation unit 114A performs a discrimination process SA155 on the 2N data sets to select N data sets that are estimated to include normal vectors that are approximate to the correct normal vector. More specifically, in the discrimination process SA155, the calculation unit 114A performs, for example, dynamic programming on the 2N data sets to compare the normal vectors included in each of the two data sets calculated from one captured image between the captured images, and selects a path that minimizes the sum of the differences in normal vectors between the captured images. Data sets with a gray background in FIG. 6 represent data sets that were not selected in the discrimination process SA155.

[0036] In the estimation process SA156, the calculation unit 114A estimates the normal vector of the projection surface SC as the average of N normal vectors in the data set on the path. The average of the N normal vectors is, for example, an arithmetic average for each component. That is, in the estimation process SA156, the calculation unit 114A estimates the normal vector of the projection surface SC by comparing multiple normal vectors calculated based on each of the multiple captured images. In other words, in the estimation process SA156, the calculation unit 114A estimates the normal vector of the projection surface SC based on a first normal vector of the projection surface SC calculated based on a first transformation matrix and a second normal vector of the projection surface SC calculated based on a second transformation matrix.

[0037] In acquisition processing SA157, calculation unit 114A acquires a vertical vector representing the vertical direction based on the output data of sensor 160. In geometric correction value calculation processing SA158, calculation unit 114A calculates the cross product (vector product) of the vertical vector and the normal vector estimated in estimation processing SA156 to calculate a horizontal vector that is orthogonal to the normal vector and also orthogonal to the vertical vector. The horizontal vector and the vertical vector define a screen coordinate system, which is a two-dimensional coordinate system that defines positions on projection surface SC when viewed from the front. Next, calculation unit 114A calculates a mutual conversion matrix between the projector coordinate system and the screen coordinate system. Calculation unit 114A determines the coordinates of the four corners of the corrected projection image so that it forms a rectangle on projection surface SC, and converts these coordinates into coordinates in the projector coordinate system using the mutual conversion matrix. Then, the calculation unit 114A calculates a geometric correction value for distortion correction from the coordinates of the four corners in the projector coordinate system and the coordinates of the four corners of the projection image. By calculating the geometric correction value taking the vertical vector into consideration, a projection image that is straight with respect to the installation surface (e.g., the ground) of the projection target having the projection surface SC can be obtained.

[0038] The calculation unit 114A transmits the geometric correction value calculated in the above manner to the projector 20 using the communication device 120, and causes the projector 20 to execute a process of performing correction according to the geometric correction value and projecting the projection image GA.

[0039] According to this embodiment, even when multiple candidate vectors that are candidates for the normal vector are calculated from one captured image, it is possible to select the correct candidate vector. Furthermore, according to this embodiment, even when the internal parameters of camera 130 cannot be known in advance, it is possible to accurately estimate the normal vector of projection surface SC, improving the accuracy of geometric correction that corrects the outer shape of projection image GA viewed from a direction along the normal vector, i.e., the outer shape of projection image GA when viewed from the front of projection surface SC, to a rectangle.

[0040] 2. Second embodiment FIG. 10 is a diagram illustrating an example configuration of an information processing device 10B according to a second embodiment of the present disclosure. In FIG. 10, the same components as those in FIG. 3 are denoted by the same reference numerals. As is clear from a comparison of FIG. 10 with FIG. 3, the hardware configuration of the information processing device 10B is the same as the hardware configuration of the information processing device 10A. That is, the information processing device 10B includes a processing device 110, a communication device 120, a camera 130, a display 140, an input device 150, a sensor 160, and a storage device 170. The difference between the configuration of the information processing device 10B and the configuration of the information processing device 10A is that a program PRB is stored in the storage device 170 instead of the program PRA.

[0041] The processing device 110 operating in accordance with the program PRB functions as the first control unit 111, the second control unit 112, the third control unit 113, and the calculation unit 114B shown in FIG. 10. The processing device 110 operating in accordance with the program PRB executes a calculation method whose processing flow is shown in the flowchart of FIG. 11. In FIG. 11, the same processes as those in FIG. 5 are also denoted by the same reference numerals. As is clear from a comparison of FIG. 11 with FIG. 5, the calculation method in this embodiment differs from the calculation method in the first embodiment in that it includes a calculation process SB150 instead of a calculation process SA150. The following description will focus on the calculation process SB150 and the calculation unit 114B, which are differences from the first embodiment.

[0042] In calculation processing SB150, processing device 110 functions as calculation unit 114B. Calculation unit 114B is the same as calculation unit 114A in that it calculates the aforementioned transformation matrix based on an image of the adjustment pattern captured by camera 130, and calculates a normal vector of projection surface SC based on the transformation matrix. Calculation unit 114B differs from calculation unit 114A in that it repeatedly estimates internal parameters of the camera and updates the normal vector based on the estimation results so as to minimize the reprojection error calculated from the transformation matrix calculated from each of two or more captured images.

[0043] The reprojection error refers to the difference between the actually observed point cloud coordinates and the reproduced point cloud coordinates. In this embodiment, the "actually observed point cloud coordinates" refer to the coordinates of the second corresponding points in the captured image. In this embodiment, the "reproduced point cloud coordinates" refer to the coordinates in the screen coordinate system obtained by converting the coordinates of each of the first corresponding points in the projector coordinate system using the normalization and mutual conversion matrix described above, and then converting the resulting coordinates into the camera coordinate system. When the internal parameters, external parameters, and surface normal vector of the projection surface SC are correct, the reprojection error is zero. Therefore, the reprojection error serves as an indicator of the accuracy of the estimation of the internal parameters, external parameters, and surface normal vector of the projection surface SC.

[0044] FIG. 12 is a diagram for explaining the processing content of the calculation processing SB150. In FIG. 12, the same processes as those in FIG. 6 are assigned the same reference numerals. As is clear from comparing FIG. 12 with FIG. 6, the calculation processing SB150 includes the above-mentioned calculation processing SA150, normalization processing SA151, normalization processing SA152, projective transformation calculation processing SA153, homography decomposition processing SA154, discrimination processing SA155, and estimation processing SA156. Although not shown in detail in FIG. 12, the calculation processing SB150 also includes the processes of acquisition processing SA157 and geometric correction value calculation processing SA158. The calculation processing SB150 differs from the calculation processing SA150 in that it includes reprojection error calculation processing SB151, discrimination processing SB152, and update processing SB153.

[0045] In the reprojection error calculation process SB151, the calculation unit 114B calculates a reprojection error using a transformation matrix calculated from each of two or more captured images, the internal parameters (estimated values) of the camera 130, and the internal parameters of the projector 20. In the determination process SB152, the calculation unit 114B determines whether the reprojection error calculated in the reprojection error calculation process SB151 has been minimized. In this embodiment, if the reprojection error is less than a predetermined threshold, the calculation unit 114B determines that the reprojection error has been minimized, and the determination result of the determination process SB152 is "Yes." This threshold may be set to a suitable value through experiments or the like. If the reprojection error is equal to or greater than the predetermined threshold, the determination result of the determination process SB152 is "No."

[0046] If the determination result of the determination process SB152 is "No," the calculation unit 114B executes an update process SB153 that updates the internal parameters (estimated values) of the camera 130, and then executes the normalization process SA152 and subsequent processes again. Here, specific examples of how the internal parameters (estimated values) of the camera 130 can be updated include, for example, a brute-force update in which the focal length f is sequentially increased from the initial value (1000) to 1010, 1020, and so on (or sequentially decreased to 990, 980, and so on), or an update by nonlinear optimization that adjusts the amount of increase or decrease depending on the magnitude and sign of the difference between the threshold and the reprojection error. On the other hand, if the determination result of the determination process SB152 is "Yes," the calculation unit 114B executes the aforementioned acquisition process SA157 and geometric correction value calculation process SA158, thereby terminating the execution of this calculation method.

[0047] As described above, according to this embodiment, even if the internal parameters of camera 130 are unknown, the internal parameters of camera 130 are estimated and the normal vector is updated based on the estimation results repeatedly so as to minimize the reprojection error calculated from multiple projective transformation matrices calculated from each of the captured images. Therefore, according to this embodiment, the normal vector of projection surface SC can be estimated more accurately than in the first embodiment, and geometric correction can be performed more accurately than in the first embodiment to correct the outer shape of projection image GA viewed from a direction along the normal vector, i.e., the outer shape of projection image GA when viewed from the front of projection surface SC, to a rectangle. Additionally, according to this embodiment, it is also possible to estimate the internal parameters of camera 130.

[0048] 3. Variations The above embodiments can be modified as follows. (1) In the first embodiment described above, the processing device 110 that executes the calculation method that prominently exhibits the features of the present disclosure is included in an information processing device 10A that is separate from the projector 20, but it may also be included in the projector 20. Specifically, the calculation method of the present disclosure may be executed by a computer that functions as the control center of the projector 20. Similarly, the calculation method in the second embodiment may also be executed by a computer that functions as the control center of the projector 20.

[0049] (2) In the above embodiments, the projective transformation matrix and the normal vector of the projection surface SC are calculated using N captured images in which the adjustment pattern was successfully detected. However, two or more captured images in which the distance between the captured positions is equal to or greater than a threshold may be selected from the N captured images using external parameters of the camera 130 calculated along with the normal vector, and the normal vector may be calculated using the transformation matrix corresponding to each of the selected two or more captured images. This is because the greater the distance between captured positions used, the more accurate the calculation of the normal vector. Note that the threshold may be determined according to the distance between the projection surface SC and the projector 20, and may be set to a value such as an integer multiple of the distance.

[0050] (3) In the first embodiment, the first control unit 111, the second control unit 112, the third control unit 113, and the calculation unit 114A were all software modules. However, any one, any two, any three, or all of the first control unit 111, the second control unit 112, the third control unit 113, and the calculation unit 114A may be hardware modules such as an ASIC (Application Specific Integrated Circuit). Even if any one, any two, any three, or all of the first control unit 111, the second control unit 112, the third control unit 113, and the calculation unit 114A are hardware modules, the same effects as those of the first embodiment can be achieved. Similarly, the calculation unit 114B in the second embodiment may be a hardware module.

[0051] (4) The program PRA may be manufactured as a standalone program or provided free of charge or for a fee. Specific examples of providing the program PRA include providing the program PRA by writing it to a computer-readable non-transitory recording medium such as a flash ROM, or providing the program PRA by downloading it via a telecommunications line such as the Internet. By operating a general computer in accordance with the program PRA provided in these ways, it becomes possible to cause the computer to execute the calculation method of the first embodiment. Similarly, the program PRB may be manufactured as a standalone program or provided free of charge or for a fee.

[0052] (5) In the first embodiment, the calculation process SA150 includes the normalization processes SA151 and SA152. However, the calculation process SA150 does not have to include the normalization processes SA151 and SA152. That is, in the projective transformation calculation process SA153, the calculation unit 114A may calculate the projective transformation matrix without using the internal parameters of the projector 20 and the internal parameters of the camera 130. Therefore, in the projective transformation calculation process SA153, the calculation unit 114A may calculate the projective transformation matrix based on the coordinates of the unnormalized first corresponding point and the coordinates of the unnormalized second corresponding point. In this case, the internal parameters of the projector 20 and the internal parameters of the camera 130 are used in the homography decomposition process SA154. In other words, the internal parameters of the projector 20 and the internal parameters of the camera 130 do not need to be used in the normalization process SA151 and the normalization process SA152, but may be used in any of the steps up to the estimation process SA156, which is a step for estimating the normal vector of the projection surface SC.

[0053] 4. Summary of this disclosure The present disclosure is not limited to the above-described embodiments and modifications, and can be realized in various forms without departing from the spirit thereof. For example, the present disclosure can also be realized in the following forms. The technical features in the above embodiments corresponding to the technical features in each form described below can be replaced or combined as appropriate to solve some or all of the problems of the present disclosure or to achieve some or all of the effects of the present disclosure. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. A summary of this disclosure is provided below.

[0054] (Appendix 1) A calculation method according to one aspect of the present disclosure includes: projecting a pattern detectable by a camera from a projector onto a planar projection surface; calculating, based on a first captured image obtained by capturing an image of the pattern with the camera from a first position, a first transformation matrix for transforming the coordinate system of the first captured image from one side of a coordinate system of the projector to another; calculating, based on a second captured image obtained by capturing an image of the pattern with the camera from a second position different from the first position, a second transformation matrix for transforming the coordinate system of the second captured image and the coordinate system of the projector from one side to the other; and estimating a normal vector of the projection surface based on a first normal vector of the projection surface calculated based on the first transformation matrix, a second normal vector of the projection surface calculated based on the second transformation matrix, internal parameters of the camera, and internal parameters of the projector. This calculation method allows for accurate calculation of the normal direction of the projection surface, thereby enabling highly accurate correction of the normal direction of the projection surface, i.e., the shape of the projected image viewed from the front of the projection surface.

[0055] (Appendix 2) A more preferred calculation method according to one aspect of the present disclosure is the calculation method described in (Supplementary Note 1), which further includes calculating parameters for correcting the outer shape of the image projected by the projector based on the estimated normal vector. According to the calculation method of this aspect, the shape of the projected image as viewed from the front of the projection surface can be corrected to a rectangle with high accuracy.

[0056] (Appendix 3) A more preferred calculation method according to one aspect of the present disclosure is the calculation method described in (Supplementary Note 1) or (Supplementary Note 2), which further includes estimating internal parameters of the camera so that a reprojection error calculated from the first transformation matrix and the second transformation matrix is ​​minimized. According to the calculation method of this aspect, even when it is difficult to know the internal parameters of the camera in advance, for example, when the camera is prepared arbitrarily by a user, the internal parameters of the camera can be calculated with high accuracy.

[0057] (Appendix 4) A more preferred embodiment of the calculation method is the calculation method described in (Supplementary Note 3), in which estimating the normal vector of the projection surface includes re-estimating the normal vector based on the first transformation matrix and the second transformation matrix re-calculated using the estimated internal parameters of the camera. According to the calculation method of this embodiment, even if it is difficult to know the internal parameters of the camera in advance, the internal parameters of the camera and the normal direction of the projection surface can be calculated with high accuracy, so that the shape of the projected image when the projection surface is viewed from the front can be corrected with high accuracy.

[0058] (Appendix 5) A more preferred calculation method according to one aspect of the present disclosure is the calculation method described in any one of (Supplementary Note 1) to (Supplementary Note 4), in which the camera is provided in a portable device, the portable device having a sensor for detecting the direction of gravity, and the method further includes calculating, based on the direction of gravity detected by the sensor and the estimated normal vector, parameters for correcting the image projected by the projector onto the projection surface so that the outline of the image projected by the projector onto the projection surface becomes rectangular when the projection surface is viewed from the front, with one side of the rectangle aligned with the direction of gravity. According to this aspect, a projected image that is straight with respect to the installation surface (e.g., the ground) of a projection target having a projection surface can be obtained.

[0059] (Appendix 6) In a more preferred calculation method according to one aspect of the present disclosure, estimating the normal vector comprises: The calculation method according to any one of (Supplementary Note 1) to (Supplementary Note 5) includes estimating the normal vector by using the external parameters of the camera estimated from the transformation matrix, and selecting two or more captured images, from among a plurality of captured images including the first captured image and the second captured image, whose distance between captured positions is equal to or greater than a threshold. According to this aspect, it is possible to improve the estimation accuracy of the normal vector.

[0060] (Appendix 7) A more preferable calculation method according to one aspect of the present disclosure is the calculation method described in (Supplementary Note 6), in which the threshold value is determined according to the distance between the projection surface and the projector. According to this aspect, it is possible to improve the accuracy of estimating the normal vector by taking into account the distance between the projection surface and the projector.

[0061] (Appendix 8) A program according to an embodiment of the present disclosure causes a computer to execute the following steps: project a pattern detectable by a camera from a projector onto a planar projection surface; calculate, based on a first captured image obtained by capturing an image of the pattern with the camera from a first position, a first transformation matrix for converting the coordinate system of the first captured image from one side of a coordinate system of the projector to another; calculate, based on a second captured image obtained by capturing an image of the pattern with the camera from a second position different from the first position, a second transformation matrix for converting the coordinate system of the second captured image and the coordinate system of the projector from one side to the other; and estimate a normal vector of the projection surface based on a first normal vector of the projection surface calculated based on the first transformation matrix, a second normal vector of the projection surface calculated based on the second transformation matrix, internal parameters of the camera, and internal parameters of the projector. This calculation method enables the normal direction of the projection surface to be calculated with high accuracy, thereby enabling highly accurate correction of the normal direction of the projection surface, i.e., the shape of the projected image viewed from the front of the projection surface.

[0062] (Appendix 9) The information processing device of the present disclosure includes a communication device that communicates with a projector that projects an image onto a planar projection surface, a camera that captures the image projected onto the projection surface from the projector, and a processing device, wherein the processing device executes the following: projecting a pattern that can be detected by the camera from the projector onto the projection surface; calculating, based on a first captured image obtained by capturing the pattern with the camera from a first position, a first transformation matrix for converting the coordinate system of the first captured image from one side to another of the projector's coordinate system; calculating, based on a second captured image obtained by capturing the pattern with the camera from a second position different from the first position, a second transformation matrix for converting the coordinate system of the second captured image and the projector's coordinate system from one side to another; and estimating the normal vector of the projection surface based on a first normal vector of the projection surface calculated based on the first transformation matrix, a second normal vector of the projection surface calculated based on the second transformation matrix, internal parameters of the camera, and internal parameters of the projector. According to the information processing device of this aspect, the normal direction of the projection surface can be calculated with high precision, and therefore the normal direction of the projection surface, i.e., the shape of the projected image viewed from the front of the projection surface, can be corrected with high precision. [Explanation of symbols]

[0063] 1...display system, 10A, 10B...information processing device, 20...projector, 110...processing device, 111...first control unit, 112...second control unit, 113...third control unit, 114A, 114B...calculation unit, 120...communication device, 130...camera, 140...display, 150...input device, sensor...160, 170...memory device, PRA, PRB...program.

Claims

1. Projecting a pattern detectable by a camera from a projector onto a flat projection surface; calculating, based on a first captured image obtained by capturing an image of the pattern with the camera from a first position, a first transformation matrix for transforming a coordinate system of the first captured image from one side to another side of a coordinate system of the projector; calculating a second transformation matrix for transforming from one of a coordinate system of the second captured image and a coordinate system of the projector to the other, based on a second captured image obtained by capturing an image of the pattern with the camera from a second position different from the first position; estimating a normal vector of the projection surface based on a first normal vector of the projection surface calculated based on the first transformation matrix, a second normal vector of the projection surface calculated based on the second transformation matrix, internal parameters of the camera, and internal parameters of the projector; Calculation method including.

2. The calculation method according to claim 1 , further comprising: calculating a parameter for correcting an outer shape of the image projected by the projector based on the estimated normal vector.

3. The calculation method according to claim 1 , further comprising estimating internal parameters of the camera so that a reprojection error calculated from the first transformation matrix and the second transformation matrix is ​​minimized.

4. Estimating the normal vector of the projection plane includes: The calculation method according to claim 3 , further comprising: re-estimating the normal vector based on the first transformation matrix and the second transformation matrix re-calculated using the estimated internal parameters of the camera.

5. The camera is provided on a portable device, the portable device includes a sensor for detecting a direction of gravity; and further comprising calculating, based on the direction of gravity detected by the sensor and the estimated normal vector, parameters for correcting the image projected by the projector so that the outline of the image projected by the projector onto the projection surface becomes rectangular when the projection surface is viewed from the front, and one side of the rectangle is aligned with the direction of gravity. The calculation method according to claim 1 .

6. estimating the normal vector and estimating the normal vector by selecting two or more captured images, the distance between which is equal to or greater than a threshold, from a plurality of captured images including the first captured image and the second captured image, using the external parameters of the camera estimated from the transformation matrix. The calculation method according to claim 1 .

7. The calculation method according to claim 6 , wherein the threshold value is determined according to a distance between the projection surface and the projector.

8. On the computer, Projecting a pattern detectable by a camera from a projector onto a flat projection surface; calculating, based on a first captured image obtained by capturing an image of the pattern with the camera from a first position, a first transformation matrix for transforming a coordinate system of the first captured image from one side to another side of a coordinate system of the projector; calculating a second transformation matrix for transforming from one of a coordinate system of the second captured image and a coordinate system of the projector to the other, based on a second captured image obtained by capturing an image of the pattern with the camera from a second position different from the first position; estimating a normal vector of the projection surface based on a first normal vector of the projection surface calculated based on the first transformation matrix, a second normal vector of the projection surface calculated based on the second transformation matrix, internal parameters of the camera, and internal parameters of the projector; A program that executes the following.

9. a communication device that communicates with a projector that projects an image onto a flat projection surface; a camera that captures an image projected onto the projection surface from the projector; a processing device, The processing device includes: projecting a pattern detectable by the camera from the projector onto the projection surface; calculating, based on a first captured image obtained by capturing an image of the pattern with the camera from a first position, a first transformation matrix for transforming a coordinate system of the first captured image from one side to another side of a coordinate system of the projector; calculating a second transformation matrix for transforming from one of a coordinate system of the second captured image and a coordinate system of the projector to the other, based on a second captured image obtained by capturing an image of the pattern with the camera from a second position different from the first position; estimating a normal vector of the projection surface based on a first normal vector of the projection surface calculated based on the first transformation matrix, a second normal vector of the projection surface calculated based on the second transformation matrix, internal parameters of the camera, and internal parameters of the projector; An information processing device that executes the above.

Images