Information processing device, information processing method, and program

Abstract

Improve the accuracy of camera calibration. [Solution] The information processing device according to the present invention comprises: a calibration calculation unit that performs a first calibration of the camera based on the feature point positions of a planar pattern contained in a first image captured by a camera and the feature point positions of the planar pattern in world coordinates; and an output unit that outputs a deformed pattern obtained by deforming the planar pattern so that the planar pattern is captured in front of the camera based on the first calibration. The calibration calculation unit acquires an image of the deformed pattern captured by the camera as a second image, and performs a second calibration of the camera based on the feature point positions of the deformed pattern in the second image and the feature point positions of the deformed pattern in world coordinates.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] A method is known in which a planar pattern with known positions of feature points in world coordinates is photographed, and camera calibration is performed based on the correspondence between the feature points in world coordinates and the feature points in the photographed image. Non-Patent Document 1 discloses a specific calculation method for performing the above-described calibration.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of this disclosure is to provide an information processing apparatus, an information processing method, and a program that improve the accuracy of camera calibration.

Means for Solving the Problems

[0005] One aspect of the information processing apparatus of this disclosure includes a calibration calculation unit that performs a first calibration of the camera based on the position of feature points of a planar pattern contained in a first image captured by a camera and the position of feature points of the planar pattern in world coordinates, and an output unit that outputs a deformed pattern obtained by deforming the planar pattern so that the planar pattern is captured in front of the camera based on the first calibration, wherein the calibration calculation unit acquires an image of the deformed pattern captured by the camera as a second image, and performs a second calibration of the camera based on the position of feature points of the deformed pattern in the second image and the position of feature points of the deformed pattern in world coordinates.

[0006] One aspect of the information processing method disclosed herein involves an information processing device performing a first calibration of the camera based on the feature point positions of a planar pattern contained in a first image captured by the camera and the feature point positions of the planar pattern in world coordinates; outputting a deformed pattern by deforming the planar pattern so that the planar pattern is captured in front of the camera based on the first calibration; acquiring an image of the deformed pattern captured by the camera as a second image; and performing a second calibration of the camera based on the feature point positions of the deformed pattern in the second image and the feature point positions of the deformed pattern in world coordinates.

[0007] One aspect of the information processing program disclosed herein involves causing a computer to perform a first calibration of the camera based on the feature point positions of a planar pattern contained in a first image captured by the camera and the feature point positions of the planar pattern in world coordinates; outputting a deformed pattern by deforming the planar pattern so that it is photographed in front of the camera based on the first calibration; acquiring an image of the deformed pattern captured by the camera as a second image; and performing a second calibration of the camera based on the feature point positions of the deformed pattern in the second image and the feature point positions of the deformed pattern in world coordinates. [Effects of the Invention]

[0008] According to this disclosure, it is possible to provide an information processing device, an information processing method, and a program that improve the accuracy of camera calibration. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows the hardware configuration of the information processing device in this disclosure. [Figure 2] This figure shows one functional configuration of the information processing device in this disclosure. [Figure 3] This figure shows the processing in the output unit 102 of the information processing device in this disclosure. [Figure 4] This diagram shows one operation flow of the information processing device described in this disclosure. [Figure 5] This diagram shows one operation flow of the information processing device described in this disclosure. [Figure 6] This diagram shows one operation flow of the information processing device described in this disclosure. [Figure 7] This diagram shows one operation flow of the information processing device described in this disclosure. [Figure 8] This diagram shows one operation flow of the information processing device described in this disclosure. [Figure 9] This figure shows the functional configuration of the information processing device in this disclosure. [Figure 10] This diagram shows one operation flow of the information processing device described in this disclosure. [Figure 11] This diagram shows one operation flow of the information processing device described in this disclosure. [Modes for carrying out the invention]

[0010] The following describes embodiments of the information processing device, information processing method, and program with reference to the drawings.

[0011] <First Embodiment> The information processing device 10, the camera 11, and the display unit 103, which function in conjunction with the information processing device 10, will be described with reference to Figures 1 to 4.

[0012] (Hardware configuration) First, the hardware configuration of the information processing device 10 will be explained with reference to Figure 1. Figure 1 is a block diagram showing the hardware configuration of the information processing device 10.

[0013] As shown in Figure 1, the information processing device 10 comprises a processor 1, RAM (RANDOM ACCESS MEMORY) 2, ROM (READ ONLY MEMORY) 3, and a storage device 4. The information processing device 10 is also connected to an input device 5 and an output device 6 via a data bus 7.

[0014] Processor 1 reads a computer program. For example, processor 1 is configured to read a computer program stored in at least one of RAM 2, ROM 3, and storage device 4. Alternatively, processor 1 may read a computer program stored in a computer-readable storage medium using a storage medium reader device not shown in the figure. Processor 1 may obtain (read) a computer program from a device not shown disposed outside information processing apparatus 10 via a network interface. By executing the read computer program, processor 1 controls RAM 2, storage device 4, input device 5, and output device 6. In particular, in this embodiment, when the computer program read by processor 1 is executed, a functional block for performing camera calibration is realized within processor 1. As an example, the functional block may be calibration calculation unit 101 and output unit 102 described later. Also, as processor 1, one of a CPU (CENTRAL PROCESSING UNIT), GPU (GRAPHICS PROCESSING UNIT), FPGA (FIELD-PROGRAMMABLE GATE ARRAY), DSP (DEMAND-SIDE PLATFORM), ASIC (APPLICATION SPECIFIC INTEGRATED CIRCUIT) may be used, or a plurality may be used in parallel.

[0015] RAM 2 temporarily stores the computer program executed by processor 1. RAM 2 temporarily stores data used when processor 1 executes a computer program. RAM 2 may be, for example, D-RAM (DYNAMIC RAM).

[0016] ROM 3 stores the computer program executed by processor 1. ROM 3 may also store other fixed data. ROM 3 may be, for example, P-ROM (PROGRAMMABLE ROM).

[0017] The storage device 4 stores data that the information processing device 10 stores in the long term. The storage device 4 may operate as a temporary storage device of the processor 1. The storage device 4 may include, for example, at least one of a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device.

[0018] The input device 5 is a device that receives an input regarding the information processing device 10 from the outside. The input device 5 may include, for example, at least one of a keyboard, a mouse, and a touch panel, or may receive an input of media information from an external device such as a camera.

[0019] The output device 6 is a device that outputs information regarding the information processing device 10 to the outside. The output device 6 may be, for example, a display device (e.g., a display) capable of displaying information regarding the information processing device 10.

[0020] (Functional configuration) Next, the functional configuration of the information processing device 10 will be described while referring to FIG. 2. FIG. 2 is a block diagram showing the functional configuration of the information processing system 100.

[0021] As shown in FIG. 2, the information processing system 100 is composed of an information processing device 10, a camera 11, and a display unit 103. The information processing device 10 includes a calibration calculation unit 101 and an output unit 102 as processing blocks or physical processing circuits for realizing its functions. Note that each of the calibration calculation unit 101 and the output unit 102 may be realized, for example, by the above-described processor 1 reading a program. Further, a camera 11 as an input device 5 and a display unit 103 as an output device 6 that function in conjunction with the information processing device 10 are connected to the information processing device 10. The camera 11 has a photographing unit 104 and is capable of photographing.

[0022] The calibration calculation unit 101 performs camera calibration using the image captured by the camera 11, which photographs the planar pattern displayed on the display unit 103. The image captured by the camera 11 of the planar pattern will hereafter be referred to as the first image. The calibration calculation unit 101 performs first calibration using the correspondence between the feature point positions of the planar pattern on the first image and the feature point positions of the planar pattern on world coordinates. The planar pattern only needs to be planar and have known feature point positions on world coordinates. The planar pattern may be displayed by an electronic device such as a display or projector. The display unit 103 may be an electronic device such as a display or projector. If the feature point positions on world coordinates are unknown, the calibration calculation unit 101 may calculate the feature point positions of the planar pattern on world coordinates using the image data of the planar pattern and the display unit 103. Specifically, when the display unit 103 is a display, the calibration calculation unit 101 detects the feature point positions on the image data of the planar pattern. After that, the calibration calculation unit 101 calculates the feature positions of the planar pattern on world coordinates. The calculation may be performed, for example, based on the position of feature points (PIXEL) on a display showing a planar pattern and the PPI (PIXEL PER INCH) of the display. The planar pattern may be a calibration board with an arbitrary pattern, such as a CHARUCO BOARD (HTTPS: / / DOCS.OPENCV.ORG / 3.4 / DF / D4A / TUTORIAL_CHARUCO_DETECTION.HTML). The calibration calculation unit 101 may detect the position of feature points of the planar pattern on the first image from the first image by using a feature point detection method (for example, SIFT (SCALE INVARIANT FEATURE TRANSFORM)). When using the above-mentioned calibration board, the calibration calculation unit 101 may detect the position of feature points of the planar pattern on the first image using a grid point detection method included in open-source software such as OPENCV (HTTPS: / / OPENCV.ORG / ).If the correspondence between the feature point positions in world coordinates and the feature point positions in the first image is unknown, the calibration calculation unit 101 may perform calculations to determine the correspondence between each feature point. For example, the calibration calculation unit 101 may perform feature point matching based on feature quantities such as SIFT. As a camera calibration method, a method is used in which the intrinsic and extrinsic parameters of the camera are calculated by geometric calculations (matrix operations) based on the correspondence between the feature point positions of the planar pattern in the first image and the feature point positions of the planar pattern in world coordinates. For example, the calibration calculation unit 101 may use the method described in Non-Patent Literature 1.

[0023] The output unit 102 outputs a deformed pattern, which is a modified version of the planar pattern, based on the parameters calculated by the first calibration performed by the calibration calculation unit 101, so that it appears as if it were facing forward from the viewpoint of the camera 11. When the output deformed pattern is captured again by the camera 11 at the same viewpoint position as when the first image was taken, the captured image will have the same shape as when the planar pattern is viewed from the front.

[0024] An example of a specific method for deforming a planar pattern will be explained with reference to Figure 3. Furthermore, the deformation method described below is just one example; other methods may be used as long as the transformation results in a planar pattern that appears to be viewed from the front by camera 11. Figure 3 shows the deformation process of output unit 102, which outputs a deformed pattern that appears to be viewed from the front by camera 11, the camera being calibrated. Output unit 102 outputs the deformed pattern by performing steps S001 to S004. Camera 11 is the camera being calibrated. Display unit 103 is a display unit that displays the planar pattern or the deformed pattern. Display unit 103 may be an electronic device such as a display or projector. First image 8 is identical to the first image described above. Front-facing image 9 is an image that has been transformed so that the planar pattern appears to be viewed from the front within the pixel area of ​​the planar pattern shown in first image 8.

[0025] The processes from steps S001 to S004 will now be explained. In step S001, the output unit 102 calculates a matrix (hereinafter referred to as the H matrix) that performs a planar homography from each of the four corner points of the planar pattern in world coordinates displayed on the display unit 103 to each of the four corner points of the planar pattern on the first image 8. In step S002, the output unit 102 generates a frontal image 9 by performing a projection transformation within the pixel area of ​​the planar pattern in the first image 8 so that the planar pattern appears as if it were facing forward. An example of this projection transformation is given below. The output unit 102 refers to the image data of the planar pattern displayed on the display unit 103, specifies a region within the pixel area of ​​the planar pattern in the first image 8 that has the same shape as the image data, and performs a projection transformation on that region. In step S003, the output unit 102 performs a projection transformation on the pixel area of ​​the planar pattern in the frontal image 9 created in step S002 using the inverse matrix of the H matrix. This provides a display area for the deformed pattern that appears to be in the front when viewed from camera 11. In step S004, the output unit 102 performs a projection transformation on the planar pattern displayed on the display unit 103 to match the display area of ​​the deformed pattern obtained in step S003, thereby creating a deformed pattern that appears to be in the front when viewed from camera 11.

[0026] Furthermore, the output unit 102 has a function to output a deformed pattern. The output unit 102 may be configured to display the deformed pattern on an electronic device such as a display or projector. The output unit 102 may also output a deformed pattern to display the deformed pattern on the display unit 103.

[0027] After the output unit 102 outputs the deformed pattern, the camera 11, which has the same viewpoint position as when the first image was taken, captures the outputted deformed pattern. The calibration calculation unit 101 performs a second calibration using the correspondence between the feature point positions of the deformed pattern on the captured image and the feature point positions of the deformed pattern in world coordinates. The captured image in which the deformed pattern is captured will henceforth be referred to as the second image. The deformed pattern is displayed so that it is visible from the front on the second image used for calibration. The detection of feature points of the deformed pattern on the second image and the correspondence between feature points in world coordinates and feature points on the second image may be performed using the method used in the first calibration. The camera calibration method may also be performed using the method used in the first calibration. The calibration calculation unit 101 may calculate the feature point positions of the deformed pattern in world coordinates based on the deformation processing of the planar pattern performed by the output unit 102 and the feature point positions on the display that shows the deformed pattern. Specifically, the calibration calculation unit 101 detects the feature point positions on the image data of the planar pattern, and then performs the deformation process (projection transformation) performed by the output unit 102 on the detected feature points to determine the feature point positions on the deformed pattern. The calibration calculation unit 101 calculates the feature positions of the deformed pattern in world coordinates based on the feature point positions (PIXELs) on the display showing the deformed pattern and the PPI of the display.

[0028] (Flow of operations) The operation flow of the information processing device 10 will be explained with reference to Figure 4. Figure 4 is a flowchart showing the operation flow of the information processing device.

[0029] Steps S101 to S103 will be explained below. In step S101, the calibration calculation unit 101 performs the first calibration. Specifically, the calibration calculation unit 101 performs the first camera calibration using the image captured by the camera 11, which has captured the planar pattern displayed on the display unit 103. Then, in step S102, the output unit 102 outputs a deformed pattern, which is a modified version of the planar pattern, based on the parameters calculated by the first calibration, so that it appears to be in front of the camera 11's viewpoint. Finally, in step S103, the calibration calculation unit 101 performs the second calibration, and the operation of the information processing device ends (End). The second calibration is performed by the camera 11 capturing the deformed pattern output in step S102, and the calibration calculation unit 101 using the captured deformed pattern.

[0030] (Technical effects) Next, we will explain the technical effects obtained by the information processing device 10.

[0031] As explained in Figures 1 to 4, since the camera can capture the deformation pattern in the second image from the front, the accuracy of feature point detection of the deformation pattern on the second image can be improved. As a result, more precise feature point positions can be used for calibration, and the information processing device 10 can improve the parameter estimation accuracy in the calibration.

[0032] <Second Embodiment> Another example of the operation of the information processing device 10 will be explained with reference to Figure 5. The hardware configuration and functional configuration of the information processing device 10 according to the second embodiment may be the same as those of the first embodiment. In the following embodiments, the parts that differ from the first embodiment will be explained in detail, and the explanation of overlapping parts will be omitted as appropriate.

[0033] (Flow of operations) First, we will explain the operation flow of the information processing device 10, referring to Figure 5. Figure 5 is a flowchart showing the operation flow of the information processing device 10.

[0034] Steps S201 to S207 will be explained below. In step S201, the calibration calculation unit 101 receives an image captured by the camera 11 that has captured a planar pattern or deformed pattern displayed on the display unit 103. In step S201, which is the first step performed, the image of the planar pattern is input as the first image, and the image of the deformed pattern output in step S207, which will be described later, is input as the second image. Next, in step S202, the calibration calculation unit 101 detects feature points of the planar pattern or deformed pattern on the captured image. In step S203, the calibration calculation unit 101 determines whether the number of feature points detected in step S202 is equal to or greater than the number of feature points sufficient to perform camera calibration in step S204 (i.e., the minimum number required to execute the calibration algorithm). If the result of the determination is that the number of feature points is sufficient (Yes), the process proceeds to step S205; if the result is that the number of feature points is not sufficient (No), the process proceeds to step S204. In step S204, the output unit 102 scales and outputs the planar pattern or deformed pattern so that the entire deformed pattern is displayed to the maximum extent within the display area where the deformed pattern is displayed. The output unit 102 scales the planar pattern or deformed pattern so that the size of the area where the pattern is displayed changes while maintaining its shape. The output planar pattern or deformed pattern is displayed on the display unit 103, and the process proceeds back to step S201. In step S205, the calibration calculation unit 101 performs camera calibration based on the feature point positions of the planar pattern or deformed pattern on the captured image detected in step S202 and the feature point positions of the planar pattern or deformed pattern on the world coordinate system. The camera calibration is either the first camera calibration or the second camera calibration described above. In step S206, the calibration calculation unit 101 determines whether the captured image input in step S201 is the second image.If the judgment determines that the captured image is the second image (Yes), the operation ends when the output unit 102 outputs the camera parameters calculated by the camera calibration in step S205 (i.e., the second camera calibration) as the final processing result (End). If the judgment determines that the captured image is not the second image (i.e., the first image) (No), the process proceeds to step S207. In step S207, the output unit 102 outputs the deformed pattern visible from the front of the camera 11 based on the results of the camera calibration performed in step S205 (i.e., the first camera calibration). After that, the process proceeds back to step S201, where the calibration calculation unit 101 acquires the captured image of the deformed pattern (i.e., the second image).

[0035] (Technical effects) Next, we will explain the technical effects obtained by the information processing device 10.

[0036] As explained in Figure 5, the information processing device 10 can output a scaled planar pattern or a deformed pattern even if it is not possible to detect enough feature points in the captured image to perform camera calibration. This makes it possible for the information processing device 10 to perform camera calibration even when it is difficult to perform camera calibration.

[0037] <Third Embodiment> Another example of the operation of the information processing device 10 will be described with reference to Figure 6. The hardware configuration and functional configuration of the information processing device 10 may be the same as in the first embodiment. Steps common to Figure 5 are given the same numbering and their explanations are omitted as appropriate.

[0038] (Flow of operations) First, the operation flow of the information processing device 10 will be explained with reference to Figure 6. Figure 6 is a flowchart showing the operation flow of the information processing device 10. The flowchart shown in Figure 6 differs from the flowchart in Figure 5 in that it lacks steps S203 and S204 and has step S301.

[0039] Step S301 will now be explained. Step S301 is a process performed by the output unit 102. In step S301, the output unit 102 outputs a planar pattern of a first size that is larger than the displayable area of ​​the display unit 103. The output planar pattern is displayed across the entire display area of ​​the display unit 103. On the display unit 103, a portion of the planar pattern will be displayed across the entire displayable area.

[0040] The steps from step S301 onward are the same as the flowchart in Figure 5, which does not include steps S203 and S204. The difference from Figure 5 due to the presence of step S301 will be explained below. In step S207, which is performed after step S301, if a deformation process as shown in step S004 of Figure 3 is performed, the planar pattern outside the display area will appear within the display area. This allows the display unit 103 to display the deformation pattern over a wider area.

[0041] (Technical effects) Next, the technical effects obtained by the information processing device 10 will be explained with reference to Figures 3 and 6.

[0042] When a deformation process is performed as shown in step S004 of Figure 3, the display area of ​​the pattern changes between the planar pattern and the deformed pattern. In other words, the display area of ​​the pattern on the display unit 103 may become smaller due to the deformation process. To counter this, by pre-displaying a portion of the planar pattern that is larger than the displayable area of ​​the display unit 103 across the entire display unit 103, when deformation is performed, the planar pattern outside the display area appears within the display area, thereby mitigating the effect of the pattern's display area becoming smaller. As a result, the information processing device 10 can create a pattern display state that makes it easier to detect feature points, thereby improving the accuracy of camera calibration.

[0043] <Fourth Embodiment> Another example of the operation of the information processing device 10 will be described with reference to Figures 7 and 8. The hardware and functional configuration of the information processing device 10 may be the same as in the first embodiment. Steps common to Figure 5 are given the same numbering and their explanations are omitted as appropriate.

[0044] (Flow of operations) Figures 7 and 8 show the operation flow of a fourth embodiment with a different processing order. First, a first example of the operation flow of the information processing device 10 will be described with reference to Figure 7. Figure 7 is a flowchart of the operation flow of the information processing device 10. Steps common to Figures 5 and 6 are given the same numbers and explanations are omitted as appropriate. The flowchart shown in Figure 7 differs from that in Figure 5 in that it does not have steps S203 and S204, but has steps S401 to S404.

[0045] Steps S401 to S404 will be explained below. Steps S401 to S404 are performed when it is determined in step S206 that the captured image is the second image (Yes). In step S401, the calibration calculation unit 101 determines whether the interval between feature points of the deformation pattern on the second image detected in step S202 is shorter than the first threshold. The interval between feature points may be, for example, the sum or average value of the Euclidean distances of each feature point on the second image. The first threshold may be set to any value as long as it is smaller than the second threshold, which will be described later. If the determination in step 401 is found to be shorter than the first threshold, the process proceeds to step S403 (Yes); otherwise, the process proceeds to step S402 (No). In step S402, the calibration calculation unit 101 determines whether the interval between feature points of the deformation pattern on the second image detected in step S202 is longer than the second threshold. An example of the interval between feature points is as described above. The second threshold can be set to any value as long as it is greater than the first threshold. If the result of the determination in step S402 is determined to be longer than the second threshold, the process proceeds to step S404 (Yes); otherwise, the process is terminated (No). In step S403, the output unit 102 applies a scaling transformation process to enlarge the deformed pattern output in step S207 and outputs the deformed pattern after scaling. In step S404, the output unit 102 applies a scaling transformation process to reduce the deformed pattern output in step S207 and outputs the deformed pattern after scaling. After the scaling transformation process of enlargement or reduction of the deformed pattern is performed in step S403 or S404, the process proceeds to step S201, where the second calibration is performed by reacquiring the second image, and the process is repeated until the determination in both steps S401 and S402 is NO.

[0046] Referring to Figure 8, a second example of the operation flow of the information processing device 10 will be described. In the flowchart steps of Figure 8, similar processing is performed to the steps of the flowchart in Figure 7. However, the flowchart in Figure 8 differs from the flowchart in Figure 7 in that the determination of whether the captured image is the second image in step S206 is performed before the camera calibration in step S205. In step S206, if the calibration calculation unit 101 determines that the captured image is the second image (Yes), the process proceeds to step S401; if it determines that the captured image is not the second image (No), the process proceeds to step S205. Also, in the flowchart in Figure 8, if it is determined in step S402 that the interval between feature points is not longer than the second threshold (No), the same camera calibration as in step S205 is performed in step S208, which is another difference from the flowchart in Figure 7. If both the determinations in steps S401 and S402 are NO, the camera calibration in step S208 is performed and the operation ends.

[0047] (Technical effects) Next, we will explain the technical effects obtained by the information processing device 10.

[0048] As explained in Figures 7 and 8, the information processing device 10 has the function of enlarging and outputting a deformed pattern if the interval between feature points on the second image is shorter than the first threshold, and reducing the deformed pattern if the interval is longer than the second threshold. By performing the determinations made in steps S401 and S402 and the corresponding enlargement and reduction processing in steps S403 and S404, the information processing device 10 improves the detection accuracy of the feature points when detecting them again in step S202. Therefore, the information processing device 10 has the effect of enabling highly accurate camera calibration. For example, let's assume that the information processing device 10 performs camera calibration in step S205 using a checkerboard calibration board. In this case, it is possible that the interval between grid points (feature points) detected on the captured image falls below the threshold due to reasons such as the calibration board being too far from the camera 11, making it impossible to accurately detect the position of the feature points. In such cases, the information processing device 10 displays the deformed pattern enlarged on the display unit 103, enabling accurate feature point detection and thereby enabling highly accurate calibration. Similarly, if the calibration board is too close to the camera 11, the spacing between grid points (feature points) detected on the captured image may exceed a threshold, making it impossible to accurately detect the feature point positions. In such cases, the information processing device 10 displays a reduced version of the deformed pattern on the display unit 103, enabling accurate feature point detection and thereby allowing for highly accurate calibration.

[0049] <Fifth Embodiment> Another example of the operation of the information processing device 10 will be described with reference to Figures 9 and 10. Below, the differences from the first embodiment will be explained in detail, and overlapping parts will be omitted as appropriate.

[0050] (Functional configuration) The functional configuration of the information processing device 10 will be described with reference to Figures 9 and 10. Figure 9 is a block diagram showing the functional configuration of the information processing system 100.

[0051] In comparison with Figure 2, Figure 9 differs in that the information processing device 10 further includes a correspondence relationship calculation unit 201. The correspondence relationship calculation unit 201 may be implemented by the processor 1 described above reading a program.

[0052] The correspondence calculation unit 201 performs a correspondence calculation if a first correspondence relationship, which is the positional relationship between the feature point positions of the planar pattern on the first image and the feature point positions of the planar pattern on the world coordinate system, has not been established. An example of a specific correspondence is described below. First, the correspondence calculation unit 201 performs feature point detection and feature calculation on the first image, which is a captured image. For example, a feature point matching method such as SIFT may be used for feature point detection and feature calculation. Next, the correspondence calculation unit 201 performs the same feature point detection and feature calculation on the image data of the planar pattern displayed on the display unit 103. After that, the correspondence calculation unit 201 matches the feature points detected on the first image with the feature points detected on the image data of the planar pattern and extracts the feature points that have a correspondence relationship. Finally, the correspondence calculation unit 201 obtains the feature point positions of the planar pattern on the world coordinate system by calculating where the feature point positions on the image data of the planar pattern, for which the correspondence relationship has been identified through matching, will be displayed when the planar pattern is displayed on the display unit 103. Specifically, for example, if the display unit 103 is a display, the correspondence calculation unit 201 calculates the feature positions of the planar pattern in world coordinates based on the feature point positions (PIXELs) on the display showing the planar pattern and the PPI (PIXEL PER INCH) of the display.

[0053] The correspondence calculation unit 201 can perform correspondence calculations even when a second correspondence relationship, which is the positional relationship between the feature point positions of the deformed pattern on the second image and the feature point positions of the deformed pattern on the world coordinate system, has not been set. For the specific correspondence process, the correspondence calculation unit 201 may use the same method as for calculating the correspondence between the feature point positions of the planar pattern on the first image and the feature point positions of the planar pattern on the world coordinate system.

[0054] (Flow of operations) The operation flow of the information processing device 10 will be explained with reference to Figure 10. Figure 10 is a flowchart showing the operation flow of the information processing device. Steps common to both Figure 5 and Figure 10 are numbered similarly, and explanations are omitted as appropriate. The flowchart shown in Figure 10 differs from that in Figure 5 in that it does not have steps S203 and S204, but does have steps S501 and S502.

[0055] Steps S501 and S502 will be explained below. Steps S501 and S502 are processes performed by the correspondence calculation unit 201. Step S501 is performed after step S202. In step S501, the correspondence calculation unit 201 determines whether the feature point positions of the planar pattern in the first image or the deformed pattern in the second image are associated with the corresponding feature point positions of the planar pattern or deformed pattern in world coordinates. If the determination is made that the two are associated (Yes), the process proceeds to step S205; if the determination is made that the two are not associated (No), the process proceeds to step S502. In step S502, the correspondence calculation unit 201 performs a calculation to associate the feature point positions of the planar pattern in the first image or the deformed pattern in the second image with the corresponding feature point positions of the planar pattern or deformed pattern in world coordinates. After that, the process proceeds to step S205. After step S205, in step S206, if the calibration calculation unit 101 determines that the captured image is the second image (Yes), the operation ends. On the other hand, if the calibration calculation unit 101 determines in step S206 that the captured image is the first image (No), the calibration calculation unit 101 performs the process in step S207. After step S207, the process returns to step S201, and steps S201, S202, and S501 are performed in that order. For each process after the process proceeds to step S501 again, the same processes as described in this paragraph are performed.

[0056] (Technical effects) Next, we will explain the technical effects obtained by the information processing device 10.

[0057] As explained in Figures 9 to 10, the information processing device 10 has the function of associating the feature point positions of the planar pattern in the first image or the deformed pattern in the second image with the corresponding feature point positions of the planar pattern or deformed pattern in world coordinates. This has the effect of enabling camera calibration using any planar pattern.

[0058] <Sixth Embodiment> Another example of the operation of the information processing device 10 will be described with reference to Figure 11. The hardware configuration and functional configuration of the information processing device 10 may be the same as in the first embodiment. In the following, the parts that differ from the first embodiment will be described in detail, and the explanation of overlapping parts will be omitted as appropriate.

[0059] (Flow of operations) The operation flow of the information processing device 10 will be explained with reference to Figure 11. Figure 11 is a flowchart showing the operation flow of the information processing device 10. Steps that perform the same processing as in Figure 5 are numbered in the same way. The flowchart shown in Figure 11 differs from the flowchart in Figure 5 in that it does not have steps S203 and S204, but does have step S601, and that step S206, which determines whether the captured image is the second image or not, is performed before the camera calibration in step S205. In Figure 11, the processing of each step that is common with Figure 5 is the same as in Figure 5.

[0060] Step S601 will now be explained. Step S601 is performed by the calibration calculation unit 101 when it is determined in step S206 that the captured image is the second image (Yes). Here, the calibration calculation unit 101 performs further camera calibration (second calibration) by using the camera parameters obtained in the camera calibration (first calibration) in step S103 as initial values ​​for the camera parameters to be calculated. That is, when the calibration calculation unit 101 of the information processing device 10 performs the second calibration after the output unit 102 outputs the deformed pattern, it uses the parameters calculated in the first calibration as initial values ​​for the estimated camera parameters.

[0061] (Technical effects) Next, we will explain the technical effects obtained by the information processing device 10.

[0062] As explained in Figure 11, the information processing device 10 uses the camera parameters calculated in the first calibration as initial values ​​for the parameters to be calculated in the second camera calibration. This improves the estimation accuracy of the camera parameters calculated by the camera calibration.

[0063] <Note> The embodiments described above may also be described as follows, but are not limited to the following.

[0064] (Note 1) The information processing device comprises a calibration calculation unit that performs a first calibration of the camera based on the position of feature points of a planar pattern contained in a first image captured by the camera and the position of feature points of the planar pattern on a world coordinate system, and an output unit that outputs a deformed pattern obtained by deforming the planar pattern so that the planar pattern is captured in front of the camera based on the first calibration, wherein the calibration calculation unit acquires an image of the deformed pattern captured by the camera as a second image, and performs a second calibration of the camera based on the position of feature points of the deformed pattern in the second image and the position of feature points of the deformed pattern on a world coordinate system.

[0065] (Note 2) The output unit outputs a scaled-up version of the planar pattern displayed by the camera to a display area capable of displaying the planar pattern when the calibration calculation unit cannot detect the feature points on the first image necessary for performing the first calibration, and the calibration calculation unit performs the first calibration using the feature point positions of the scaled-up planar pattern included in the first image captured by the camera, as described in Appendix 1.

[0066] (Note 3) The output unit outputs, when the calibration calculation unit cannot detect the feature points on the second image necessary for performing the second calibration, the deformed pattern displayed when the camera captured the image, scaled down, to a display area capable of displaying the deformed pattern, and the calibration calculation unit performs the second calibration using the feature point positions of the scaled-down deformed pattern included in the second image captured by the camera, as described in Appendix 1 or 2.

[0067] (Note 4) The output unit is an information processing device according to any one of the appendices 1 to 3, which scales the deformation pattern in a display area for displaying the deformation pattern so that the entire deformation pattern is displayed to the maximum extent within the display area, and outputs the scaled deformation pattern.

[0068] (Note 5) The output unit displays the planar pattern in a display area for displaying the planar pattern, with the planar pattern of a first size larger than the display area, across the entire display area, and the calibration calculation unit performs the first calibration using the first image, which includes the planar pattern of the first size, captured by the camera, as described in any of the appendices 1 to 4.

[0069] (Note 6) The calibration calculation unit performs the second calibration by outputting an enlarged version of the deformed pattern to the output unit and then reacquiring the second image if the interval between feature points detected on the second image is shorter than the first threshold, and performs the second calibration by outputting a reduced version of the deformed pattern to the output unit and then reacquiring the second image if the interval between feature points detected on the second image is longer than the second threshold, and the second threshold is greater than the first threshold, as described in any of Appendix 1 to 5.

[0070] (Note 7) The information processing apparatus according to any one of the appendices 1 to 6 further comprises a correspondence calculation unit that calculates at least one of the first correspondence or the second correspondence in the case where a first correspondence relationship, which is the correspondence relationship between the feature point positions of the planar pattern in the first image and the feature point positions of the planar pattern on the world coordinates, is not set, or a second correspondence relationship, which is the correspondence relationship between the feature point positions of the deformed pattern in the second image and the feature point positions of the deformed pattern on the world coordinates, is not set.

[0071] (Note 8) The calibration calculation unit is an information processing device according to any one of the appendices 1 to 7, which, when performing the second calibration after the output unit outputs the deformation pattern, uses the parameters calculated by the first calibration as the initial values ​​of the estimated camera parameters.

[0072] (Note 9) This method is performed by an information processing device that performs a first calibration of the camera based on the position of feature points of a planar pattern contained in a first image captured by the camera and the position of feature points of the planar pattern in world coordinates, outputs a deformed pattern by deforming the planar pattern so that the planar pattern is captured in front of the camera based on the first calibration, acquires an image of the deformed pattern captured by the camera as a second image, and performs a second calibration of the camera based on the position of feature points of the deformed pattern in the second image and the position of feature points of the deformed pattern in world coordinates.

[0073] (Note 10) This program causes a computer to perform the following actions: first calibration of the camera based on the position of feature points of a planar pattern contained in a first image captured by the camera and the position of feature points of the planar pattern in world coordinates; outputting a deformed pattern by deforming the planar pattern so that the planar pattern is captured in front of the camera based on the first calibration; acquiring an image of the deformed pattern captured by the camera as a second image; and performing a second calibration of the camera based on the position of feature points of the deformed pattern in the second image and the position of feature points of the deformed pattern in world coordinates.

[0074] Furthermore, some or all of the configurations described in Appendices 2 to 8, which are dependent on the information processing device described in Appendice 1 above, may also be dependent on Appendice 9 (Method) and Appendice 10 (Program) in the same dependent relationship as in Appendices 2 to 8. Moreover, not limited to Appendices 1 to 8, Appendice 9, and Appendice 10, some or all of the configurations described as appendices may also be dependent on various hardware, software, various recording means for recording software, or systems, without departing from the embodiments described above.

[0075] This disclosure may be modified as appropriate, insofar as it does not contradict the gist or idea of ​​the invention as can be inferred from the claims and the specification as a whole, and information processing devices, information processing methods and programs, including such modifications, are also included in the technical idea of ​​this disclosure.

[0076] Each embodiment can be combined with other embodiments as appropriate. This disclosure is not limited to the embodiments described. [Explanation of symbols]

[0077] 1 processor 2 RAM 3 ROM 4 Storage device 5 Input devices 6. Output device 7 Data bus 10 Information Processing Devices 11 Cameras 100 Information Processing Systems 101 Calibration Calculation Unit 102 Output section 103 Display section 104 Photography Department 201 Correspondence Calculation Unit

Claims

1. A calibration calculation unit performs a first calibration of the camera based on the position of feature points of a planar pattern contained in a first image captured by the camera and the position of the feature points of the planar pattern on a world coordinate system. Based on the first calibration, an output unit outputs a deformed pattern obtained by deforming the planar pattern so that the planar pattern is photographed in front of the camera, Equipped with, The calibration calculation unit acquires an image of the deformation pattern captured by the camera as a second image, and performs a second calibration of the camera based on the feature point positions of the deformation pattern in the second image and the feature point positions of the deformation pattern on the world coordinate system. Information processing device.

2. The output unit, when the calibration calculation unit cannot detect the feature points on the first image necessary for performing the first calibration, outputs the planar pattern, which is a scaled version of the planar pattern displayed when the camera captured the image, to a display area capable of displaying the planar pattern. The calibration calculation unit performs the first calibration using the scaled feature point positions of the planar pattern included in the first image captured by the camera. The information processing apparatus according to claim 1.

3. The output unit, when the calibration calculation unit cannot detect the feature points on the second image necessary for performing the second calibration, outputs the deformed pattern, which is a scaled version of the deformed pattern displayed when the camera captured the image, to a display area capable of displaying the deformed pattern. The calibration calculation unit performs the second calibration using the scaled feature point positions of the deformed pattern contained in the second image captured by the camera. The information processing apparatus according to claim 1 or 2.

4. The output unit scales the deformation pattern in the display area where the deformation pattern is displayed so that the entire deformation pattern is displayed to the maximum extent within the display area, and outputs the scaled deformation pattern. The information processing apparatus according to claim 1 or 2.

5. The output unit displays the planar pattern in a display area for displaying the planar pattern, with a first size that is larger than the display area, across the entire display area. The calibration calculation unit performs the first calibration using the first image, which includes the planar pattern of the first size and is captured by the camera. The information processing apparatus according to claim 1 or 2.

6. The calibration calculation unit, If the interval between feature points detected on the second image is shorter than the first threshold, the second calibration is performed by outputting the enlarged deformed pattern to the output unit and then reacquiring the second image. If the interval between feature points detected on the second image is longer than the second threshold, the second calibration is performed by outputting the reduced deformed pattern to the output unit and then reacquiring the second image. The second threshold is greater than the first threshold. The information processing apparatus according to claim 1 or 2.

7. The system further includes a correspondence calculation unit that calculates at least one of the first correspondence or the second correspondence if, in at least one of the following cases, a first correspondence relationship, which is the correspondence between the feature point positions of the planar pattern in the first image and the feature point positions of the planar pattern on the world coordinates, is not set, or a second correspondence relationship, which is the correspondence between the feature point positions of the deformed pattern in the second image and the feature point positions of the deformed pattern on the world coordinates, is not set. The information processing apparatus according to claim 1 or 2.

8. When the calibration calculation unit performs the second calibration after the output unit outputs the deformation pattern, it uses the parameters calculated by the first calibration as the initial values ​​of the estimated camera parameters. The information processing apparatus according to claim 1 or 2.

9. Based on the position of feature points in a planar pattern contained in the first image captured by the camera and the position of the feature points in the planar pattern on world coordinates, the first calibration of the camera is performed. Based on the first calibration, a deformed pattern is output, which is a modified version of the planar pattern, so that the planar pattern is photographed in front of the camera. The image of the aforementioned deformed pattern captured by the camera is acquired as a second image. The camera's second calibration is performed based on the position of the feature points of the deformation pattern in the second image and the position of the feature points of the deformation pattern in the world coordinate system. The method by which an information processing device executes.

10. Based on the position of feature points in a planar pattern contained in the first image captured by the camera and the position of the feature points in the planar pattern on world coordinates, the first calibration of the camera is performed. Based on the first calibration, a deformed pattern is output, which is a modified version of the planar pattern, so that the planar pattern is photographed in front of the camera. The image of the aforementioned deformed pattern captured by the camera is acquired as a second image. The camera's second calibration is performed based on the position of the feature points of the deformation pattern in the second image and the position of the feature points of the deformation pattern in the world coordinate system. A program that causes a computer to perform a task.

Images