Information processing device, information processing method, and program

Abstract

Improve the accuracy of camera calibration. [Solution] The information processing device of the present invention comprises a calibration calculation unit that performs a first calibration of the camera based on the positions of feature points of a flat pattern contained in a first image captured by the camera and the positions of feature points of the flat pattern on world coordinates, and an output unit that outputs a deformed pattern obtained by deforming the flat pattern based on the first calibration so that the flat pattern is captured facing forward relative to the camera, and the calibration calculation unit obtains an image of the deformed pattern captured by the camera as a second image, and performs a second calibration of the camera based on the positions of feature points of the deformed pattern in the second image and the positions of feature points of the deformed pattern on world coordinates.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] A known method is to photograph a plane pattern whose feature points are known in world coordinates, and perform camera calibration based on the correspondence between the feature points in the world coordinates and the feature points in the photographed image. Non-Patent Document 1 discloses a specific calculation method for performing the above-mentioned calibration. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] ZYZHANG., “A FLEXIBLE NEW TECHNIQUE FOR CAMERA CALIBRATION”, IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE (VOLUME: 22, ISSUE: 11, NOVEMBER 2000), [Retrieved September 12, 2024], Internet<URL: https: / / ieeexplore.ieee.org / document / 888718> Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present disclosure is to provide an information processing device, an information processing method, and a program that improve the accuracy of camera calibration. [Means for solving the problem]

[0005] One aspect of the information processing device disclosed herein includes a calibration calculation unit that performs a first calibration of the camera based on the positions of feature points of a flat pattern included in a first image captured by the camera and the positions of feature points of the flat pattern on world coordinates, and an output unit that outputs a deformed pattern obtained by deforming the flat pattern based on the first calibration so that the flat pattern is captured facing forward relative to the camera, and the calibration calculation unit acquires an image of the deformed pattern captured by the camera as a second image, and performs a second calibration of the camera based on the positions of feature points of the deformed pattern in the second image and the positions of feature points of the deformed pattern on the world coordinates.

[0006] One aspect of the information processing method disclosed herein is an information processing device that performs a first calibration of the camera based on the positions of feature points of a flat pattern included in a first image captured by the camera and the positions of feature points of the flat pattern on world coordinates, outputs a deformed pattern by deforming the flat pattern so that the flat pattern is captured facing forward relative to the camera based on the first calibration, obtains an image of the deformed pattern captured by the camera as a second image, and performs a second calibration of the camera based on the positions of feature points of the deformed pattern in the second image and the positions of feature points of the deformed pattern on the world coordinates.

[0007] One aspect of the information processing program disclosed herein causes a computer to perform a first calibration of the camera based on the positions of feature points of a flat pattern included in a first image captured by the camera and the positions of feature points of the flat pattern on world coordinates; based on the first calibration, output a deformed pattern by deforming the flat pattern so that the flat pattern is captured facing forward relative to the camera; obtain an image of the deformed pattern captured by the camera as a second image; and perform a second calibration of the camera based on the positions of feature points of the deformed pattern in the second image and the positions of feature points of the deformed pattern on the world coordinates. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to provide an information processing device, an information processing method, and a program that improve the accuracy of camera calibration. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a hardware configuration of an information processing device according to the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating one functional configuration of an information processing device according to the present disclosure. [Figure 3] FIG. 2 is a diagram illustrating processing in an output unit 102 of an information processing device according to the present disclosure. [Figure 4] FIG. 2 is a diagram showing an operation flow of an information processing device according to the present disclosure. [Figure 5] FIG. 2 is a diagram showing an operation flow of an information processing device according to the present disclosure. [Figure 6] FIG. 2 is a diagram showing an operation flow of an information processing device according to the present disclosure. [Figure 7] FIG. 2 is a diagram showing an operation flow of an information processing device according to the present disclosure. [Figure 8] FIG. 2 is a diagram showing an operation flow of an information processing device according to the present disclosure. [Figure 9] FIG. 1 is a diagram illustrating a functional configuration of an information processing device according to the present disclosure. [Figure 10] FIG. 2 is a diagram showing an operation flow of an information processing device according to the present disclosure. [Figure 11] FIG. 2 is a diagram showing an operation flow of an information processing device according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of an information processing device, an information processing method, and a program will be described with reference to the drawings.

[0011] First Embodiment The information processing device 10, and the camera 11 and display unit 103 that function in conjunction with the information processing device 10 will be described with reference to FIGS. 1 to 4. FIG.

[0012] (Hardware configuration) First, the hardware configuration of the information processing device 10 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the hardware configuration of the information processing device 10.

[0013] 1, an information processing device 10 includes a processor 1, a RAM (RANDOM ACCESS MEMORY) 2, a ROM (READ ONLY MEMORY) 3, and a storage device 4. The information processing device 10 is also connected to an input device 5 and an output device 6 via a data bus 7.

[0014] The processor 1 loads a computer program. For example, the processor 1 is configured to load a computer program stored in at least one of the RAM 2, the ROM 3, and the storage device 4. Alternatively, the processor 1 may load a computer program stored in a computer-readable storage medium using a storage medium reading device (not shown). The processor 1 may acquire (load) the computer program from a device (not shown) located outside the information processing device 10 via a network interface. The processor 1 controls the RAM 2, the storage device 4, the input device 5, and the output device 6 by executing the loaded computer program. In particular, in this embodiment, when the processor 1 executes the loaded computer program, a functional block for performing camera calibration is realized within the processor 1. For example, the functional block may be a calibration calculation unit 101 or an output unit 102 (described later). Furthermore, the processor 1 may be one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), a DSP (Demand-Side Platform), or an ASIC (Application Specific Integrated Circuit), or may be multiple processors used in parallel.

[0015] The RAM 2 temporarily stores computer programs executed by the processor 1. The RAM 2 temporarily stores data used when the processor 1 executes the computer programs. The RAM 2 may be, for example, a dynamic RAM (D-RAM).

[0016] The ROM 3 stores computer programs executed by the processor 1. The ROM 3 may also store fixed data. The ROM 3 may be, for example, a programmable ROM (P-ROM).

[0017] The storage device 4 stores data that the information processing device 10 will store for a long period of time. The storage device 4 may operate as a temporary storage device for the processor 1. The storage device 4 may include, for example, at least one of a hard disk device, a magneto-optical disk device, an SSD (SOLID STATE DRIVE), and a disk array device.

[0018] The input device 5 is a device that externally receives input related to the information processing device 10. The input device 5 may include, for example, at least one of a keyboard, a mouse, and a touch panel, or may receive input of media information from an external device such as a camera.

[0019] The output device 6 is a device that outputs information related to the information processing device 10 to the outside. The output device 6 may be, for example, a display device (for example, a display) that can display information related to the information processing device 10.

[0020] (Functional configuration) Next, the functional configuration of the information processing device 10 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the functional configuration of the information processing system 100.

[0021] As shown in FIG. 2, the information processing system 100 is composed of an information processing device 10, a camera 11, and a display unit 103. The information processing device 10 includes a calibration calculation unit 101 and an output unit 102 as processing blocks or physical processing circuits for realizing its functions. Note that the calibration calculation unit 101 and the output unit 102 may each be realized by, for example, the above-mentioned processor 1 reading a program. Also, the camera 11 as an input device 5 and the display unit 103 as an output device 6, which function in conjunction with the information processing device 10, are connected to the information processing device 10. The camera 11 has an imaging unit 104 and is capable of taking images.

[0022] The calibration calculation unit 101 performs camera calibration using an image captured by the camera 11 of the flat pattern displayed on the display unit 103. Hereinafter, the image captured by the camera 11 of the flat pattern is referred to as the "first image." The calibration calculation unit 101 performs the first calibration using a correspondence relationship between the feature point positions of the flat pattern on the first image and the feature point positions of the flat pattern on the world coordinate system. The flat pattern may be flat as long as the feature point positions on the world coordinate system are known. The flat pattern may be displayed by an electronic device such as a display or projector. The display unit 103 may be an electronic device such as a display or projector. If the feature point positions on the world coordinate system are not known, the calibration calculation unit 101 may calculate the feature point positions of the flat pattern on the world coordinate system using the image data of the flat pattern and the display unit 103. Specifically, if the display unit 103 is a display, the calibration calculation unit 101 detects the feature point positions on the image data of the flat pattern. Then, the calibration calculation unit 101 calculates the feature point positions of the flat pattern on the world coordinate system. The calculation may be performed based on, for example, the feature point positions (pixels) on a display displaying the flat pattern and the PPI (pixels per inch) of the display. The flat pattern may be a flat or checkered calibration board with an arbitrary pattern (e.g., CHARUCO BOARD (https: / / DOCS.OPENCV.ORG / 3.4 / DF / D4A / TUTORIAL_CHARUCO_DETECTION.HTML)). The calibration calculation unit 101 may detect the feature point positions of the flat pattern on the first image from the first image by using a feature point detection method (e.g., SIFT (SCALE INVARIANT FEATURE TRANSFORM)). When using the above-mentioned calibration board, the calibration calculation unit 101 may detect the feature point positions of the flat pattern on the first image by using a lattice point detection method included in open source software such as OPENCV (https: / / OPENCV.ORG / ).When the correspondence between feature point positions in world coordinates and feature point positions in the first image is unknown, the calibration calculation unit 101 may perform calculations to determine the correspondence between each feature point. For example, the calibration calculation unit 101 may perform feature point matching based on feature quantities such as SIFT. As a camera calibration method, a method is used in which the internal parameters and external parameters of the camera are calculated by geometric calculations (matrix operations) based on the correspondence between feature point positions of a planar pattern on the first image and feature point positions of the planar pattern on world coordinates. For example, the calibration calculation unit 101 may use the method described in Non-Patent Document 1.

[0023] The output unit 102 outputs a deformed pattern obtained by deforming the flat pattern so that it appears to be from the front from the viewpoint of the camera 11, based on the parameters calculated by the first calibration performed by the calibration calculation unit 101. When the output deformed pattern is photographed again by the camera 11 having the same viewpoint position as when the first image was photographed, the shape of the deformed pattern in the photographed image will be the same as when the flat pattern is viewed from the front.

[0024] An example of a specific method for transforming a flat pattern will be described with reference to FIG. 3. The transformation method described below is merely an example, and other methods may be used as long as the transformation results in the flat pattern appearing to be in a frontal position when viewed from camera 11. FIG. 3 shows the transformation process of output unit 102, which outputs a deformed pattern that makes the flat pattern appear to be in a frontal position when viewed from camera 11, which is the subject of calibration. Output unit 102 outputs the deformed pattern by performing steps S001 to S004. Camera 11 is the camera to be calibrated. Display unit 103 is a display unit that displays the flat pattern or the deformed pattern. Display unit 103 may be an electronic device such as a display or a projector. First image 8 is the same as the first image described above. Frontalized image 9 is an image that has been transformed within the pixel region of the flat pattern shown in first image 8 so that the flat pattern appears to be in a frontal position.

[0025] The processes of steps S001 to S004 will be described. In step S001, the output unit 102 calculates a matrix (hereinafter referred to as the H matrix) that forms planar homography (PLANAR HOMOGRAPHY) from each of the four corner points of the flat pattern in the world coordinate system displayed on the display unit 103 to each of the four corner points of the flat pattern on the first image 8. In step S002, the output unit 102 generates a frontalized image 9 by performing a projective transformation within the pixel region of the flat pattern in the first image 8 so that the flat pattern is displayed in a frontalized state. Here is an example of the projective transformation. The output unit 102 references the image data of the flat pattern displayed on the display unit 103, specifies an area within the pixel region of the flat pattern in the first image 8 that has a shape similar to that of the image data, and performs a projective transformation on that area. In step S003, the output unit 102 performs a projective transformation using the inverse matrix of the H matrix on the pixel region of the flat pattern in the frontalized image 9 created in step S002. This obtains a display area for the deformed pattern that appears to be in front when viewed from camera 11. In step S004, output unit 102 performs projective transformation on the planar pattern displayed on display unit 103 to match the display area for the deformed pattern obtained in step S003, thereby creating a deformed pattern that makes the planar pattern appear to be in front when viewed from camera 11.

[0026] Furthermore, the output unit 102 has a function of outputting the deformed pattern. The output unit 102 may display the deformed pattern on an electronic device such as a display or a projector. The output unit 102 may output the deformed pattern so that the deformed pattern is displayed on the display unit 103.

[0027] After the output unit 102 outputs the deformed pattern, the camera 11, which has the same viewpoint as when the first image was captured, captures the output deformed pattern. The calibration calculation unit 101 performs a second calibration using the correspondence between the feature point positions of the deformed pattern on the captured image and the feature point positions of the deformed pattern on world coordinates. The captured image of the deformed pattern is hereinafter referred to as the second image. The deformed pattern is displayed on the second image used for the calibration so that it appears to be from the front. The method used in the first calibration may be used to detect the feature points of the deformed pattern on the second image and to associate the feature points on world coordinates with the feature points on the second image. The method used in the first calibration may be used as the camera calibration technique. The calibration calculation unit 101 may calculate the feature point positions of the deformed pattern on world coordinates based on the flat pattern deformation process performed by the output unit 102 and the feature point positions on a display displaying the deformed pattern. Specifically, the calibration calculation unit 101 detects the positions of feature points on the image data of the flat pattern, and then obtains the positions of the feature points on the deformed pattern by performing the transformation process (projective transformation) on the detected feature points performed by the output unit 102. The calibration calculation unit 101 calculates the feature positions of the deformed pattern on the world coordinate system based on the feature point positions (pixels) on the display showing the deformed pattern and the PPI of the display.

[0028] (Operation flow) The flow of operations of the information processing device 10 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of operations of the information processing device.

[0029] Steps S101 to S103 will be described below. In step S101, the calibration calculation unit 101 performs a first calibration. In detail, the calibration calculation unit 101 performs a first camera calibration using an image captured by the camera 11 that captures the flat pattern displayed on the display unit 103. Thereafter, in step S102, the output unit 102 outputs a deformed pattern obtained by deforming the flat pattern so that it appears straight ahead from the viewpoint of the camera 11, based on the parameters calculated by the first calibration. Finally, in step S103, the calibration calculation unit 101 performs a second calibration, and the operation of the information processing device ends (End). The second calibration is performed by having the camera 11 capture the deformed pattern output in step S102, and the calibration calculation unit 101 uses the captured deformed pattern.

[0030] (Technical Effects) Next, the technical effects obtained by the information processing device 10 will be described.

[0031] 1 to 4, the camera can capture the deformed pattern in the second image from the front, which improves the accuracy of detecting feature points of the deformed pattern in the second image. This allows more precise feature point positions to be used for calibration, which enables the information processing device 10 to improve the accuracy of parameter estimation in the calibration.

[0032] Second Embodiment Another example of the operation of the information processing device 10 will be described with reference to Fig. 5. The hardware configuration and functional configuration of the information processing device 10 according to the second embodiment may be the same as those of the first embodiment. In the following embodiments, differences from the first embodiment will be described in detail, and overlapping portions will not be described as appropriate.

[0033] (Operation flow) First, the flow of operations of the information processing device 10 will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the flow of operations of the information processing device 10.

[0034] Steps S201 to S207 will be described below. In step S201, the calibration calculation unit 101 inputs an image captured by the camera 11 of a flat pattern or a deformed pattern displayed on the display unit 103. In step S201, which is performed first, the captured image of the flat pattern is input as a first image, and the captured image of the deformed pattern output in step S207, which will be described later, is input as a second image. Next, in step S202, the calibration calculation unit 101 detects feature points of the flat pattern or the deformed pattern in the captured image. In step S203, the calibration calculation unit 101 determines whether the number of feature points detected in step S202 is sufficient to perform camera calibration in step S204 (i.e., the minimum number required to execute the calibration algorithm). If it is determined that the number of feature points is sufficient (Yes), the process proceeds to step S205. If it is determined that the number of feature points is insufficient (No), the process proceeds to step S204. In step S204, the output unit 102 scales and outputs the flat pattern or the deformed pattern so that the entire deformed pattern is displayed at its maximum size within the display area where the deformed pattern is displayed. The output unit 102 performs scale conversion in the area where the flat pattern or the deformed pattern is displayed so that the size of the area where the pattern is displayed changes while maintaining the shape of the pattern. The output flat pattern or the deformed pattern is displayed on the display unit 103, and the process returns to step S201. In step S205, the calibration calculation unit 101 performs camera calibration based on the feature point positions of the flat pattern or the deformed pattern on the captured image detected in step S202 and the feature point positions of the flat pattern or the deformed pattern in world coordinates. The camera calibration is either the first camera calibration or the second camera calibration described above. In step S206, the calibration calculation unit 101 determines whether the captured image input in step S201 is the second image.As a result of the determination, if it is determined that the captured image is the second image (Yes), the output unit 102 outputs the camera parameters calculated by the camera calibration (i.e., second camera calibration) in step S205 as the final processing result, and the operation ends (End), whereas if it is determined that the captured image is not the second image (i.e., the first image) (No), the process proceeds to step S207. In step S207, the output unit 102 outputs the deformed pattern as seen from the front from the camera 11, based on the result of the camera calibration (i.e., first camera calibration) performed in step S205. Thereafter, the process proceeds again to step S201, where the calibration calculation unit 101 acquires the captured image of the deformed pattern (i.e., the second image).

[0035] (Technical Effects) Next, the technical effects obtained by the information processing device 10 will be described.

[0036] 5, even when sufficient feature points for performing camera calibration cannot be detected in the captured image, the information processing device 10 can output a scaled flat pattern or a deformed pattern. This allows the information processing device 10 to perform camera calibration even when it is difficult to perform camera calibration.

[0037] Third Embodiment Another example of the operation of the information processing device 10 will be described with reference to Fig. 6. The hardware configuration and functional configuration of the information processing device 10 may be the same as those in the first embodiment. The same numbers are used for steps common to Fig. 5, and descriptions thereof will be omitted as appropriate.

[0038] (Operation flow) First, the flow of operations of the information processing device 10 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the flow of operations of the information processing device 10. The flowchart shown in Fig. 6 differs from the flowchart in Fig. 5 in that it does not include steps S203 and S204 but includes step S301.

[0039] Step S301 will be described. Step S301 is a process performed by the output unit 102. In step S301, the output unit 102 outputs a flat pattern of a first size that is larger than the displayable area of ​​the display unit 103. The output flat pattern is displayed in the entire display area of ​​the display unit 103. A portion of the flat pattern is displayed in the entire displayable area of ​​the display unit 103.

[0040] The steps from step S301 onwards are the same as those in the flowchart of Fig. 5, which does not include steps S203 and S204. The difference from Fig. 5 due to the presence of step S301 will be described below. In step S207, which is performed after step S301, if a deformation process such as that shown in step S004 in Fig. 3 is performed, the flat pattern outside the display area will appear within the display area. This allows the deformed pattern to be displayed over a wider area on display unit 103.

[0041] (Technical Effects) Next, the technical effects obtained by the information processing device 10 will be described with reference to FIGS.

[0042] When a deformation process such as that shown in step S004 of FIG. 3 is performed, the display area of ​​the pattern changes between the flat pattern and the deformed pattern. In other words, the deformation process may reduce the display area of ​​the pattern on the display unit 103. In response to this, by displaying a portion of the flat pattern that is larger than the displayable area of ​​the display unit 103 in advance across the entire display unit 103, the effect of the flat pattern outside the display area appearing within the display area when the deformation process is performed, which reduces the reduction in the display area of ​​the pattern, can be reduced. This allows the information processing device 10 to create a display state of the pattern that makes it easy to detect feature points, thereby improving the accuracy of camera calibration.

[0043] <Fourth embodiment> Another example of the operation of the information processing device 10 will be described with reference to Figures 7 and 8. The hardware configuration and functional configuration of the information processing device 10 may be the same as those in the first embodiment. The same numbers are used for steps common to Figure 5, and descriptions thereof will be omitted as appropriate.

[0044] (Operation flow) 7 and 8 show the operational flow of the fourth embodiment, each having a different processing order. First, a first example of the operational flow of the information processing device 10 will be described with reference to FIG. 7. FIG. 7 is a flowchart showing the operational flow of the information processing device 10. Steps common to FIGS. 5 and 6 are numbered the same and explanations will be omitted where appropriate. The flowchart shown in FIG. 7 differs from FIG. 5 in that steps S203 and S204 are not included, and steps S401 to S404 are included.

[0045] Steps S401 to S404 will be described below. Steps S401 to S404 are performed when it is determined in step S206 that the captured image is the second image (Yes). In step S401, the calibration calculation unit 101 determines whether the interval between the feature points of the deformed pattern on the second image detected in step S202 is shorter than a first threshold value. The interval between the feature points may be, for example, the sum or average of the Euclidean distances of the feature points on the second image. The first threshold value may be set to any value smaller than a second threshold value, which will be described later. If it is determined in step S401 that the interval between the feature points is shorter than the first threshold value, the process proceeds to step S403 (Yes); otherwise, the process proceeds to step S402 (No). In step S402, the calibration calculation unit 101 determines whether the interval between the feature points of the deformed pattern on the second image detected in step S202 is longer than a second threshold value. Examples of the interval between the feature points are as described above. The second threshold may be set to any value greater than the first threshold. If the result of the determination in step S402 is that the length is longer than the second threshold, the process proceeds to step S404 (Yes); otherwise, the process is terminated (No). In step S403, the output unit 102 performs a scale conversion process to enlarge the deformed pattern output in step S207 and outputs the scaled deformed pattern. In step S404, the output unit 102 performs a scale conversion process to reduce the deformed pattern output in step S207 and outputs the scaled deformed pattern. After the scale conversion process to enlarge or reduce the deformed pattern in step S403 or S404, the process proceeds to step S201, where a second calibration is performed by re-acquiring a second image, and the process is repeated until both the determinations in steps S401 and S402 are NO.

[0046] A second example of the operational flow of the information processing device 10 will be described with reference to FIG. 8. The steps of the flowchart in FIG. 8 are similar to the steps of the flowchart in FIG. 7. However, the flowchart in FIG. 8 differs from the flowchart in FIG. 7 in that the determination of whether the captured image is the second image in step S206 is performed before the camera calibration in step S205 is performed. If the calibration calculation unit 101 determines in step S206 that the captured image is the second image (Yes), the process proceeds to step S401. If the calibration calculation unit 101 determines that the captured image is not the second image (No), the process proceeds to step S205. The flowchart in FIG. 8 also differs from the flowchart in FIG. 7 in that if the calibration calculation unit 101 determines in step S402 that the interval between the feature points is not longer than the second threshold (No), the camera calibration similar to that in step S205 is performed in step S208. If the determinations in both steps S401 and S402 are NO, the camera calibration in step S208 is performed and the operation ends.

[0047] (Technical Effects) Next, the technical effects obtained by the information processing device 10 will be described.

[0048] As described with reference to FIGS. 7 and 8 , the information processing device 10 has a function of enlarging and outputting the deformed pattern if the distance between feature points on the second image is shorter than a first threshold, and reducing and outputting the deformed pattern if the distance is longer than a second threshold. The information processing device 10 performs the determinations in steps S401 and S402 and the corresponding scaling processes in steps S403 and S404, thereby improving the accuracy of feature point detection when detecting feature points again in step S202. Therefore, the information processing device 10 has the effect of enabling highly accurate camera calibration. For example, assume that the information processing device 10 performs camera calibration in step S205 using a checkered calibration board. In this case, it is conceivable that the distance between grid points (feature points) detected on the captured image falls below a threshold, for example, because the calibration board is too far from the camera 11, making it impossible to accurately detect the feature point positions. In such a case, the information processing device 10 enlarges and displays the deformed pattern on the display unit 103, enabling accurate feature point detection and thus highly accurate calibration. Similarly, it is also possible that the interval between the grid points (feature points) detected on the captured image exceeds a threshold value, making it impossible to accurately detect the feature point positions, due to factors such as the calibration board being too close to the camera 11. In such cases, the information processing device 10 can reduce and display the deformed pattern on the display unit 103, thereby enabling accurate feature point detection and highly accurate calibration.

[0049] Fifth Embodiment Another example of the operation of the information processing device 10 will be described with reference to Figures 9 and 10. Below, differences from the first embodiment will be described in detail, and descriptions of overlapping parts will be omitted as appropriate.

[0050] (Functional configuration) The functional configuration of the information processing device 10 will be described with reference to Fig. 9 and Fig. 10. Fig. 9 is a block diagram showing the functional configuration of the information processing system 100.

[0051] 9 differs from Fig. 2 in that the information processing device 10 further includes a correspondence calculation unit 201. The correspondence calculation unit 201 may be realized by the processor 1 described above reading a program.

[0052] The correspondence calculation unit 201 calculates a correspondence when a first correspondence, which is a positional relationship between the feature point positions of the flat pattern on the first image and the feature point positions of the flat pattern on the world coordinate system, has not been established. A specific example of the correspondence calculation will be described. First, the correspondence calculation unit 201 detects feature points and calculates feature amounts on the first image, which is a captured image. A feature point matching method, such as SIFT, may be used for the feature point detection and feature amount calculation. Next, the correspondence calculation unit 201 performs the same feature point detection and feature amount calculation on the image data of the flat pattern displayed on the display unit 103. Thereafter, the correspondence calculation unit 201 matches the feature points detected on the first image with the feature points detected on the image data of the flat pattern to extract corresponding feature points. Finally, the correspondence calculation unit 201 calculates the positions of the feature points on the image data of the flat pattern, for which a correspondence has been identified through matching, at which points the feature points will be displayed when the flat pattern is displayed on the display unit 103, thereby obtaining the feature point positions of the flat pattern on the world coordinate system. Specifically, for example, when the display unit 103 is a display, the correspondence calculation unit 201 calculates the feature position of the planar pattern on the world coordinate system based on the feature point position (pixel) on the display displaying the planar pattern and the PPI (pixel per inch) of the display.

[0053] The correspondence calculation unit 201 can calculate the correspondence even when a second correspondence, which is a positional relationship between the feature point positions of the deformed pattern on the second image and the feature point positions of the deformed pattern on the world coordinate system, is not set. Regarding the specific correspondence processing, the correspondence calculation unit 201 may use a method similar to that used to calculate the correspondence between the feature point positions of the flat pattern on the first image and the feature point positions of the flat pattern on the world coordinate system.

[0054] (Operation flow) The flow of operations of the information processing device 10 will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the flow of operations of the information processing device. Steps common to Fig. 5 are given the same numbers and descriptions thereof will be omitted as appropriate. The flowchart shown in Fig. 10 differs from Fig. 5 in that it does not include steps S203 and S204, but includes steps S501 and S502.

[0055] Steps S501 and S502 will be described below. Steps S501 and S502 are processes performed by the correspondence calculation unit 201. Step S501 is performed after step S202. In step S501, the correspondence calculation unit 201 determines whether the feature point positions of the flat pattern in the first image or the deformed pattern in the second image correspond to the feature point positions of the corresponding flat pattern or deformed pattern in world coordinates. If the determination result indicates that the two correspond to each other (Yes), the process proceeds to step S205. If the determination result indicates that the two do not correspond to each other (No), the process proceeds to step S502. In step S502, the correspondence calculation unit 201 performs a calculation to associate the feature point positions of the flat pattern in the first image or the deformed pattern in the second image with the feature point positions of the corresponding flat pattern or deformed pattern in world coordinates. Then, the process proceeds to step S205. After step S205, if the calibration calculation unit 101 determines in step S206 that the captured image is the second image (Yes), the operation ends. On the other hand, if the calibration calculation unit 101 determines in step S206 that the captured image is the first image (No), the calibration calculation unit 101 performs the process of step S207. After step S207, the process returns to step S201, and steps S201, S202, and S501 are performed in this order. After the process proceeds to step S501 again, the same processes as those described in this paragraph are performed.

[0056] (Technical Effects) Next, the technical effects obtained by the information processing device 10 will be described.

[0057] 9 and 10, the information processing device 10 has a function of associating the feature point positions of the flat pattern in the first image or the deformed pattern in the second image with the feature point positions of the corresponding flat pattern or deformed pattern in world coordinates, thereby enabling the information processing device 10 to perform camera calibration using any flat pattern.

[0058] Sixth Embodiment Another example of the operation of the information processing device 10 will be described with reference to Fig. 11. The hardware configuration and functional configuration of the information processing device 10 may be the same as those in the first embodiment. Below, differences from the first embodiment will be described in detail, and overlapping portions will not be described as appropriate.

[0059] (Operation flow) The flow of operations of the information processing device 10 will be described with reference to FIG. 11. FIG. 11 is a flowchart showing the flow of operations of the information processing device 10. The same numbers are used for steps that perform the same processes as those in FIG. 5. The flowchart shown in FIG. 11 differs from the flowchart in FIG. 5 in that it does not include steps S203 and S204 but includes step S601, and in that step S206, which determines whether the captured image is the second image, is performed before the camera calibration in step S205. In FIG. 11, the processing of each step that is common to FIG. 5 is the same as that in FIG. 5.

[0060] Step S601 will now be described. Step S601 is performed by the calibration calculation unit 101 when it is determined in step S206 that the captured image is the second image (Yes). Here, the calibration calculation unit 101 performs further camera calibration (second calibration) by using the camera parameters calculated in the camera calibration (first calibration) in step S103 as initial values ​​of the camera parameters to be calculated. That is, when performing the second calibration after the output unit 102 outputs the deformed pattern, the calibration calculation unit 101 of the information processing device 10 uses the parameters calculated in the first calibration as initial values ​​of the camera parameters to be estimated.

[0061] (Technical Effects) Next, the technical effects obtained by the information processing device 10 will be described.

[0062] 11, the information processing device 10 uses the camera parameters calculated in the first calibration as the initial values ​​of the parameters to be calculated in the second camera calibration, thereby improving the estimation accuracy of the camera parameters calculated by the camera calibration.

[0063] <Additional Notes> The above-described embodiment can be further described as follows, but is not limited to the following.

[0064] (Appendix 1) The information processing device includes a calibration calculation unit that performs a first calibration of the camera based on the positions of feature points of a flat pattern included in a first image captured by the camera and the positions of feature points of the flat pattern on world coordinates, and an output unit that outputs a deformed pattern obtained by deforming the flat pattern based on the first calibration so that the flat pattern is captured facing forward relative to the camera, wherein the calibration calculation unit obtains an image of the deformed pattern captured by the camera as a second image, and performs a second calibration of the camera based on the positions of feature points of the deformed pattern in the second image and the positions of feature points of the deformed pattern on the world coordinates.

[0065] (Appendix 2) The information processing device described in Appendix 1 is such that, when the calibration calculation unit cannot detect feature points on the first image necessary to perform the first calibration, the output unit outputs a flat pattern in a display area capable of displaying the flat pattern so that the flat pattern is scaled from the flat pattern displayed when the camera was taking the image, and the calibration calculation unit performs the first calibration using the feature point positions of the scaled flat pattern included in the first image taken by the camera.

[0066] (Appendix 3) The information processing device is described in Appendix 1 or 2, wherein, when the calibration calculation unit cannot detect feature points on the second image necessary for performing the second calibration, the output unit outputs a deformed pattern in a display area capable of displaying the deformed pattern so that the deformed pattern is scaled from the deformed pattern displayed when the camera was taking the image, and the calibration calculation unit performs the second calibration using the feature point positions of the scaled deformed pattern included in the second image taken by the camera.

[0067] (Appendix 4) The output unit is an information processing device described in any one of Appendices 1 to 3, which scales the deformed pattern in a display area where the deformed pattern is displayed so that the entire deformed pattern is displayed at its maximum within the display area, and outputs the scaled deformed pattern.

[0068] (Appendix 5) The information processing device is described in any of Appendices 1 to 4, wherein the output unit displays the flat pattern of a first size, which is larger than the display area, in the entire display area where the flat pattern is displayed, and the calibration calculation unit performs the first calibration using the first image captured by the camera and including the flat pattern of the first size.

[0069] (Appendix 6) The information processing device according to any one of Appendices 1 to 5, wherein, when the interval between feature points detected on the second image is shorter than a first threshold, the calibration calculation unit performs the second calibration by having the output unit output an enlarged version of the deformed pattern and then re-acquiring the second image, and when the interval between feature points detected on the second image is longer than a second threshold, the calibration calculation unit performs the second calibration by having the output unit output a reduced version of the deformed pattern and then re-acquiring the second image, and the second threshold is greater than the first threshold.

[0070] (Appendix 7) The information processing device described in any of Appendices 1 to 6 further includes a correspondence calculation unit that calculates at least one of the first correspondence relationship or the second correspondence relationship in at least one of the cases where a first correspondence relationship is not set, which is a correspondence relationship between the feature point positions of the flat pattern in the first image and the feature point positions of the flat pattern on the world coordinates, or where a second correspondence relationship is not set, which is a correspondence relationship between the feature point positions of the deformed pattern in the second image and the feature point positions of the deformed pattern on the world coordinates.

[0071] (Appendix 8) The information processing device is one described in any one of Appendices 1 to 7, wherein when the output unit outputs the deformed pattern and performs the second calibration, the calibration calculation unit uses the parameters calculated by the first calibration as initial values ​​of the estimated parameters of the camera.

[0072] (Appendix 9) This method is executed by an information processing device, which performs a first calibration of the camera based on the positions of feature points of a flat pattern contained in a first image captured by the camera and the positions of feature points of the flat pattern on world coordinates, outputs a deformed pattern by deforming the flat pattern so that the flat pattern is captured facing forward relative to the camera based on the first calibration, obtains an image of the deformed pattern captured by the camera as a second image, and performs a second calibration of the camera based on the positions of feature points of the deformed pattern in the second image and the positions of feature points of the deformed pattern on world coordinates.

[0073] (Appendix 10) This is a program that causes a computer to perform a first calibration of the camera based on the positions of feature points of a flat pattern included in a first image captured by the camera and the positions of feature points of the flat pattern on world coordinates; based on the first calibration, output a deformed pattern by deforming the flat pattern so that the flat pattern is captured facing forward relative to the camera; obtain an image of the deformed pattern captured by the camera as a second image; and perform a second calibration of the camera based on the positions of feature points of the deformed pattern in the second image and the positions of feature points of the deformed pattern on world coordinates.

[0074] Furthermore, some or all of the configurations described in Supplementary Notes 2 to 8 that are dependent on the information processing device of Supplementary Note 1 above may also be dependent on Supplementary Note 9 (method) and Supplementary Note 10 (program) in the same manner as Supplementary Note 2 to Supplementary Note 8. Furthermore, not limited to Supplementary Notes 1 to 8, Supplementary Note 9, and Supplementary Note 10, some or all of the configurations described as Supplements may be similarly made dependent on various hardware, software, various recording means for recording software, or systems within the scope of the above-mentioned embodiments.

[0075] This disclosure may be modified as appropriate within the scope that does not contradict the gist or idea of ​​the invention that can be read from the claims and the entire specification, and information processing devices, information processing methods, and programs that involve such modifications are also included in the technical idea of ​​this disclosure.

[0076] Each embodiment can be combined with other embodiments as appropriate, and the present disclosure is not limited to the described embodiments. [Explanation of symbols]

[0077] 1 processor 2 RAM 3 ROM 4 Storage device 5 Input Devices 6 Output Devices 7 Data Bus 10. Information processing equipment 11 Camera 100 Information Processing Systems 101 Calibration Calculation Unit 102 Output section 103 Display section 104 Photography Department 201 Correspondence Calculation Unit

Claims

1. a calibration calculation unit that performs a first calibration of the camera based on feature point positions of a flat pattern included in a first image captured by the camera and feature point positions of the flat pattern on world coordinates; an output unit that outputs a deformed pattern obtained by deforming the flat pattern based on the first calibration so that the flat pattern is photographed in front of the camera; Equipped with The calibration calculation unit acquires an image of the deformed pattern photographed by the camera as a second image, and performs a second calibration of the camera based on the positions of feature points of the deformed pattern in the second image and the positions of feature points of the deformed pattern on the world coordinate system. Information processing device.

2. when the calibration calculation unit cannot detect feature points on the first image required for performing the first calibration, the output unit outputs the flat pattern in a display area capable of displaying the flat pattern such that the flat pattern obtained by scaling the flat pattern displayed when photographed by the camera is displayed; The calibration calculation unit performs the first calibration using feature point positions of the scale-converted flat pattern included in the first image captured by the camera. The information processing device according to claim 1 .

3. when the calibration calculation unit cannot detect feature points on the second image required for performing the second calibration, the output unit outputs the deformed pattern such that the deformed pattern obtained by scaling the deformed pattern displayed at the time of photographing by the camera is displayed in a display area capable of displaying the deformed pattern, The calibration calculation unit performs the second calibration using feature point positions of the scale-converted deformed pattern included in the second image captured by the camera.

3. The information processing device according to claim 1 or 2.

4. The output unit scales the deformed pattern in a display area where the deformed pattern is displayed so that the entire deformed pattern is displayed at its maximum size within the display area, and outputs the scaled deformed pattern.

3. The information processing device according to claim 1 or 2.

5. the output unit displays the flat pattern in a first size that is larger than the display area in a display area where the flat pattern is displayed, over the entire display area; The calibration calculation unit performs the first calibration using the first image captured by the camera and including the flat pattern of the first size.

3. The information processing device according to claim 1 or 2.

6. The calibration calculation unit When an interval between feature points detected on the second image is shorter than a first threshold, the second calibration is performed by outputting the enlarged deformed pattern to the output unit and then re-acquiring the second image; If the interval between the feature points detected on the second image is longer than a second threshold, the output unit outputs the reduced deformed pattern, and then the second image is reacquired, thereby performing the second calibration; The second threshold is greater than the first threshold.

3. The information processing device according to claim 1 or 2.

7. and a correspondence relationship calculation unit that calculates at least one of the first correspondence relationship and the second correspondence relationship when a first correspondence relationship between feature point positions of the flat pattern in the first image and feature point positions of the flat pattern on the world coordinate system is not set, or when a second correspondence relationship between feature point positions of the deformed pattern in the second image and feature point positions of the deformed pattern on the world coordinate system is not set.

3. The information processing device according to claim 1 or 2.

8. When performing the second calibration after the output unit outputs the deformed pattern, the calibration calculation unit uses the parameters calculated by the first calibration as initial values ​​of the estimated parameters of the camera.

3. The information processing device according to claim 1 or 2.

9. performing a first calibration of the camera based on the positions of feature points of a flat pattern included in a first image captured by the camera and the positions of feature points of the flat pattern on world coordinates; outputting a deformed pattern obtained by deforming the flat pattern based on the first calibration so that the flat pattern is photographed from the front with respect to the camera; an image of the deformed pattern captured by the camera is acquired as a second image; A second calibration of the camera is performed based on the feature point positions of the deformed pattern in the second image and the feature point positions of the deformed pattern on the world coordinate system. A method executed by an information processing device.

10. performing a first calibration of the camera based on the positions of feature points of a flat pattern included in a first image captured by the camera and the positions of feature points of the flat pattern on world coordinates; outputting a deformed pattern obtained by deforming the flat pattern based on the first calibration so that the flat pattern is photographed from the front with respect to the camera; an image of the deformed pattern captured by the camera is acquired as a second image; A second calibration of the camera is performed based on the feature point positions of the deformed pattern in the second image and the feature point positions of the deformed pattern on the world coordinate system. A program that makes a computer do something.

Images