Imaging device calibration apparatus, imaging device calibration method, and recording medium

Abstract

The present disclosure relates to an imaging device calibration apparatus, an imaging device calibration method, and a recording medium, which enable calibration to be performed reliably and with increased accuracy. There is provided an imaging device calibration apparatus including: an image acquisition unit that acquires, from an imaging device, a captured image indicating a graph pattern displayed in a time division manner on a display panel inclined with respect to an optical axis direction of the imaging device; a feature point information generation unit, a method for capturing an image by integrating, on a single plane, feature points at a first time of a graph pattern included in an image frame at a first time at which an image is captured, and a graph pattern at a second time including an arrangement of a set of feature points included in an image frame at a second time at which an image is captured and different from the graph pattern at the first time generating integrated feature point information from the feature points at the second time; and a parameter acquisition unit that acquires a parameter for calibrating the imaging apparatus based on the integrated feature point information.

Description

Technical Field

[0001] The present disclosure relates to an image pickup device calibration apparatus, an image pickup device calibration method, and a recording medium, and particularly to an image pickup device calibration apparatus, an image pickup device calibration method, and a recording medium that allow calibration with higher accuracy to be reliably performed. Background Art

[0002] The technology for acquiring the position and movement of a target object or a camera device itself and identifying the attributes of the target object by detecting and analyzing images in a captured image taken by the camera device has been widely introduced into systems including security cameras, vehicle-mounted cameras, cameras installed on robots, etc., and cameras installed on game equipment and information terminals.

[0003] In order to ensure the processing accuracy in these systems, a calibration process of acquiring device-specific information such as internal parameters of the imaging device, distortion correction coefficients, and external parameters is performed in advance. A well-known calibration technique is Zhang's method, in which a planar chart of a checkered pattern is imaged from multiple viewpoints, and parameters are optimized so that the positions of feature points on the captured image correctly correspond to the positions of the feature points on the plane of the chart in real space (see Non-Patent Document 1).

[0004] Zhang's method requires setting up the imaging device and chart at several positions and postures and then repeating the imaging, which has problems such as large workload. In addition, a simple calibration method is unlikely to achieve the same level of accuracy as Zhang's method. In this regard, for example, the technology disclosed in Patent Document 1 is proposed.

[0005] Patent document 1 discloses a technology for performing calibration of an imaging device with a small number of steps and with high accuracy by acquiring the same information as that obtained by imaging one plane from different viewpoints in one imaging by using a chart including a plurality of planes forming a predetermined angle. The pattern shown on the chart is designed to obtain an orthogonal and uniform feature point arrangement when imaging in a state inclined relative to an imaging surface.

[0006] Citation List

[0007] Patent Literature

[0008] Patent Document 1: Japanese Patent No. 6764533

[0009] Non-patent literature

[0010] Non-patent document 1: Zhengyou Zhang, “A Flexible New Technique for Camera Calibration,” Microsoft Research Technical Report, MSR-TR-98-71, December 2, 1998. Summary of the invention

[0011] Technical issues

[0012] However, the number and accuracy of feature point coordinate groups that can be obtained from a captured image obtained in one imaging differ depending on the resolution and viewing angle of the camera, so there is a possibility that high-precision calibration cannot be performed.

[0013] The present disclosure has been made in view of the above circumstances, and an object thereof is to enable reliable execution of higher-precision calibration.

[0014] Solution to the problem

[0015] According to one aspect of the present disclosure, a camera calibration device is a camera calibration device including: an image acquisition unit that acquires, from a camera, a captured image indicating a chart pattern displayed in a time-division manner on a display panel tilted relative to an optical axis direction of the camera; a feature point information generation unit that generates integrated feature point information by integrating feature points of the chart pattern at a first time and feature points of the chart pattern at a second time on a single plane, the chart pattern at the first time being included in an image frame at the first time of the captured image, the chart pattern at the second time being included in the image frame at the second time of the captured image and having an arrangement of a feature point group different from that of the feature point group of the chart pattern at the first time; and a parameter acquisition unit that acquires parameters for calibration of the camera based on the integrated feature point information.

[0016] A camera calibration method according to one aspect of the present disclosure is a camera calibration method comprising: acquiring, from a camera, a captured image indicating a chart pattern displayed in a time-division manner on a display panel tilted relative to an optical axis direction of the camera; generating integrated feature point information by integrating feature points of the chart pattern at a first time at a first time and feature points of the chart pattern at a second time at a second time on a single plane, the chart pattern at the first time being included in an image frame at the first time of the captured image, the chart pattern at the second time being included in the image frame at the second time of the captured image and having an arrangement of a feature point group different from the arrangement of the feature point group of the chart pattern at the first time; and acquiring parameters for calibration of the camera based on the integrated feature point information.

[0017] A recording medium according to one aspect of the present disclosure is a recording medium that records a program that includes instructions that cause a computer to perform the following operations: obtain, from an imaging device, a captured image indicating a chart pattern displayed in a time-division manner on a display panel tilted relative to an optical axis direction of the imaging device; generate integrated feature point information by integrating feature points of the chart pattern at a first time at a first time and feature points of the chart pattern at a second time at a second time on a single plane, the chart pattern at the first time being included in an image frame at the first time of the captured image, the chart pattern at the second time being included in the image frame at the second time of the captured image and having an arrangement of a feature point group that is different from the arrangement of the feature point group of the chart pattern at the first time; and obtain parameters for calibration of the imaging device based on the integrated feature point information.

[0018] In a camera calibration apparatus, a camera calibration method, and a recording medium according to one aspect of the present disclosure, a captured image indicating a chart pattern displayed in a time-division manner on a display panel tilted relative to an optical axis direction of the camera is obtained from the camera; integrated feature point information is generated by integrating feature points of the chart pattern at a first time at a first time and feature points of the chart pattern at a second time at a second time on a single plane, the chart pattern at the first time being included in an image frame at the first time of the captured image, the chart pattern at the second time being included in an image frame at the second time of the captured image and having an arrangement of a feature point group different from the arrangement of the feature point group of the chart pattern at the first time; and parameters for calibration of the camera are acquired based on the integrated feature point information.

[0019] Note that the camera calibration apparatus according to an aspect of the present disclosure may be an independent apparatus, or may be an internal block constituting a single apparatus. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a diagram showing an example of an external configuration of a calibration system to which the present disclosure is applied.

[0021] Figure 2 yes Figure 1 Top view of the calibration system in Figure 2.

[0022] Figure 3 is a block diagram showing a configuration example of a calibration system to which the present disclosure is applied.

[0023] Figure 4 It is shown Figure 3 A block diagram of a configuration example of a camera parameter estimation unit in FIG.

[0024] Figure 5 This is a flowchart describing the flow of processing for generating and displaying a graph pattern.

[0025] Figure 6 is a diagram showing an example of a chart pattern image.

[0026] Figure 7 It is a diagram showing an example of an ID correspondence table.

[0027] Figure 8 is a diagram showing an example of a display sequence of a chart pattern image.

[0028] Fig. 9 is a diagram showing an example of a two-dimensional corner position on a chart pattern image.

[0029] Fig.10 is a diagram showing an example of a conversion table.

[0030] Fig.11 is a flowchart describing the flow of the camera calibration process.

[0031] Fig.12 is a diagram showing an example of deflection correction data.

[0032] Fig.13 is a diagram showing an example of a coordinate system in deflection correction.

[0033] Fig.14 is a diagram schematically showing a deflection correction map.

[0034] Fig.15 is a diagram showing an example of a case where bilinear interpolation is performed as the interpolation process.

[0035] Fig.16 : is a flowchart describing the detailed flow of the camera parameter estimation process.

[0036] Fig.17 : is a flowchart describing the detailed flow of the feature point information generation process.

[0037] Fig.18 Schematically shows the processing contents of feature point information generation processing.

[0038] Fig.19 Schematically shows the processing result of the feature point information generation processing.

[0039] Fig. 20 is a diagram showing a configuration example of a system during chart pattern calibration.

[0040] Fig.21 is a flowchart describing the flow of the chart pattern calibration process.

[0041] Fig. 22 is a diagram showing an example of the correspondence relationship between the original object coordinates and the new object coordinates.

[0042] Fig.23 is a diagram showing an example of the correction amount.

[0043] Fig.24 is a top view illustrating a first example of multi-scene imaging.

[0044] Fig.25 is a top view illustrating a second example of multi-scene imaging.

[0045] Fig.26 is a block diagram showing another configuration example of a calibration system to which the present disclosure is applied.

[0046] Fig. 27 is a block diagram showing another configuration example of a calibration system to which the present disclosure is applied.

[0047] Fig.28 is a block diagram showing a configuration example of a computer. DETAILED DESCRIPTION

[0048] <External Configuration>

[0049] Figure 1 is a diagram showing an example of an external configuration of a calibration system to which the present disclosure is applied. Figure 2 yes Figure 1 A top view of the calibration system in FIG. 1 is provided, and the description will be made with reference thereto as appropriate. A system refers to a logical collection of multiple devices.

[0050] exist Figure 1In the present invention, the calibration system includes an imaging device 11 to be calibrated, a display device 12-1 that displays a chart pattern image PI1, a display device 12-2 that displays a chart pattern image PI2, a light source 13-1 that applies light to a display panel 12A-1 of the display device 12-1, and a light source 13-2 that applies light to a display panel 12A-2 of the display device 12-2.

[0051] The imaging device 11 includes at least one camera. For example, the camera included in the imaging device 11 may be a stereo camera in which two cameras are arranged at a known distance to the left and right. Alternatively, the imaging device 11 may include three or more cameras. What configuration the camera included in the imaging device 11 has is determined according to the content of information processing performed using the imaging device 11. In the following, the data of the captured image is simply referred to as a captured image in some cases.

[0052] For example, in the imaging device 11, a camera device (RGB camera device) having a visible light sensor including an image sensor such as a CMOS (complementary metal oxide semiconductor) and a CCD (charge coupled device), and a signal processing circuit that performs signal processing on an image signal obtained by the visible light sensor to generate data of a captured image (RGB image) can be used.

[0053] In the imaging device 11, it is not necessary to use a camera including a visible light sensor, and a non-visible light camera including a non-visible light sensor for acquiring spatial information and time information may be used. For example, a ToF camera including a ToF (time of flight) sensor may be used as the non-visible light camera. An example of a ToF camera is a depth camera of a direct ToF (time of flight) method using a SPAD element. The ToF camera is generally configured to acquire an infrared light image (IR image) having a lower resolution than that of an RGB camera. Even in the case of using such a low-resolution infrared light image, the camera calibration in the present disclosure can be appropriately performed.

[0054] For example, Figure 2 As shown, the imaging device 11 includes camera devices 11A-1 to 11A-3. The camera device 11A-1 and the camera device 11A-2 are each an RGB camera device including a visible light sensor. The camera device 11A-3 is a ToF camera device including a ToF sensor. Figure 2 In the example of FIG. 1 , the camera 11A- 3 is provided between the camera 11A- 1 and the camera 11A- 2 which are configured as a stereo camera.

[0055] For example, the resolution of the captured image generated by each of the camera 11A-1 and the camera 11A-2 is 1280×960. The resolution of the infrared light image generated by the camera 11A-3 is 160×120. When these images are compared with each other, the former is a high-resolution image, and the latter is a low-resolution image. In this way, in the imaging device 11, the resolution of the image to be generated is different for each camera 11A in some cases. In addition, in some cases, the distortion shape and the viewing angle of the lens are different for each camera 11A.

[0056] The display device 12-1 and the display device 12-2 are arranged to form a predetermined angle with respect to the imaging device 11. For example, Figure 2 As shown, the display device 12-1 and the display device 12-2 are set to be erected in a manner to form an angle θ (0°<θ<180°) with the side obtained by matching the sides of two display panels 12A-1 and 12A-2 of the same area as the axis. Taking into account the viewing angle, resolution, etc. of the camera device to be calibrated, the angle θ can be set to approximately 60° to 90°. The jig 14 fixes the relative positional relationship between the imaging device 11 and the display devices 12-1 and 12-2, so that the display surfaces of the display panels 12A-1 and 12A-2 are inclined relative to the optical axis direction of each camera device 11A of the imaging device 11, and the imaging device 11 has a position and posture at a predetermined distance from the display panels 12A-1 and 12A-2.

[0057] The display device 12-1 includes electronic paper or the like, and includes a display panel 12A-1. The display panel 12A-1 displays a chart pattern image PI1 for calibration. The display device 12-2 includes electronic paper or the like, and includes a display panel 12A-2. The display panel 12A-2 displays a chart pattern image PI2 for calibration. In addition, in the case where the pigment of the electronic paper corresponds to the wavelength of infrared light, the combination of the electronic paper and the IR light source allows the calibration in the present disclosure to be performed based on an infrared light image obtained from a ToF camera or the like.

[0058] The chart pattern image PI1 includes an image of a chart pattern, such as a checkered pattern of black and white between adjacent rectangles in a rectangle divided into a grid pattern. The chart pattern image PI2 includes an image of a chart pattern, such as a checkered pattern, similarly to the chart pattern image PI1. The chart pattern included in the chart pattern image PI1 displayed on the display panel 12A-1 and the chart pattern included in the chart pattern image PI2 displayed on the display panel 12A-2 change over time.

[0059] The light source 13-1 and the light source 13-2 apply light of a wavelength band corresponding to the optical sensor included in the camera 11A. The light source 13-1 is set at a position where the display panel 12A-1 can be uniformly irradiated with light and no reflection occurs. The light source 13-2 is set at a position where the display panel 12A-2 can be uniformly irradiated with light and no reflection occurs. Note that each of the display devices 12-1 and 12-2 does not necessarily have to include electronic paper, and may include an LCD (liquid crystal display) or the like. The light sources 13-1 and 13-2 are set according to the configuration of the display devices 12-1 and 12-2, but do not necessarily have to be set according to the configuration.

[0060] Each camera 11A of the imaging device 11 acquires the chart pattern image PI1 displayed on the display panel 12A-1 and the chart pattern image PI2 displayed on the display panel 12A-2. The captured images are provided to the camera calibration device 22 (described later). Figure 3 ), and is used for camera calibration processing. In the camera calibration processing, calculations for calibration are performed so that images of feature points in captured images appear at positions reflecting original positions in three-dimensional space, and camera parameters such as internal parameters and external parameters are obtained. The internal parameters include, for example, the optical center and focal length of individual cameras, the shape of lens distortion, etc., and may also include distortion correction coefficients.

[0061] In the present disclosure, captured images that can be used for camera calibration include not only RGB images acquired by a visible light sensor, but also non-visible light images acquired by a non-visible light camera, more specifically, an infrared light camera. The non-visible light image may include an infrared light image (IR image) acquired by the camera 11A (ToF camera) including a ToF sensor.

[0062] <System Configuration>

[0063] Figure 3 is a block diagram showing a configuration example of a calibration system to which the present disclosure is applied. Figure 3 In the present invention, the calibration system includes, in addition to the imaging device 11 and the display devices 12-1 and 12-2, a chart pattern generating device 21 that generates a chart pattern image and various tables and a camera calibration apparatus 22 that performs calculations for calibration.

[0064] The chart pattern generating device 21 includes a PC (Personal Computer), a server, a dedicated device, etc. The chart pattern generating device 21 is connected to the display devices 12 - 1 and 12 - 2 and the camera calibration device 22 via a predetermined interface. The chart pattern generating device 21 includes a generating unit 101 and a display control unit 102 .

[0065] The generation unit 101 generates a chart pattern image representing a chart pattern, and supplies the generated chart pattern image to the display control unit 102. The chart pattern includes a pattern such as a checkered pattern and a mark. The display control unit 102 displays the chart pattern image supplied from the generation unit 101 on the display device 12-1 and the display device 12-2.

[0066] When the chart pattern image is generated, the chart pattern image is generated based on the internal parameters input to the chart pattern image. The internal parameters include information related to each camera 11A included in the imaging device 11 to be calibrated. For example, the internal parameters include information related to the resolution of the captured image corresponding to the number of pixels of the optical sensor included in the camera 11A and the angle of view of the camera 11A.

[0067] The generation unit 101 generates a chart pattern image representing a chart pattern corresponding to at least one of the resolution of the captured image corresponding to the number of pixels of the optical sensor included in each camera 11A or the angle of view of the camera 11A based on the internal parameters. In this way, a chart pattern image optimized for each camera 11A is generated. More specifically, the chart pattern image shows a pattern in which the corner interval per pixel is equal to or greater than a certain interval, and the corner interval can change over time. Therefore, even when calibrating a plurality of camera 11A in which at least one of the resolution or the angle of view is different, each camera 11A can acquire an image of at least one chart pattern corresponding to its own sub-pixel accuracy. Therefore, the detection of feature points (corners) by low-resolution camera devices is ensured. The increase in the density and accuracy of corner detection acquired by low-resolution camera devices will be described in detail below. In addition, the density of the chart pattern can also be changed according to the distortion shape of the lens included in the camera 11A by making the internal parameters include information related to the distortion shape of the lens included in the camera 11A.

[0068] In addition to generating a chart pattern image, the generation unit 101 also generates an ID correspondence table and a conversion table, and outputs them to the camera calibration device 22. The ID correspondence table is a table (correspondence information) that associates a tag ID, a panel ID, and a group ID (group ID) with each other. The tag ID is an ID for identifying a chart pattern. A tag embedded with a tag ID is added to a chart pattern represented by a chart pattern image. The panel ID is an ID for identifying a display panel 12A. The group ID is an ID for identifying a group corresponding to a position / posture relationship (hereinafter, referred to as a position / posture relationship) indicating at least one of a relative position or a relative posture between the camera 11A and the display panel 12A (chart pattern displayed on the display panel 12A).

[0069] The conversion table is a table (conversion information) for converting a two-dimensional index representing the positional relationship of the feature points of the chart pattern into known position coordinates of the 3D object. The two-dimensional index represents the position of each corner in the checkered pattern by a two-dimensional integer matrix. Here, the position coordinates of the 3D object correspond to the position coordinates of the feature points on the display surface of the display panel 12A. That is, the position coordinates of the feature points in the chart pattern included in the chart pattern image displayed on the display panel 12A correspond to the position coordinates of the feature points of the chart pattern as an actual object.

[0070] The camera calibration device 22 includes a PC, a server, a dedicated device, etc. The camera calibration device 22 is connected to the chart pattern generating device 21 via a predetermined interface. The camera calibration device 22 includes a table storage unit 201, a deflection correction data storage unit 202, a deflection correction unit 203, a table storage unit 204, a camera parameter estimation unit 205, and a camera parameter storage unit 206.

[0071] The table storage unit 201 records the ID correspondence table and the conversion table output from the chart pattern generating device 21. The deflection correction data storage unit 202 records the deflection correction data. The deflection correction data is data including a correction amount corresponding to a deformation amount compared to an ideal state with respect to the display surface of the display panel 12A of the display device 12.

[0072] The deflection correction unit 203 acquires the conversion table recorded on the table storage unit 201 and the deflection correction data recorded on the deflection correction data storage unit 202. The deflection correction unit 203 corrects the conversion table based on the deflection correction data and records the corrected conversion table in the table storage unit 204.

[0073] The camera parameter estimation unit 205 acquires the captured image output from the imaging device 11 and the corrected conversion table recorded in the table storage unit 204. The camera parameter estimation unit 205 detects the image of the feature point of the chart pattern from the captured image, and generates feature point information that associates the position coordinates of the image of the feature point and the position coordinates of the feature point on the display surface of the display panel 12A with each other. When generating the feature point information, the corrected conversion table is used. The camera parameter estimation unit 205 performs an operation for calibration based on the generated feature point information, acquires a predetermined camera parameter, and records it in the camera parameter storage unit 206.

[0074] The camera parameter estimation unit 205 can acquire the ID correspondence table recorded in the table storage unit 201, and perform processing using the ID correspondence table. For example, by using the ID correspondence table, the panel ID can be identified based on the tag ID obtained from the image of the tag included in the captured image, and the display panel 12A displaying the chart pattern image can be identified. In addition, by using the ID correspondence table, a plurality of group IDs can be identified in a distinguished manner based on the tag ID obtained from the captured image, and feature points of a plurality of image frames of the captured image corresponding to the same group ID and the same panel ID can be integrated onto the display surface of the corresponding display panel 12A.

[0075] Figure 4 2 is a block diagram showing a configuration example of the camera parameter estimation unit 205. Figure 4 In the embodiment, the camera parameter estimation unit 205 includes an image acquisition unit 221, a feature point information generation unit 222, and a camera parameter acquisition unit 223.

[0076] The image acquisition unit 221 acquires a captured image from the imaging device 11, and supplies the acquired captured image to the feature point information generation unit 222. In the case where the imaging device 11 includes the camera devices 11A-1 to 11A-3, the optical sensor included in each camera device 11A acquires a captured image including an image of a chart pattern displayed on the display panels 12A-1 and 12A-2.

[0077] The feature point information generating unit 222 detects the image of the feature point and the mark from the captured image, and for each display surface of the display panel 12A displaying the chart pattern identified by the mark, associates the two-dimensional position coordinates (two-dimensional coordinates) of the image of the detected feature point with the three-dimensional position coordinates (three-dimensional coordinates) of the feature point on the display surface of the display panel 12A. At this time, for each feature point identification index, the position coordinates on the display surface of the display panel 12A, that is, the three-dimensional coordinates on the display surface are identified according to the index using the corrected conversion table recorded on the table storage unit 204. This allows the two-dimensional coordinates in the captured image to be associated with the three-dimensional coordinates on the display plane of the display panel 12A for each feature point. Feature point information including the corresponding information of the feature point obtained in this way is provided to the camera parameter acquiring unit 223. For example, in the case of including a checkered pattern as a chart pattern, the two-dimensional angular position in the checkered pattern can be used as a feature point.

[0078] The camera parameter acquisition unit 223 uses the corresponding information of the feature points included in the feature point information from the feature point information generation unit 222 as input data to specify and acquire camera parameters such as internal parameters and external parameters using an existing algorithm. The acquired camera parameters are recorded on the camera parameter storage unit 206.

[0079] Note that Figure 3 In the embodiment, the chart pattern generating device 21, the camera calibration device 22, and the display devices 12-1 and 12-2 can be connected via an interface corresponding to a predetermined communication method such as a wired method and a wireless method, and various types of data can be exchanged. Alternatively, the chart pattern image generated by the chart pattern generating device 21 can be provided to the display devices 12-1 and 12-2, or the ID correspondence table and the conversion table can be provided to the camera calibration device 22 via a recording medium such as a memory card. In the former case, the display control unit 102 is set on the display devices 12-1 and 12-2 side. Although Figures 1 to 3 A configuration in which two display devices 12 , namely, the display device 12 - 1 and the display device 12 - 2 are provided is shown, but one display device 12 including the display panel 12A- 1 and the display panel 12A- 2 may be provided.

[0080] Each block in the chart pattern generating device 21 and the camera calibration device 22 is implemented by hardware or software. For example, in the chart pattern generating device 21, the generating unit 101 and the display control unit 102 are implemented by a processor such as a CPU (Central Processing Unit) executing a program. In the camera calibration device 22, the deflection correction unit 203 and the camera parameter estimation unit 205 are implemented by a processor such as a CPU executing a program. In addition, in the camera calibration device 22, the table storage unit 201, the deflection correction data storage unit 202, the table storage unit 204, and the camera parameter storage unit 206 are implemented by a storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), an SSD (Solid State Drive), and an HDD (Hard Disk Drive). Note that the table storage unit 201, the deflection correction data storage unit 202, the table storage unit 204, and the camera parameter storage unit 206 may include the same storage device or different storage devices. The ID correspondence table and the conversion table may be recorded on different storage devices.

[0081] <Graph pattern generation / display>

[0082] Details of generation and display of a chart pattern performed by the chart pattern generating means 21 will be described. Figure 5 This is a flowchart describing the flow of processing for generating and displaying a graph pattern.

[0083] In step S11, the generation unit 101 generates a chart pattern image including an image of a chart pattern based on the internal parameters. The chart pattern image is displayed on each of the display device 12-1 and the display device 12-2 while changing over time. Therefore, the generation unit 101 generates a plurality of types of chart pattern images having different chart patterns for each display device 12.

[0084] Figure 6 is a diagram showing an example of a chart pattern image. Figure 6 As shown, a checkered pattern is set on the entire chart pattern image PI, and a marker M is set in a specific area. In the marker M, a marker ID that is an identification ID unique to the chart pattern is embedded. As the marker M, an AR marker can be used. For example, as an AR marker, an ARUCO marker that can be used in OpenCV, which is a representative open source image processing software library, etc. can be used. The ARUCO marker is described in the following document A, for example.

[0085] Reference A: S. Garrido-Jurado, R. Munoz-Salinas, F. J. Madrid-Cuevas, and M. J. Marin-Jimenez, “Automatic generation and detection of highly reliable fiducialmarkers under occlusion,” Pattern Recognition, Vol. 47, No. 6, June 2014, pp. 2280–2292.

[0086] In the chart pattern included in the chart pattern image PI, the feature point is the vertex (corner) of each rectangle. Note that the chart pattern represented by the chart pattern image PI is not limited to a checkered pattern (chessboard), and the feature points only need to be distributed in a shape or arrangement that is easily detected using an existing search method. For example, by adopting a feature point arrangement that forms an orthogonal system, such as a circular grid in which black circles are arranged vertically and horizontally, it is possible to make the detection of feature points and the acquisition of attributes easier. It is suitable to enable a pattern that can easily detect feature points with sub-pixel accuracy of the display panel 12A.

[0087] Refer again Figure 5, in step S12, the generation unit 101 generates an ID correspondence table. In the ID correspondence table, the panel ID (panel_id) and the group ID (frame_group_id) are associated with the marker ID (marker_id). The panel ID is an ID for identifying the display panel 12A of the display device 12. For example, the display panel 12A-1 is represented by panel_id=1, and the display panel 12A-2 is represented by panel_id=2. The group ID is an identification ID for determining whether the position / posture relationship between the camera 11A included in the imaging device 11 and the chart pattern image PI displayed on the display device 12 is the same. The group ID is assigned to each chart pattern image PI (each image frame of the chart pattern image PI) according to the position and posture of the camera 11A.

[0088] Figure 7 : is a diagram showing an example of an ID correspondence table (marker_id_table). Figure 7 In the ID correspondence table in , the panel ID and the group ID are associated with the tag ID, and the tag ID can be converted into the panel ID and the group ID. The combination of the panel ID and the group ID is the plane ID (plane_id).

[0089] Figure 8 An example of a display sequence of a chart pattern image is shown. Figure 8 , the time axis extends from top to bottom in the figure, and shows the chart pattern image PI1 displayed on the display panel 12A-1 of the display device 12-1 and the chart pattern image PI2 displayed on the display panel 12A-2 of the display device 12-2 in time sequence.

[0090] At time t=0, a chart pattern image PI10 including a marker M10 embedded with marker_id=0 is displayed on the display panel 12A-1, and a chart pattern image PI20 including a marker M20 embedded with marker_id=1 is displayed on the display panel 12A-2. Figure 7 In the ID correspondence table, panel_id=1 is associated with marker_id=0, and panel_id=2 is associated with marker_id=1 (the row of "t=0").

[0091] At time t=1 to 4, chart pattern images PI11 to PI14 having markers with marker_id=2, 4, 6, and 8 embedded therein are sequentially displayed on the display panel 12A-1. At time t=1 to 4, chart pattern images PI21 to PI24 having markers with marker_id=3, 5, 7, and 10 embedded therein are sequentially displayed on the display panel 12A-2. Figure 7In the ID correspondence table in , panel_id=1 is associated with marker_id=2, 4, 6, and 8, and panel_id=2 is associated with marker_id=3, 5, 7, and 9 (rows of "t=1 to 4").

[0092] exist Figure 8 , the density of the chart pattern is different between the chart pattern images PI10 and PI20 displayed on the display panels 12A-1 and 12A-2 at time t=0 and the chart pattern images PI11 to PI14 and PI21 to PI24 displayed on the display panels 12A-1 and 12A-2 at time t=1 to 4. Therefore, the sub-pixel accuracy of each camera 11A of the imaging device 11 corresponds to one of the chart pattern images with different feature point group densities. The chart pattern images PI10 and PI20 can each be regarded as a high-resolution chart pattern with a relatively high feature point group density. In addition, the chart pattern images PI11 to PI14 and PI21 to PI24 can each be regarded as a low-resolution chart pattern with a relatively low feature point group density. For example, calibration of a non-visible light sensor is performed based on a low-resolution chart pattern with a low chart pattern density, and the non-visible light sensor has fewer pixels arranged on a light receiving surface in two dimensions than a visible light sensor.

[0093] For example, the camera 11-1 and the camera 11-2 are each an RGB camera including a visible light sensor, and generate a high-resolution captured image. Therefore, at time t=0, the camera 11-1 and the camera 11-2 only need the chart pattern images PI10 and PI20 with a high density of feature point groups of the chart pattern. That is, the chart pattern images PI10 and PI20 displayed at time t=0 each include a high-resolution chart pattern.

[0094] Meanwhile, the camera 11-3 is a ToF camera including a ToF sensor, and generates a captured image with a low resolution. Therefore, at time t=1 to 4, the camera 11-3 sequentially acquires images of chart pattern images PI11 to PI14 and PI21 to PI24 with low density of feature point groups of the chart pattern. That is, the chart pattern images PI11 to PI14 and PI21 to PI24 displayed at time t=1 to 4 each include a low-resolution chart pattern. Therefore, during the time period of time t=1 to 4, the low-resolution camera is able to acquire a captured image of a low-resolution chart pattern corresponding to sub-pixel accuracy, and is able to reliably detect feature points (corners). Meanwhile, since the density of the feature point group of each low-resolution chart pattern is low, in some cases the information of the feature points used for calibration cannot be fully obtained from one low-resolution chart pattern. Therefore, by integrating different low-resolution chart pattern images PI11 to PI14 and PI21 to PI24 on a single plane (the same plane), the density of the feature point group is increased. In the present disclosure, information of feature points integrated on the same plane is referred to as integrated feature point information in some cases.

[0095] The density of the feature point group of the chart pattern may vary according to the viewing angle of the camera 11A rather than according to the resolution of the captured image generated by the camera 11A. For example, in the case where the camera 11A has a wide viewing angle, only the chart pattern images PI11 to PI14 and PI21 to PI24 having a low density of the chart pattern need to be photographed. Meanwhile, in the case where the camera 11A has a narrow viewing angle, only the chart pattern images PI10 and PI20 having a high density of the feature point group of the chart pattern need to be photographed. Furthermore, the density of the feature point group of the chart pattern may also vary according to the distortion shape of the lens included in the camera 11A. At this time, the imaging device 11 does not necessarily need to include a plurality of cameras 11A, and a chart pattern that changes over time may be imaged by one camera 11A.

[0096] Although in Figure 8 Although not shown in the display sequence of , it is assumed that after time t5, the position / posture relationship between the camera devices 11-1 to 11-3 and the chart pattern image PI changes due to a change in the position where the imaging device 11 is set. In this case, the chart pattern image PI15 with a marker having marker_id=10 embedded therein is displayed on the display panel 12A-1, and the chart pattern image PI25 with a marker having marker_id=11 embedded therein is displayed on the display panel 12A-2.

[0097] At this time, Figure 7In the ID correspondence table, panel_id=1 and frame_group_id=1 are associated with marker_id=10, and panel_id=2 and frame_group_id=1 are associated with marker_id=11 (row of "t=5"). That is, frame_group_id is different between marker_id=0 to 9 and marker_id=10 to 11. In this way, the group ID can be used to determine whether a feature point (corner) is located on a chart pattern image captured by a camera 11A having the same position / posture relationship. The plane ID is used to identify whether the feature point (corner) is on the same plane. Here, the same plane corresponds to the display surface of the same display panel 12A, and in the case where the position / posture relationship is the same, corresponds to the plane of the chart pattern as a real object.

[0098] Refer again Figure 5 , in step S13, the generation unit 101 generates a conversion table. In the conversion table, a two-dimensional index (two-dimensional angular index) representing the positional relationship of the feature point in the chart pattern (two-dimensional angular position in the checkered pattern) is associated with the position coordinates of the display panel 12A. Here, the position coordinates of the feature point of the image of the chart pattern included in the chart pattern image displayed on the display panel 12A are regarded as the position coordinates on the display surface of the display panel 12A. Since it is only necessary to know the position coordinates of the chart pattern relative to the display panel 12A as information required for calibration, the two-dimensional position coordinates are basically sufficient.

[0099] When generating a chart pattern image, the squares of the checkered pattern are arranged so that the mark with the mark ID embedded therein becomes the reference of the two-dimensional index. Then, for each mark ID, a conversion table capable of converting the two-dimensional index (i, j) representing the feature point as the two-dimensional angular position into the position coordinates (X, Y, Z) relative to the display surface of the display panel 12A is generated. In the following, the position coordinates relative to the display surface of the display panel 12A are referred to as the position coordinates on the display panel 12A in some cases.

[0100] Fig. 9 is a diagram showing an example of two-dimensional angular positions of feature points on a chart pattern image. Fig. 9 , the upper left vertex of the square two squares above the marker M is set as the origin (0, 0) of the coordinate system of the two-dimensional index (i, j). However, i and j are each integer values.

[0101] Here, for example, for the display panel 12A of the display device 12, the horizontal and vertical physical sizes (unit: mm) are represented by panel_width and panel_height, respectively, the horizontal and vertical display pixel numbers (unit: pix) are represented by screen_width and screen_height, respectively, the size of one corner (unit: pix) is represented by corner_size, and the number of pixels to be offset (unit: pix) is represented by offset_x and offset_y. At this time, the position coordinates (X, Y, Z) of an arbitrary two-dimensional index (i, j) on the display panel 12A are represented as follows.

[0102] X=(panel_width / screen_width)*corner_size*i

[0103] Y=(panel_height / screen_height)*corner_size*j

[0104] Z=0

[0105] Using these relationships, a conversion table for converting a two-dimensional index (i, j) into position coordinates (X, Y, Z) on the display panel 12A can be generated. Fig.10 is a diagram showing an example of a conversion table (table_ij_to_XYZ(marker_id)). Fig.10 In the conversion table, the two-dimensional index (i, j) and the position coordinate (X, Y, Z) on the display panel 12A are associated with each other, and the two-dimensional index (i, j) can be converted into the position coordinate (X, Y, Z) on the display panel 12A. Here, the position coordinate (X, Y, Z) on the display panel 12A is expressed as the two-dimensional position coordinate (X, Y) with Z=0.

[0106] In this way, when a chart pattern image is generated, an ID correspondence table ( Figure 7 ) and the conversion table ( Fig.10 ), as a database that can be expanded using the tag ID embedded in the tag as a key. The ID correspondence table and the conversion table are output from the chart pattern generating device 21 to the camera calibration device 22 and recorded in the table storage unit 201.

[0107] Refer again Figure 5In step S14, it is determined whether camera calibration is to be performed. In the case where camera calibration is to be performed (S14: Yes), the process proceeds to step S15. In step S15, the display control unit 102 displays the chart pattern image PI1 on the display panel 12A-1 of the display device 12-1, and displays the chart pattern image PI2 on the display panel 12A-2 of the display device 12-2.

[0108] Here, multiple types of chart pattern images generated according to resolution and viewing angle are displayed while changing over time (displayed in a time-division manner). Figure 8 As shown in the display sequence in FIG. 1 , at times t=0 to 4, chart pattern images PI10 to PI14 are sequentially displayed on display panel 12A-1, and chart pattern images PI20 to PI24 are sequentially displayed on display panel 12A-2. During camera calibration, these chart pattern images PI1 and PI2 of which the chart pattern changes with time are imaged by each camera 11A of the imaging device 11 to obtain a captured image.

[0109] The details of the generation and display of the chart pattern have been described above. In the generation and display of the chart pattern, before the camera calibration, a chart pattern image including a chart pattern optimized according to the camera 11A to be calibrated is generated. In addition, an ID correspondence table and a conversion table are generated and provided to the camera calibration device 22. Thereafter, during the camera calibration, the generated chart pattern image is displayed on each of the plurality of display panels 12A and is imaged by the camera 11A to be calibrated.

[0110] <Camera Calibration>

[0111] The camera calibration in the present disclosure is applicable not only to visible light cameras, but also to non-visible light cameras that use infrared light to image structured light and the like. In the structured light method, an infrared light pattern is usually projected based on the principle of triangulation to perform depth measurement, and the non-visible light camera that supports this method is configured to image infrared light images. In the case where the camera calibration in the present disclosure is applied to a non-visible light camera, an IR light source capable of performing uniform illumination is installed to illuminate the chart pattern image and the non-visible light camera when the camera 11A images the chart pattern image. The non-visible light camera as the camera 11A includes a ToF camera. The details of the camera calibration performed by the camera calibration device 22 will be described below. Fig.11 is a flowchart describing the flow of the camera calibration process.

[0112] In step S31, the deflection correction unit 203 performs deflection correction on the conversion table recorded on the table storage unit 201. For example, in the case where the display device 12 includes electronic paper, it is difficult to perform calibration with high accuracy when the flatness is insufficient due to the deflection of the display panel 12A. In this regard, the deflection correction process of the conversion table is performed to correct the position coordinates on the display panel 12A to the position coordinates corresponding to the deflection of the display panel 12A. This makes it possible to perform calibration with high accuracy even in the case where a chart pattern image is displayed on the electronic paper.

[0113] In the deflection correction process, the deflection correction data is used to correct the deflection correction stored in the conversion table ( Fig.10 ) on the display panel 12A to obtain the corrected position coordinates (X', Y', Z') on the display panel 12A. The deflection correction data is recorded on the deflection correction data storage unit 202. The deflection correction data is generated in the chart pattern calibration described below.

[0114] Fig.12 is a diagram showing an example of deflection correction data. Deflection correction data is prepared for each display panel 12A. Fig.12 In the deflection correction data in the display panel 12A, the original position coordinates (X org , Y org , Z org ) and the correction amount (ΔX, ΔY, ΔZ) are associated with each other. However, since Z org are all 0, so in Fig.12 Omitted in. Fig.13 As shown, in the deflection correction process, a three-dimensional coordinate system having the upper left corner of the display surface of the display panel 12A in the display device 12 as the origin may be used.

[0115] Fig.14 is a diagram schematically showing a deflection correction map. Fig.14 The deflection correction map in φ is a map that visualizes the correction amounts ΔX, ΔY, and ΔZ with X as the horizontal axis and Y as the vertical axis, and corresponds to the plane of the display panel 12A. Fig.14 The A, B, and C parts of FIG. 1 show the correction amounts ΔX, ΔY, and ΔZ, respectively, and show that the correction amount ΔZ is large. The original position coordinates (X, Y, Z) on the display panel 12A, which is the deflection correction target, can be handled as a point on the XY plane because Z=0. Therefore, the object coordinates (X, Y) of the deflection correction data near the position coordinates (X, Y) on the display panel 12A are referred to. org , Y org ) to perform interpolation.

[0116] Regarding the correction amounts (ΔX, ΔY, ΔZ), the correction amounts obtained by performing interpolation processing (e.g., bilinear interpolation) and extrapolation processing (e.g., keeping at the neighborhood value) on the XY plane are respectively taken as ΔX interpolated , ΔY interpolated and ΔZ interpolated Applied to the original position coordinates (X, Y, Z) on the display panel 12A, the position coordinates (X', Y', Z') on the display panel 12A after deflection correction are expressed as follows.

[0117] X'=X+ΔX interpolated

[0118] Y'=Y+ΔY interpolated

[0119] Z'=0+ΔZ interpolated

[0120] Fig.15 is a diagram showing an example of a case where bilinear interpolation is performed as interpolation processing. Fig.15 As shown, bilinear interpolation may be performed with reference to correction amounts (ΔX, ΔY, ΔZ) of four points ( p1 to p4 ) on the deflection correction map surrounding the position coordinates (X, Y) on the display panel 12A.

[0121] In this way, deflection correction can be performed at an arbitrary point (X, Y) on the XY plane of the display panel 12A, and the position coordinates (X, Y, Z) on the display panel 12A of the two-dimensional index (i, j) can be corrected to the position coordinates (X', Y', Z') on the display panel 12A. Then, the conversion table storing the position coordinates (X', Y', Z') after deflection correction is recorded in the table storage unit 204.

[0122] Refer again Fig.11 In step S32, the camera parameter estimation unit 205 performs camera parameter estimation processing. Fig.16 The flowchart describes the detailed process of the camera device parameter estimation process.

[0123] In step S51, the image acquisition unit 221 acquires a captured image generated by each camera 11A of the imaging device 11. For example, the cameras 11A-1 to 11A-3 image the chart pattern images PI1 and PI2 displayed on the display panels 12A-1 and 12A-2 of the display devices 12-1 and 12-2 to acquire a captured image I (frame, view) of each viewpoint (view=0, 1, 2, ...) of the camera 11A.

[0124] For example, Figure 8As shown in the display sequence in FIG. 1 , the chart pattern images PI10 to PI14 are sequentially displayed on the display panel 12A-1, and the chart pattern images PI20 to PI24 are sequentially displayed on the display panel 12A-2. The camera devices 11A-1 to 11A-3 acquire a plurality of image frames obtained by imaging the chart pattern image PI that changes over time in the same state of position / posture relationship (a state in which the position / posture relationship is maintained). In some cases, the plurality of image frames will be simply referred to as captured images.

[0125] Note that noise reduction processing can be performed by adding a plurality of image frames obtained by imaging in the same position / posture relationship. For example, 16 frames of image frames can be added for each corresponding pixel to obtain an average value, and a frame of image with the average value as the pixel value can be generated.

[0126] In step S52, the feature point information generating unit 222 performs a feature point information generating process. In the feature point information generating process, an index of each feature point is obtained based on the positional relationship with the image of the marker detected from the captured image, and is converted into position coordinates in the chart pattern image displayed on the display panel 12A. Thus, feature point information is generated that associates the position coordinates of the feature point on the captured image with the position coordinates of the chart pattern image displayed on the display panel 12A on the display panel 12A.

[0127] The feature point information generation process (S52) is repeated for the captured images (all image frames) acquired by all the cameras 11A included in the imaging device 11 (S53: No). Therefore, in the feature point information generation process, a list (corners_list(view, plane_id)) storing the two-dimensional position coordinates (u, v) corresponding to each feature point (corner) on the same plane (which can be identified by plane_id) and the position coordinates (X', Y', Z') on the display panel 12A is generated as feature point information for each viewpoint (view=0, 1, 2, ...) of the camera 11A.

[0128] Here, for example, in the case where the imaging device 11 includes a low-resolution camera and a high-resolution camera, the feature point information generating unit 222 can generate the feature point information based on the low-resolution chart pattern ( Figure 8 The chart pattern images PI11 to PI14 and PI21 to PI24 in FIG. 1 and the high-resolution chart pattern images ( Figure 8 The integrated feature point information is generated by using the chart pattern images PI10 and PI20 in FIG. Fig.17The flowchart of describes the details of the feature point information generation process. In a case where the feature point information generation process is completed for all captured images generated by all camera devices 11A (S53: YES), the process proceeds to step S54.

[0129] In step S54, the camera parameter acquisition unit 223 performs calibration processing using the feature point information to acquire camera parameters. For this calibration processing, for example, the existing technology described in Document B described below can be used.

[0130] Document B: Zhengyou Zhang, "AFlexible New Technique for CameraCalibration", IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11): 1330-1334, 2000.

[0131] The camera parameter acquisition unit 223 uses the corresponding information (corners_list(view, plane_id)) of the feature points included in the feature point information as an input parameter to specify the camera parameters using an existing algorithm. As the camera parameters, internal parameters (e.g., camera matrix A(view) or lens distortion coefficient K(view)) for each camera 11A (view=0, 1, 2, ...) can be estimated. For example, in the case of using Zhang's method, feature point information in multiple planes (planes of a chart pattern identified by multiple detected plane_ids) in a captured image obtained in one imaging can be used instead of feature point information in an image of a plane chart imaged from multiple viewpoints.

[0132] Since the rotation matrix R and the translation vector t (the position / posture relationship between the camera 11A and the chart pattern image) are also obtained together, external parameters such as the mutual relationship between the plurality of camera devices 11A configured as a multi-lens camera device (for example, R(view0, view1) or t(view0, view1)) can also be estimated based on these. As a method of identifying the rotation matrix R and the translation vector t, an open source software library such as the one implemented as the cv::CalibrateCamera() function of OpenCV can be used.

[0133] The estimated camera parameters are recorded on the camera parameter storage unit 206. Alternatively, the camera parameters may be transmitted to the imaging device 11. By keeping the camera parameters in the imaging device 11, appropriate correction may be performed within the imaging device 11. Alternatively, the camera parameters may be transmitted to a different device that performs processing using an image generated by the imaging device 11.

[0134] When the processing of step S54 is completed, the processing returns to Fig.11 Step S32 of , and a series of processing ends. Fig.17 Flowchart description and Fig.16 Details of the feature point information generation process corresponding to step S52 in FIG. Fig.17 In the description of Fig.18 and Fig.19 Schematic diagram of the captured image shown. Fig.18 is a diagram showing a portion of a captured image in which an image of a marker in the captured image is enlarged.

[0135] First, an image of feature points is detected from the captured image (S70). Many methods have been practiced as methods for detecting the vertices of the squares of the checkered pattern as feature points, and any method can be used here. For example, in the case of using a method using Harris features, those whose feature values ​​are a predetermined value or higher are extracted as vertices (corners) of the square. For example, a method using Harris features is described in the following document C.

[0136] Reference C: C. Harris and M. Stephens, "A combined corner and edge detector", Proceedings of the 4th Alvey Vision Conference, 1988, pp. 147-151

[0137] However, since the position coordinates obtained by this method have insufficient accuracy in some cases, it is desirable to detect the position coordinates (u, v) in the captured image with sub-pixel accuracy using two edge brightness gradients. For this processing, the cv::FindCornerSubPix() function of OpenCV or the like can be used. For example, by detecting an image of a feature point (corner) from a captured image, the position coordinates (u, v) of a feature point candidate on the captured image are obtained ( Fig.18 Part A of the .

[0138] Next, the image of the marker in the captured image is detected (S71). For example, by detecting the image of the marker, the marker ID and the four two-dimensional coordinate groups around the marker are obtained ( Fig.18Then, for one of the detected markers, a parameter representing the distortion on the image is obtained (S72). Specifically, a homography matrix Hmarker (homography matrix) is obtained, which converts the four vertices of a rectangle with sides in the horizontal and vertical directions that should originally appear as the image of the marker into the four vertices of the actual image of the marker.

[0139] Subsequently, the marker is used as a base point to detect the vertices of the square around the marker (S73). At this time, by transforming the four vertices of the square that should have appeared as a square image using the homography matrix Hmarker acquired in S72, the approximate positions of the four vertices can be identified in consideration of the distortion occurring in its vicinity. Then, effectively detected feature points (corners) are searched around the transformed position coordinates (for example, within a range of about 0.5 pixels). Since the two-dimensional index of the detected feature point is clear according to the positional relationship with the marker, this process associates the position coordinates (u, v) of the feature point in the captured image with the two-dimensional index ( Fig.18 Part B of the

[0140] Next, by identifying the corrected position coordinates (X', Y', Z') on the display panel 12A according to the two-dimensional index (i, j) (however, Z'=0), the position coordinates (u, v) of the feature point and the corrected position coordinates (X', Y', Z') on the display panel 12A are associated with each other (S74). For the conversion of the position coordinates, a corrected conversion table (table_ij_to_XYZ(marker_id)) identified by the marker ID is used. Here, the position coordinates (u, v) of the feature point and the position coordinates (X', Y', Z') on the display panel 12A are stored in a list (corners_list(view, plane_id)). However, the plane ID (plane_id) is identified from the ID correspondence table (marker_id_table) using the marker ID. Next, parameters representing the distortion of the square detected in S73 are obtained (S75). Specifically, similar to S72, the homography matrix Hcorner that converts the four vertices of the square that should have appeared as an image of a square into the four vertices of the actual image of the square is obtained.

[0141] Next, the square is used as a base point to detect the vertices of the squares around the square (S76). Also in this case, similarly to S73, by transforming the four vertices of the square that should have appeared as an image of a square using the homography matrix Hcorner acquired in S75, the approximate positions of the vertices of the square can be identified in consideration of the distortion occurring in its vicinity. Then, by searching for and detecting effectively detected feature points (corners) around the transformed position coordinates, the position coordinates (u, v) and the two-dimensional indexes are associated with each other ( Fig.18 of the FDA).

[0142] Next, similar to S74, the two-dimensional index (i, j) is converted into the corrected position coordinates (X', Y', Z') on the display panel 12A, and is stored in association with the position coordinates (u, v) of the feature point in the captured image (S77). Also in this case, for the conversion of the position coordinates, a corrected conversion table identified by the tag ID is used. In addition, the position coordinates (u, v) of the feature point and the position coordinates (X', Y', Z') on the display panel 12A are stored in a list (corners_list(view, plane_id)).

[0143] Repeat the processing of S75 to S77 for all squares represented on the same plane (S78: No). Here, the same plane can be regarded as each display surface (single plane) of the display panel 12A. In the case where the corresponding information, i.e., the feature points, of the vertices of all squares are generated (S78: Yes), the processing of S72 to S78 is repeated for the planes corresponding to the other markers detected in S71 (S79: No). When the feature point information can be generated for all detected markers and therefore for all planes and stored in this way, the processing ends (S79: Yes). When Fig.17 At the end of a series of processing, the process returns to Fig.16 Step S52 is performed, and the processing of the subsequent steps is performed. That is, Fig.17 A series of processes in is repeated for each viewpoint (view) of each camera 11A.

[0144] Therefore, for each viewpoint (view) of each camera 11A, a list (corners_list(view, plane_id)) storing position coordinates (u, v) corresponding to feature points (corners) on the same plane and position coordinates (X', Y', Z') on the display panel 12A is obtained as feature point information. That is, for each feature point, the position coordinates in the captured image and the position coordinates on the display panel 12A are associated with each other. For example, Fig.19As shown, when processing is performed on a captured image of view=1, a tag ID is obtained from each of the images of markers M1 and M2 in the captured image. At this time, the plane ID is identified from the tag ID with reference to the ID correspondence table. For example, the tag ID (marker_id) embedded in the marker M1 is used as a key for identifying panel_id=1 and frame_group_id=1. In addition, the tag ID embedded in the marker M2 is used as a key for identifying panel_id=2 and frame_group_id=1. Since the values ​​of view and plane_id are specified, the position coordinates (u, v) of the feature point and the position coordinates (X', Y', Z') on the display panel 12A can be stored in corners_list(view, plane_id).

[0145] The details of the camera calibration have been described above. In the camera calibration, a chart pattern image displayed on each of the plurality of display panels 12A is imaged by the camera 11A to be calibrated to generate a plurality of captured images. The camera calibration device 22 detects images of a plurality of chart patterns representing a feature point group having a density different from that of the captured images, and generates feature point information that associates the position coordinates of the images of the feature points with the position coordinates of the feature points on the display surface of the display panel 12A. Then, the camera calibration device 22 performs calculations for calibration based on the feature point information, and acquires camera parameters related to the camera 11A.

[0146] <Chart pattern calibration>

[0147] The calibration of the chart pattern will be described. By performing the calibration of the chart pattern, deflection correction data to be used in the deflection correction process is generated.

[0148] Fig. 20 is a diagram showing an example of a configuration of a system during calibration of a chart pattern. Fig. 20 In the embodiment, the imaging device 31 images the image of the chart pattern represented by the chart pattern image PI displayed on the display panel 12A of the display device 12 so that the entire display panel 12A is included in the viewing angle. At this time, the imaging device 31 images the chart pattern from multiple viewpoints while changing the position and posture as shown by the dotted line in the figure. The multiple captured images taken by the imaging device 31 are output to the calibration device 32 connected via a predetermined interface.

[0149] The calibration device 32 includes a PC, a server, a dedicated device, etc. The calibration device 32 can be Figure 3The calibration device 32 includes a processing unit 301. The processing unit 301 performs 3D reconstruction processing using a plurality of captured images taken by the imaging device 31. In this 3D reconstruction processing, for example, the prior art described in the following document D can be used. By using the method of document D, it is possible to estimate a new position coordinate group on the display panel 12A taking into account a slight deformation of the display surface of the display panel 12A displaying the chart pattern image. This makes it possible to obtain the deformation amount (ΔX, ΔY, ΔZ) on the plane of the display panel 12A relative to the ideal state.

[0150] Reference D: K. H. Trobl and G. Hirzinger, “More Accurate Pinhole Camera Calibration with Imperfect Planar Target.” In ICCV Workshops, February 2011.

[0151] Now, refer to Fig.21 The flowchart of describes the process of the chart pattern calibration process. In step S91, the processing unit 301 obtains a plurality of captured images obtained by imaging the chart pattern image PI for each display panel 12A. The plurality of captured images are obtained by imaging the chart pattern image PI displayed on the display device 12 by the imaging device 31 while changing the position and posture. In the case where the display device 12-1 and the display device 12-2 are used during the camera calibration, the imaging device 31 captures images from a plurality of viewpoints for each of the display panel 12A-1 and the display panel 12A-2 to obtain a plurality of captured images. The captured image for each display panel 12A can be identified by the panel ID (panel_id).

[0152] In step S92, the processing unit 301 detects an image of a feature point from the captured image. Here, the position coordinates (u, v) of the feature point (corner) can be obtained by the above-mentioned processing ( Fig.16 The position coordinates (u, v) [pix] in the captured image correspond to the position coordinates (x org , Y org , Z org )[mm]. For example, by constraining three points closest to the upper left, lower left, and lower right positions with the upper left position of the four corners of the plane as the origin in the display panel 12A included in the captured image, the feature point (corner) can be estimated.

[0153] In step S93, the processing unit 301 performs calibration using the method of the above-mentioned document D. The method of document D is also implemented in OpenCV, and by replacing (X org , Y org , Z org ) and (u, v) are input into the API (application programming interface) called cv::calibrateCameraRO() in OpenCV4 or subsequent OpenCV, and a new position coordinate group (X, V) on the display panel 12A can be obtained. new , Y new , Z new )[mm].

[0154] In step S94, the processing unit 301 generates deflection correction data. Here, the difference (X) from the original position coordinates on the display panel 12A to the new position coordinates on the display panel 12A is new -X org , Y new -Y org , Z new -Z org ) is used as the correction amount (ΔX, ΔY, ΔZ), and the original position coordinates (X org , Y org , Z org ) and the correction amount (ΔX, ΔY, ΔZ) are used as deflection correction data for each display panel 12A (panel_id).

[0155] Fig. 22 is a diagram showing an example of the correspondence relationship between the original position coordinates on the display panel 12A and the corrected position coordinates on the display panel 12A. Fig.23 is a diagram showing an example of the correction amount.

[0156] Fig. 22 The original position coordinates (X org , Y org , Z org )and Fig.23 The pair of correction amounts (ΔX, ΔY, ΔZ) in is used as deflection correction data and recorded in the deflection correction data storage unit 202 ( Figure 3 When the processing of step S94 is completed, a series of processing ends.

[0157] The details of the chart pattern calibration have been described above. In the chart pattern calibration, the chart pattern to be used in the deflection correction process ( Fig.11 Deflection correction data used in S31).

[0158] <Multi-scene imaging>

[0159] The imaging device 11 does not necessarily need to image the display device 12 while being fixed in a certain position and posture (single-scene imaging), and can perform imaging in various positions and postures (multi-scene imaging). For example, in a case where single-scene imaging cannot achieve sufficient accuracy, multi-scene imaging can be performed. In multi-scene imaging, the imaging device 11 captures images multiple times while changing the relative position and posture between the imaging device 11 and the chart pattern image displayed on the display device 12. Here, for example, images are captured while moving the imaging device 11 to be calibrated or a jig that fixes the imaging device 11 to be calibrated.

[0160] Fig.24 and Fig.25 Each is a top view showing an example of multi-scene imaging. In this example, Figure 8 As shown in the display sequence of , it is assumed that at time t=0 to 4, the chart pattern images PI10 to PI14 are sequentially displayed on the display panel 12A-1, and the chart pattern images PI20 to PI24 are sequentially displayed on the display panel 12A-2. Fig.24 The state of multi-scene imaging at time t=0 is shown. Fig.25 The states of multi-scene imaging are shown at times t = 1 to 4. In this example, the imaging device 11 includes a camera 11A-a capable of acquiring a captured image at a high resolution (e.g., 1280×1080) and a camera 11A-b capable of acquiring a captured image at a low resolution (e.g., 240×180).

[0161] like Fig.24 As shown, at time t=0, the imaging device 11 captures images at positions and postures of the camera 11A-a and the camera 11A-b where the chart pattern images PI1 and PI2 fall within the viewing angle of the camera 11A-a. At this time, the chart pattern image PI10 is displayed on the display panel 12A-1, and the chart pattern image PI20 is displayed on the display panel 12A-2 ( Figure 8 ). The chart pattern image PI10 and the chart pattern image PI20 each show a chart pattern with a relatively high feature point group density. The chart pattern images PI10 and PI20 can each be regarded as a chart pattern for the high-resolution camera 11A-a. That is, based on the chart pattern images PI10 and PI20, internal parameters of the camera 11A-a as a high-resolution camera can be acquired.

[0162] Then, if Fig.25As shown, at time t=1 to 4, the imaging device 11 captures images at positions and postures of the camera 11A-a and the camera 11A-b where the chart pattern images PI1 and PI2 fall within the viewing angle of the camera 11A-b. Here, for example, the position of the imaging device 11 is changed by the movement of the imaging device 11 on the track 91 indicated by the dotted line in the up-down direction in the figure. At this time, the chart pattern images PI11 to PI14 are sequentially displayed on the display panel 12A-1, and the chart pattern images PI21 to PI24 are displayed on the display panel 12A-2 ( Figure 8 ). Note that since the chart pattern images PI11 to PI14 and the chart pattern images PI21 to PI24 are each a chart pattern with a relatively low density of feature point groups, there is a possibility that sufficient feature point information cannot be obtained using only one image frame (one chart pattern) as internal parameters of the camera 11A-b of a low-resolution camera. Therefore, the present disclosure proposes to improve the density and accuracy of feature point groups by integrating feature point groups of multiple chart patterns that change over time.

[0163] In addition, Fig.25 In the case where the chart pattern images PI1 and PI2 fall within the viewing angle of the camera 11A-a, the external parameters between the camera 11A-a and the camera 11A-b can be obtained using the captured image acquired by the camera 11A-a and the captured image generated by the camera 11A-b. Here, since the panel ID and the group ID can be identified from the ID correspondence table using the tag ID from the tag M included in the chart pattern images PI1 and PI2 as a keyword, it is possible to identify whether the position / posture relationship between the chart pattern image PI displayed on the imaging device 11 (camera 11A) and the display device 12 (display panel 12A) is the same. Note that since the camera 11A-a is a high-resolution camera, feature point information of sufficient accuracy can be obtained from the captured image not only in the case of imaging a chart pattern for a high-resolution camera but also in the case of imaging a chart pattern for a low-resolution camera.

[0164] At this time, in the ID correspondence table, frame_group_id=0 is associated with marker_id=0, 1 in the chart pattern images PI10 and PI20 displayed at time t=0, and frame_group_id=1 is associated with marker_id=2 to 9 in the chart pattern images PI11 to PI14 and PI21 to PI24 displayed at time t=1 to 4. In addition, panel_id=1 is associated with marker_id=0, 2, 4, 6, and 8, and panel_id=2 is associated with marker_id=1, 3, 5, 7, and 9. For example, in the feature point information generation process, with reference to the ID correspondence table, when the group ID associated with the marker ID obtained from the marker included in the captured image is the same, feature point information is generated as a feature point group on the same plane. Then, when the calibration process is performed, when the group ID of the feature point group is the same, the camera parameters can be acquired by integrating the feature point group on the same plane.

[0165] At the same time, in the case where the group ID is different, by generating feature point information as a feature point group on another plane, it is possible to obtain the camera parameters as a feature point group on another plane when performing the calibration process. This allows calibration to be performed using only a plurality of acquired captured images (image frames) in multiple scenarios in which the position / posture relationship between the imaging device 11 (camera 11A) and the chart pattern image PI displayed on the display device 12 (display panel 12A) is changed. Note that although in Fig.24 and Fig.25 , a case where the imaging device 11 is a multi-camera device is shown in , but in the case of using a wide-angle camera, the wide-angle camera only needs to capture an image while moving so that the lens distortion shape of the peripheral portion can be estimated.

[0166] As described above, in the calibration system to which the present disclosure is applied, the camera 11A to be calibrated images a chart pattern image including a chart pattern optimized according to the camera 11A to be calibrated and displayed on each of the plurality of display panels 12A. Here, the optimized chart pattern displayed on the display panel 12A is a plurality of chart patterns having different feature point group densities so as to correspond to different sub-pixel accuracies of the plurality of cameras 11A. Then, the camera calibration device 22 detects an image of a feature point of the chart pattern that changes with time from a captured image acquired by the camera 11A, generates feature point information that associates the position coordinates of the image of the feature point with the position coordinates of the feature point on the display surface of the display panel 12A, and performs an operation for calibration based on the feature point information to obtain a camera parameter related to the camera 11A.

[0167] In addition, in the camera calibration in the present disclosure, in order to calibrate the camera 11A with different resolutions and viewing angles, the accuracy and density of the feature point information are improved by integrating the feature point group of the chart pattern that changes over time. Therefore, even if different cameras 11A are used, sufficient feature point information for calibration can be obtained. Therefore, even if at least one of the resolution, viewing angle or lens distortion shape is different, the camera calibration can be appropriately performed. For example, in single-scene imaging in which the imaging device 11 is fixed at a certain position and posture relative to the display device 12, or in a small number of imaging scenes, the imaging device 11 including multiple cameras 11A with different parameters can be calibrated with high accuracy. The camera calibration in the present disclosure does not necessarily need to be applied to visible light cameras, and is also applicable to non-visible light cameras. In addition, since the chart pattern generation device 21 that performs display control of the chart pattern image and the camera calibration device 22 that obtains the captured image can be separate devices and each operates independently, the system configuration can be further simplified.

[0168] <Modification example>

[0169] <Another Configuration Example>

[0170] Fig.26 is a block diagram showing another configuration example of a calibration system to which the present disclosure is applied. Fig.26 In the present invention, the calibration system includes an imaging device 11, display devices 12-1 and 12-2, and a processing device 41. Fig.26 In the calibration system, Figure 3 Compared with the calibration system in FIG. 1 , a processing device 41 is provided to replace the chart pattern generating device 21 and the camera calibration device 22 .

[0171] The processing device 41 includes a PC, a server, a dedicated device, etc. The processing device 41 includes a chart pattern generation unit 21A and a camera calibration unit 22A. The chart pattern generation unit 21A has Figure 3 The camera calibration unit 22A has a function similar to that of the chart pattern generating device 21 in FIG. Figure 3 In this way, the chart pattern generating device 21 and the camera calibration device 22 may be configured as separate devices, or may be configured as one device, that is, the processing device 41.

[0172] Fig. 27 is a block diagram showing yet another configuration example of a calibration system to which the present disclosure is applied. Fig. 27 In the present invention, the calibration system includes display devices 12-1 and 12-2, a chart pattern generating device 21 and an imaging device 51. Fig. 27In the calibration system, Figure 3 Compared with the calibration system in FIG. 1 , an imaging device 51 is provided to replace the imaging device 11 and the camera calibration apparatus 22 .

[0173] The imaging device 51 includes an imaging unit 11B and a camera calibration unit 22B. The imaging unit 11B has Figure 3 The camera calibration unit 22B has functions similar to those of the imaging device 11 in FIG. Figure 3 In this way, the calibration process can be performed on the imaging device side, or can be performed by a different device other than the imaging device.

[0174] Note that, although the above description shows a configuration in which two display devices 12, namely, the display device 12-1 and the display device 12-2 are provided, three or more display devices 12 may be provided and a chart pattern image may be displayed on the respective display devices 12. Even with such a configuration, the display panel 12A of each display device 12 may be identified by the panel ID.

[0175] In the calibration system to which the present disclosure is applied, the chart pattern generating device 21 and the camera calibration device 22 can operate independently without cooperation, but can also operate in cooperation. In addition, when the display panel 12A that is not affected by deflection (is less affected by deflection) is used as the display panel 12A of the display device 12, it is not necessarily necessary to perform the deflection correction processing ( Fig.11 In this case, in the camera parameter estimation unit 205, it is only necessary to use the conversion table recorded in the table storage unit 201 instead of the corrected conversion table recorded in the table storage unit 204.

[0176] <Computer Configuration>

[0177] The above-described series of processing may be executed by hardware, or may be executed by software. When the series of processing is executed by software, a program constituting the software is installed in a computer. Fig.28 : is a block diagram showing a configuration example of hardware of a computer that executes the above-described series of processes by a program.

[0178] In the computer, a CPU 1001, a ROM 1002, and a RAM 1003 are connected to each other via a bus 1004. An input / output interface 1005 is further connected to the bus 1004. An input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010 are connected to the input / output interface 1005.

[0179] The input unit 1006 includes a keyboard, a mouse, a microphone, etc. The output unit 1007 includes a display, a speaker, etc. The storage unit 1008 includes a hard disk, a nonvolatile memory, etc. The communication unit 1009 includes a network interface, etc. The drive 1010 drives a removable recording medium 1011 such as a semiconductor memory, a magnetic disk, an optical disk, and a magneto-optical disk.

[0180] In the computer described above, the CPU 1001 loads the program recorded on the ROM 1002 or the storage unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 , and executes the program to perform the above-described series of processing.

[0181] The program executed by the computer (CPU 1001) can be recorded on the removable recording medium 1011 as a package medium etc. and provided. Furthermore, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, and digital satellite broadcasting.

[0182] In the computer, by inserting the removable recording medium 1011 into the drive 1010, the program can be installed into the storage unit 1008 via the input / output interface 1005. In addition, the program can be received by the communication unit 1009 via wired or wireless transmission and installed in the storage unit 1008. In addition, the program can be pre-installed in the ROM 1002 or the storage unit 1008.

[0183] Here, in this specification, the processing performed by the computer according to the program does not necessarily need to be performed in chronological order in the order shown in the flowchart. That is, the program executed by the computer according to the program includes processing performed in parallel or individually (for example, parallel processing or processing by object). In addition, the program may be processed by one computer (processor), or may be distributed and processed by a plurality of computers.

[0184] Note that the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications may be made without departing from the essence of the present disclosure. For example, a calibration system to which the present disclosure is applied may have a cloud computing configuration in which one function is shared and collaboratively processed by multiple devices via a network. In addition, the effects described in this specification are merely exemplary and not restrictive, and other effects may be exhibited.

[0185] Furthermore, the present disclosure may adopt the following configurations.

[0186] (1) A camera calibration device, comprising:

[0187] an image acquisition unit that acquires, from an imaging device, a captured image indicating a chart pattern displayed in a time-division manner on a display panel tilted with respect to an optical axis direction of the imaging device;

[0188] a feature point information generating unit that generates integrated feature point information by integrating, on a single plane, feature points at a first time of a chart pattern at a first time included in the image frame at the first time of the captured image and feature points at a second time of the chart pattern at a second time included in the image frame at the second time of the captured image and having an arrangement of a feature point group different from that of the feature point group of the chart pattern at the first time; and

[0189] A parameter acquisition unit is configured to acquire parameters used for calibration of the camera device based on the integrated feature point information.

[0190] (2) The camera calibration device according to (1) above, further comprising:

[0191] a conversion information storage unit, the conversion information storage unit recording conversion information for converting the two-dimensional coordinates of the feature point at the first time into the three-dimensional coordinates and converting the two-dimensional coordinates of the feature point at the second time into the three-dimensional coordinates,

[0192] The feature point information generating unit performs:

[0193] identifying two-dimensional coordinates of a feature point at the first time and two-dimensional coordinates of a feature point at the second time based on a first marker included in the chart pattern at the first time and a second marker included in the chart pattern at the second time and having an ID different from the ID of the first marker,

[0194] Based on the conversion information, converting the two-dimensional coordinates of the feature point at the first time and the two-dimensional coordinates of the feature point at the second time into corresponding three-dimensional coordinates, and

[0195] The integrated feature point information is generated based on the converted three-dimensional coordinates.

[0196] (3) The camera calibration device according to (2) above, further comprising:

[0197] a corresponding information storage unit, the corresponding information storage unit recording corresponding information for associating a panel ID for identifying the display panel with the ID of the first mark and the ID of the second mark,

[0198] The feature point information generation unit identifies the panel ID based on the corresponding information, the ID of the first tag, and the ID of the second tag, associates the feature points at the first time and the feature points at the second time with the display surface of the display panel identified by the identified panel ID, and integrates the feature points at the first time and the feature points at the second time.

[0199] (4) The camera calibration device according to (3) above, wherein:

[0200] The display panel includes a first display panel and a second display panel tilted relative to the optical axis direction.

[0201] The panel ID includes a first panel ID for identifying the first display panel and a second panel ID for identifying the second display panel, and

[0202] The feature point information generating unit performs:

[0203] associating some feature points at the first time with the first display panel based on the first panel ID indicated by one of the plurality of the chart patterns at the first time,

[0204] associating the remaining feature points at the first time with the second display panel based on the second panel ID indicated by another one of the plurality of the chart patterns at the first time,

[0205] associating some feature points at the second time with the first display panel based on the first panel ID indicated by one of the plurality of the chart patterns at the second time,

[0206] associating the remaining feature points at the second time with the second display panel based on the second panel ID indicated by another one of the plurality of chart patterns at the second time, and

[0207] For each of a display surface of the first display panel and a display surface of the second display panel, a feature point at the first time and a feature point at the second time are integrated.

[0208] (5) The camera calibration device according to (3) above, wherein:

[0209] The correspondence information associates a first group ID indicating a first position / posture relationship with each of the ID of the first tag and the ID of the second tag, the first position / posture relationship indicating a relationship of at least one of a relative position or a relative posture between the camera device and the display panel, and associates a second group ID indicating a second position / posture relationship with each of the ID of the first tag and the ID of the second tag, the second position / posture relationship being different from the first position / posture relationship and indicating a relationship of at least one of a relative position or a relative posture between the camera device and the display panel, and

[0210] The feature point information generating unit identifies one of the first group ID and the second group ID based on the corresponding information, the ID of the first tag and the ID of the second tag, and integrates the feature points at the first time and the feature points at the second time for the display surface of the display panel based on the identified one group ID and the identified panel ID.

[0211] (6) The camera calibration apparatus according to any one of (2) to (5) above, further comprising:

[0212] a correction unit that corrects the position coordinates of the feature point on the display surface of the display panel based on correction data including a correction amount corresponding to a deformation amount compared to an ideal state with respect to the display surface of the display panel,

[0213] The feature point information generating unit generates the integrated feature point information based on the correction data and the conversion information.

[0214] (7) The camera calibration device according to (2) above, wherein:

[0215] The feature point information generating unit performs:

[0216] Associating the two-dimensional coordinate and the three-dimensional coordinate of the feature point at the first time with each other as a pair,

[0217] Associating the two-dimensional coordinate and the three-dimensional coordinate of the feature point at the second time with each other as a pair, and

[0218] With respect to the display surface of the display panel, the pair of the two-dimensional coordinates and the three-dimensional coordinates of the feature point at the first time and the pair of the two-dimensional coordinates and the three-dimensional coordinates of the feature point at the second time are integrated.

[0219] (8) The camera calibration apparatus according to any one of (1) to (7) above, wherein:

[0220] The camera device includes a relatively low-resolution camera device and a relatively high-resolution camera device,

[0221] The chart pattern includes a low-resolution chart pattern and a high-resolution chart pattern, the low-resolution chart pattern includes a chart pattern at the first time and a chart pattern at the second time and has a relatively low-density feature point group, the high-resolution chart pattern has a time different from the first time and the second time and has a relatively high-density feature point group,

[0222] The feature point information generating unit generates the integrated feature point information based on the low-resolution chart pattern acquired from the low-resolution camera, and generates high-resolution feature point information based on the high-resolution chart pattern acquired from the high-resolution camera, and

[0223] The parameter acquisition unit acquires a parameter for calibrating the low-resolution camera and a parameter for calibrating the high-resolution camera based on the integrated feature point information and the high-resolution feature point information.

[0224] (9) The camera calibration apparatus according to any one of (1) to (8) above, wherein:

[0225] The display panel is electronic paper,

[0226] The image acquisition unit acquires an infrared light image indicating a chart pattern displayed by the electronic paper in a time-division manner from the camera device, and

[0227] The feature point information generating unit generates the integrated feature point information based on the infrared light image.

[0228] (10) A camera device calibration method comprising:

[0229] acquiring, from an imaging device, a captured image indicating a chart pattern displayed in a time-division manner on a display panel tilted with respect to an optical axis direction of the imaging device;

[0230] generating integrated feature point information by integrating feature points at a first time of a chart pattern at a first time included in an image frame at the first time of the captured image and feature points at a second time of a chart pattern at a second time on a single plane, the chart pattern at the first time being included in the image frame at the second time of the captured image and having an arrangement of feature point groups different from that of the feature point groups of the chart pattern at the first time; and

[0231] Parameters for calibration of the camera device are acquired based on the integrated feature point information.

[0232] (11) A recording medium recording a program including instructions for causing a computer to:

[0233] acquiring, from an imaging device, a captured image indicating a chart pattern displayed in a time-division manner on a display panel tilted with respect to an optical axis direction of the imaging device;

[0234] generating integrated feature point information by integrating feature points at a first time of a chart pattern at a first time included in an image frame at the first time of the captured image and feature points at a second time of a chart pattern at a second time on a single plane, the chart pattern at the first time being included in the image frame at the second time of the captured image and having an arrangement of feature point groups different from that of the feature point groups of the chart pattern at the first time; and

[0235] Parameters for calibration of the camera device are acquired based on the integrated feature point information.

[0236] Reference numerals list

[0237] 11 imaging device, 11A-1 to 11A-3, 11A-a, 11A-b, 11A camera device, 11B imaging unit, 12-1, 12-2 display device, 12A-1, 12A-2, 12A display panel, 13-1, 13-2 light source, 21 chart pattern generating device, 21A chart pattern generating unit, 22 camera device calibration device, 22A, 22B camera device calibration unit, 31 imaging device, 32 calibration device, 41 processing device, 51 imaging device, 101 generating unit, 102 display control unit, 201 table storage unit, 202 deflection correction data storage unit, 203 deflection correction unit, 204 table storage unit, 205 camera device parameter estimation unit, 206 camera device parameter storage unit, 221 image acquisition unit, 222 feature point information generating unit, 223 camera device parameter acquisition unit, 301 processing unit, 1001 CPU

Claims

1. A camera calibration device, comprising: an image acquisition unit that acquires, from an imaging device, a captured image indicating a chart pattern displayed in a time-division manner on a display panel tilted with respect to an optical axis direction of the imaging device; a feature point information generating unit that generates integrated feature point information by integrating, on a single plane, feature points at a first time of a chart pattern at a first time included in the image frame at the first time of the captured image and feature points at a second time of the chart pattern at a second time included in the image frame at the second time of the captured image and having an arrangement of a feature point group different from that of the feature point group of the chart pattern at the first time; as well as A parameter acquisition unit is configured to acquire parameters used for calibration of the camera device based on the integrated feature point information.

2. The camera calibration device according to claim 1, further comprising: a conversion information storage unit, the conversion information storage unit recording conversion information for converting the two-dimensional coordinates of the feature point at the first time into the three-dimensional coordinates and converting the two-dimensional coordinates of the feature point at the second time into the three-dimensional coordinates, The feature point information generating unit performs: identifying two-dimensional coordinates of a feature point at the first time and two-dimensional coordinates of a feature point at the second time based on a first marker included in the chart pattern at the first time and a second marker included in the chart pattern at the second time and having an ID different from the ID of the first marker, Based on the conversion information, converting the two-dimensional coordinates of the feature point at the first time and the two-dimensional coordinates of the feature point at the second time into corresponding three-dimensional coordinates, and The integrated feature point information is generated based on the converted three-dimensional coordinates.

3. The camera calibration apparatus according to claim 2, further comprising: a corresponding information storage unit, the corresponding information storage unit recording corresponding information for associating a panel ID for identifying the display panel with the ID of the first mark and the ID of the second mark, The feature point information generation unit identifies the panel ID based on the corresponding information, the ID of the first tag, and the ID of the second tag, associates the feature points at the first time and the feature points at the second time with the display surface of the display panel identified by the identified panel ID, and integrates the feature points at the first time and the feature points at the second time.

4. The camera calibration apparatus according to claim 3, wherein: The display panel includes a first display panel and a second display panel tilted relative to the optical axis direction. The panel ID includes a first panel ID for identifying the first display panel and a second panel ID for identifying the second display panel, and The feature point information generating unit performs: associating some feature points at the first time with the first display panel based on the first panel ID indicated by one of the plurality of the chart patterns at the first time, associating the remaining feature points at the first time with the second display panel based on the second panel ID indicated by another one of the plurality of chart patterns at the first time, associating some feature points at the second time with the first display panel based on the first panel ID indicated by one of the plurality of chart patterns at the second time, associating the remaining feature points at the second time with the second display panel based on the second panel ID indicated by another one of the plurality of chart patterns at the second time, and For each of a display surface of the first display panel and a display surface of the second display panel, a feature point at the first time and a feature point at the second time are integrated.

5. The camera calibration apparatus according to claim 3, wherein: The correspondence information associates a first group ID indicating a first position / posture relationship with each of the ID of the first tag and the ID of the second tag, the first position / posture relationship indicating a relationship of at least one of a relative position or a relative posture between the camera device and the display panel, and associates a second group ID indicating a second position / posture relationship with each of the ID of the first tag and the ID of the second tag, the second position / posture relationship being different from the first position / posture relationship and indicating a relationship of at least one of a relative position or a relative posture between the camera device and the display panel, and The feature point information generating unit identifies one of the first group ID and the second group ID based on the corresponding information, the ID of the first tag and the ID of the second tag, and integrates the feature points at the first time and the feature points at the second time for the display surface of the display panel based on the identified one group ID and the identified panel ID.

6. The camera calibration apparatus according to claim 2, further comprising: a correction unit that corrects the position coordinates of the feature point on the display surface of the display panel based on correction data including a correction amount corresponding to a deformation amount compared to an ideal state with respect to the display surface of the display panel, The feature point information generating unit generates the integrated feature point information based on the correction data and the conversion information.

7. The camera calibration apparatus according to claim 2, wherein: The feature point information generating unit performs: Associating the two-dimensional coordinate and the three-dimensional coordinate of the feature point at the first time with each other as a pair, Associating the two-dimensional coordinate and the three-dimensional coordinate of the feature point at the second time with each other as a pair, and With respect to the display surface of the display panel, the pair of the two-dimensional coordinates and the three-dimensional coordinates of the feature point at the first time and the pair of the two-dimensional coordinates and the three-dimensional coordinates of the feature point at the second time are integrated.

8. The camera calibration apparatus according to claim 1, wherein: The camera device includes a relatively low-resolution camera device and a relatively high-resolution camera device, The chart pattern includes a low-resolution chart pattern and a high-resolution chart pattern, the low-resolution chart pattern includes a chart pattern at the first time and a chart pattern at the second time and has a relatively low-density feature point group, the high-resolution chart pattern has a time different from the first time and the second time and has a relatively high-density feature point group, The feature point information generating unit generates the integrated feature point information based on the low-resolution chart pattern acquired from the low-resolution camera, and generates high-resolution feature point information based on the high-resolution chart pattern acquired from the high-resolution camera, and The parameter acquisition unit acquires a parameter for calibrating the low-resolution camera and a parameter for calibrating the high-resolution camera based on the integrated feature point information and the high-resolution feature point information.

9. The camera calibration apparatus according to claim 1, wherein: The display panel is electronic paper, The image acquisition unit acquires an infrared light image indicating the chart pattern displayed by the electronic paper in a time-division manner from the camera device, and The feature point information generating unit generates the integrated feature point information based on the infrared light image.

10. A camera device calibration method, comprising: acquiring, from an imaging device, a captured image indicating a chart pattern displayed in a time-division manner on a display panel tilted with respect to an optical axis direction of the imaging device; generating integrated feature point information by integrating feature points at a first time of a chart pattern at a first time included in an image frame at the first time of the captured image and feature points at a second time of a chart pattern at a second time on a single plane, the chart pattern at the first time being included in the image frame at the second time of the captured image and having an arrangement of a feature point group different from an arrangement of a feature point group of the chart pattern at the first time; as well as Parameters for calibration of the camera device are acquired based on the integrated feature point information.

11. A recording medium recording a program, wherein the program includes instructions for causing a computer to execute the following: acquiring, from an imaging device, a captured image indicating a chart pattern displayed in a time-division manner on a display panel tilted with respect to an optical axis direction of the imaging device; generating integrated feature point information by integrating feature points at a first time of a chart pattern at a first time included in an image frame at the first time of the captured image and feature points at a second time of a chart pattern at a second time included in the image frame at the second time of the captured image and having an arrangement of feature point groups different from an arrangement of feature point groups of the chart pattern at the first time on a single plane; and Parameters for calibration of the camera device are acquired based on the integrated feature point information.

Images