Information processing apparatus, information processing method, and program

Abstract

There is provided an information processing apparatus, including: a rendering unit that generates, on a basis of three-dimensional data of an environment measured by a 3D scanner and a camera parameter of a camera that images the environment, a rendering image corresponding to a camera image taken by the camera; and a camera parameter estimation unit that extracts a difference between the camera image and the rendering image and updates the camera parameter such that the difference is smaller.

Description

INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, AND PROGRAMCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of Japanese Priority Patent Application JP 2023-171381 filed October 2, 2023, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

[0003] In recent years, the development of technology for sensing the surrounding environment using a sensor that is capable of detecting the position of an object, such as light detection and ranging (LiDAR), and a camera that images an object has progressed. For example, a technology for automated driving by sensing the surrounding environment using LiDAR and a camera in an automobile has been actively developed.

[0004] In such a technology for sensing the surrounding environment, it is important to properly integrate data sensed by the sensor and data of an image taken by the camera. Specifically, it is important to correct the position between the sensor and the camera, correct the internal parameter of the camera, and integrate data sensed by the sensor and data of an image taken by the camera.

[0005] For example, the following Patent Literature 1 describes preparing marker boards for calibrating each of a camera that is capable of imaging an object and a sensor that is capable of detecting the position of the object using a laser beam. In the technology described in Patent Literature 1, it is possible to correct the position between the camera and the sensor by detecting the marker boards installed at a plurality of positions using the camera and the sensor attached to the same moving object.

[0006] Japanese Patent Application Laid-open No. 2021-38939Summary

[0007] However, it has been difficult for ordinary users to perform the technology described in Patent Literature 1 because dedicated marker boards need to be prepared. In particular, since the internal parameter of the camera in addition to the position between the camera and another camera or sensor need to be corrected, there has been a demand for a technology that is capable of more easily performing correction of the camera.

[0008] In this regard, the present disclosure proposes a new and improved information processing apparatus, information processing method, and program that are capable of more easily performing correction of a camera that images an environment.

[0009] In accordance with the present disclosure, there is provided an information processing apparatus, including: a rendering unit that generates, on a basis of three-dimensional data of an environment measured by a 3D scanner and a camera parameter of a camera that images the environment, a rendering image corresponding to a camera image taken by the camera; and a camera parameter estimation unit that extracts a difference between the camera image and the rendering image and updates the camera parameter such that the difference is smaller.

[0010] Further, in accordance with the present disclosure, there is provided an information processing method executed by a computer, including: generating, on the basis of three-dimensional data of an environment measured by a 3D scanner and a camera parameter of a camera that images the environment, a rendering image corresponding to a camera image taken by the camera; and extracting a difference between the camera image and the rendering image and updating the camera parameter such that the difference is smaller.

[0011] Further, in accordance with the present disclosure, there is provided a program that causes a computer to function as: a rendering unit that generates, on the basis of three-dimensional data of an environment measured by a 3D scanner and a camera parameter of a camera that images the environment, a rendering image corresponding to a camera image taken by the camera; and a camera parameter estimation unit that extracts a difference between the camera image and the rendering image and updates the camera parameter such that the difference is smaller.

[0012] Fig. 1 is an explanatory diagram describing a function realized by an information processing apparatus according to an embodiment of the present disclosure.Fig. 2 is a block diagram showing a functional configuration of the information processing apparatus according to the embodiment.Fig. 3 is a flowchart showing a flow of an operation executed by a sensor parameter estimation unit.Fig. 4 is a flowchart showing a flow of processing in the preceding stage executed by a camera parameter estimation unit.Fig. 5 is a flowchart showing a flow of processing in the subsequent stage executed by the camera parameter estimation unit.Fig. 6 is an explanatory diagram describing a first modified example.Fig. 7 is an explanatory diagram describing a second modified example.Fig. 8 is a flowchart showing a flow of processing in the preceding stage executed by a camera parameter estimation unit according to a third modified example.Fig. 9 is a block diagram showing an example of a hardware configuration of an information processing apparatus according to the embodiment.

[0013] A suitable embodiment of the present disclosure will be described below in detail with reference to the accompanying drawings. Note that in the present specification and drawings, components having substantially the same functional configurations will be denoted by the same reference symbols and redundant description will be omitted.

[0014] Note that description will be made in the following order. 1. Overview 2. Configuration example 3. Modified example 4. Hardware configuration example

[0015] <1. Overview> An overview of an information processing apparatus according to an embodiment of the present disclosure will be described first with reference to Fig. 1. Fig. 1 is an explanatory diagram describing a function realized by the information processing apparatus according to this embodiment.

[0016] As shown in Fig. 1, a moving object 10 on which a sensor 200 and a camera 300 are mounted is disposed in a static environment SE whose three-dimensional data has been measured by a 3D scanner 400. The information processing apparatus according to this embodiment is capable of performing correction of the sensor 200 and the camera 300 on the basis of the three-dimensional data of the static environment SE measured by the 3D scanner 400.

[0017] The correction of the sensor 200 refers to correcting the position of the sensor 200 (sensor parameter). Further, the correction of the camera 300 refers to correcting the position of the camera 300 (external parameter) and effects on images due to the focal length, center position, strain, and the like of the camera 300 (internal parameter). The external parameter and the internal parameter are referred to also as camera parameters.

[0018] The 3D scanner 400 is a coordinate-measuring machine that is capable of measuring three-dimensional data of the static environment SE over 360°. Specifically, the 3D scanner 400 may be a laser scanner that is capable of measuring the depth and reflection intensity or color of the static environment SE over 360°. The 3D scanner 400 is capable of acquiring, as three-dimensional data of the static environment SE, three-dimensional point cloud or three-dimensional mesh recording color or reflection intensity.

[0019] The static environment SE is a static environment in which no moving object is present. Specifically, the static environment SE may be an environment closed on at least three sides for each of the sensor 200 and the camera 300 mounted on the moving object 10. In such a case, the information processing apparatus according to this embodiment is capable of performing correction of the sensor 200 and the camera 300 with higher accuracy. For example, the static environment SE may be an indoor environment such as the inside of a garage.

[0020] The moving object 10 is a device on which at least one sensor 200 and at least one camera 300 are mounted. The moving object 10 is capable of performing automated driving by estimating the surrounding environment and self-position on the basis of, for example, data of an image taken by the camera 300 and point cloud data sensed by the sensor 200.

[0021] The sensor 200 is a sensor that is capable of acquiring three-dimensional point cloud data indicating the position of an object present in the static environment SE by applying a laser beam, millimeter waves, or ultrasonic waves. The sensor 200 may be, for example, LiDAR, a millimeter-wave radar, or an ultrasonic sensor. Further, the sensor 200 may include a plurality of sensors mounted on the moving object 10.

[0022] The camera 300 is an imaging device that is capable of taking an image of the static environment SE. The camera 300 may be, for example, an RGB camera, a monochrome camera, or a stereo camera. The camera 300 may include a plurality of cameras mounted on the moving object 10.

[0023] In order to estimate the surrounding environment and self-position with higher accuracy, it is important for the moving object 10 to integrate data of an image taken by the camera 300 and point cloud data sensed by the sensor 200. For example, the moving object 10 is capable of estimating the surrounding environment and self-position with higher accuracy by combining Visual-SLAM (Simultaneous Localization and Mapping) using the data of an image taken by the camera 300 and LiDAR-SLAM using the point cloud data sensed by the sensor 200.

[0024] The information processing apparatus according to this embodiment is capable of more easily performing correction of the sensor 200 and the camera 300 by using three-dimensional data obtained by measuring the static environment SE using the 3D scanner 400. Specifically, the information processing apparatus is capable of more easily performing correction of the position between the camera 300 and the sensor 200 (i.e., the sensor parameter and the external parameter of the camera 300), and effects on images due to the focal length, center position, strain, and the like of the camera 300 (i.e., the internal parameter of the camera 300) by using three-dimensional data obtained by measuring the static environment SE using the 3D scanner 400.

[0025] In accordance with this, the moving object 10 is capable of estimating the surrounding environment and the self-position of the moving object 10 with higher accuracy by integrating the data of an image taken by the camera 300 and the point cloud data sensed by the sensor 200 with high accuracy.

[0026] <2. Configuration example> Next, a configuration example of the information processing apparatus according to this embodiment will be described with reference to Fig. 2 to Fig. 5. Fig. 2 is a block diagram showing a functional configuration of an information processing apparatus 100 according to this embodiment. Fig. 3 is a flowchart showing a flow of an operation executed by a sensor parameter estimation unit 110. Fig. 4 is a flowchart showing a flow of processing in the preceding stage executed by a camera parameter estimation unit 130. Fig. 5 is a flowchart showing a flow of processing in the subsequent stage executed by the camera parameter estimation unit 130.

[0027] As shown in Fig. 2, the information processing apparatus 100 according to this embodiment includes the sensor parameter estimation unit 110, a rendering unit 120, and the camera parameter estimation unit 130. The information processing apparatus 100 is, for example, a personal computer, a laptop, a smartphone, a tablet, a server, or the like, but the information processing apparatus 100 according to an aspect of the present disclosure is not limited to such an example. For example, the information processing apparatus 100 may be implemented by an information processing system in which the sensor parameter estimation unit 110, a rendering unit 120, and the camera parameter estimation unit 130 are included in different information processing devices, or the information processing system may include only one of the different information processing devices. Assumption is made that point cloud data of the static environment SE is input from a plurality of sensors 200 mounted on the moving object 10 to the information processing apparatus 100, and image data of the static environment SE is input from a plurality of cameras 300 mounted on the moving object 10 to the information processing apparatus 100. Further, three-dimensional data of the static environment SE is input from the 3D scanner 400 to the information processing apparatus 100.

[0028] (Sensor parameter estimation unit 110) The sensor parameter estimation unit 110 will be described with reference to Fig. 3.

[0029] The sensor parameter estimation unit 110 estimates the position of the sensor 200 on the basis of the point cloud data of the static environment SE acquired by each sensor 200 and the three-dimensional data of the static environment SE acquired by the 3D scanner 400. The sensor parameter estimation unit 110 may estimate the position of each sensor 200 by registering (aligning) the three-dimensional data of the static environment SE measured by the 3D scanner 400 and the point cloud data of the static environment SE acquired by the sensor 200.

[0030] For example, the sensor parameter estimation unit 110 may estimate conversion from the coordinate system of the sensor 200 into the environment coordinate system based on the 3D scanner 400 by registering the three-dimensional data of the 3D scanner 400 and the point cloud data of the sensor 200. The coordinate system of the sensor 200 is based on the position of the sensor 200. For this reason, the sensor parameter estimation unit 110 is capable of identifying, by estimating conversion from the coordinate system of the sensor 200 into the environment coordinate system, the positional relationship between the respective sensors 200 with reference to the environment coordinate system. Note that conversionLPWfrom the coordinate system of the sensor 200 into the environment coordinate system is represented by, for example, a 4×4 position-orientation matrix including rotation and movement.

[0031] In detail, as shown in Fig. 3, the sensor parameter estimation unit 110 performs registration with point cloud matching first (S101). As a result, the sensor parameter estimation unit 110 is capable of roughly estimating the conversionLPWfrom the coordinate system of the sensor 200 into the environment coordinate system.

[0032] In the processing of Step S101, the sensor parameter estimation unit 110 resamples the three-dimensional point cloud measured by the 3D scanner 400 and the three-dimensional point cloud acquired by the sensor 200 to a predetermined grid size first in order to reduce the amount of calculation. Next, the sensor parameter estimation unit 110 calculates the feature amounts of the respective resampled three-dimensional point clouds. For example, the sensor parameter estimation unit 110 may calculate the fast point feature histogram (FPFH) or deep neural network (DNN) feature amount SpinNet.

[0033] Subsequently, the sensor parameter estimation unit 110 calculates the L2 norm of each point between the resampled three-dimensional point clouds using the calculated feature amounts, and determines that the point where the calculated L2 norm is the smallest is the matching point. Further, in order to exclude outliers included in the matching point, the sensor parameter estimation unit 110 performs robust estimation using a random sample consensus (RANSAC) algorithm or the like. This allows the sensor parameter estimation unit 110 to roughly estimate the conversionLPWfrom the coordinate system of the sensor 200 into the environment coordinate system.

[0034] For example, first, the sensor parameter estimation unit 110 acquires approximately four points at random from the matching points of the sensors 200 and calculates the position-orientation matrix for projecting the acquired matching points on the matching point of the 3D scanner 400 as the matching destination. Next, the sensor parameter estimation unit 110 projects all the matching points using the calculated position-orientation matrix and measures the number of points where the difference in distance from the matching point as the matching destination is equal to or less than a threshold value. The sensor parameter estimation unit 110 repeats the above acquisition of the matching point and projection a plurality of times and extracts the combination of matching points when the number of points where the difference in distance from the matching point is equal to or less than the threshold value is the largest. The combination of matching points extracted at this time is a combination from which outliers have been excluded. In this regard, the sensor parameter estimation unit 110 uses the extracted combination of matching points to calculate the position-orientation matrix where the error is the smallest when the matching points of the sensors 200 are projected on the matching point of the 3D scanner 400. The calculated position-orientation matrix is used as roughly estimated conversionLPWfrom the coordinate system of the sensor 200 into the environment coordinate system.

[0035] Next, as shown in Fig. 3, the sensor parameter estimation unit 110 estimates, in more detail, the position-orientation matrix of the conversionLPWthat has been roughly estimated in the processing of Step S101 by performing registration using an iterative closest point (ICP) algorithm (S103).

[0036] In the processing of Step S103, the sensor parameter estimation unit 110 estimates the position-orientation matrix of the conversionLPWcalculated in Step S101 with higher accuracy by performing the ICP algorithm using the three-dimensional point cloud before resampling. Specifically, the sensor parameter estimation unit 110 searches for the point nearest to the three-dimensional point cloud of the 3D scanner 400 from the respective points of three-dimensional point clouds of the sensors 200, associates the points with each other, and calculates the position-orientation matrix for minimizing the difference between the points associated with each other. The calculated position-orientation matrix is used as the conversionLPWfrom the coordinate system of the sensor 200 into the environment coordinate system, which has been estimated in more detail. By roughly estimating the position-orientation matrix of the conversionLPWby the processing of Step S101, it is possible to converge the calculation of the position-orientation matrix of the conversionLPWin the processing of Step S103 more quickly.

[0037] After that, the sensor parameter estimation unit 110 executes the above processing of Steps S101 and S103 for each of the sensors 200 mounted on the moving object 10.

[0038] Subsequently, as shown in Fig. 3, the sensor parameter estimation unit 110 is capable of estimating the position of each of the sensors 200 relative to the moving object 10 by using the conversion from the coordinate system of each sensor 200 into the environment coordinate system (S105).

[0039] For example, with reference to conversionMPL0from the designed vehicle center position of the moving object 10 into the position of the 0-th sensor 200, conversionMPLnfrom the vehicle center position into the position of the n-th sensor 200 is represented by the following formula 1. As a result, the sensor parameter estimation unit 110 is capable of estimating the position of each sensor 200 with reference to the designed vehicle center position of the moving object 10.

[0040]

[0041] (Rendering unit 120 and camera parameter estimation unit 130) The rendering unit 120 and the camera parameter estimation unit 130 will be described with reference to Fig. 4 and Fig. 5.

[0042] The camera parameter estimation unit 130 estimates the camera parameter of the camera 300 (i.e., the position of the camera 300 and the internal parameter of the camera 300) on the basis of data of an image of the static environment SE acquired by each camera 300 and three-dimensional data of the static environment SE acquired by the 3D scanner 400. Specifically, the camera parameter estimation unit 130 may estimate the camera parameter by comparing the rendering image rendered on the basis of the three-dimensional data of the static environment SE acquired by the 3D scanner 400 and the camera image of the static environment SE acquired by the camera 300 with each other.

[0043] The rendering image to be used for comparison with the camera image is generated by the rendering unit 120. Specifically, the rendering unit 120 may generate, on the basis of the three-dimensional data of the 3D scanner 400, the rendering image corresponding to the camera image acquired by the camera 300 with reference to a temporarily set camera parameter.

[0044] Specifically, first, the rendering unit 120 generates, as a camera parameter, a rendering image with reference to the position of the camera 300 estimated from the result of the sensor parameter estimation unit 110, the designed value of the internal parameter of the camera 300, and the like. As a result, the camera parameter estimation unit 130 is capable of estimating a camera parameter with less error than the temporarily set camera parameter by comparing the generated rendering image and the camera image with each other. Therefore, the rendering unit 120 is capable of generating a rendering image with a smaller difference from the camera image by generating a rendering image again with reference to the estimated camera parameter. The rendering unit 120 and the camera parameter estimation unit 130 is capable of estimating a camera parameter with less error by repeating the above generation of a rendering image and estimation of a camera parameter.

[0045] In detail, as shown in Fig. 4, first, the camera parameter estimation unit 130 sets a provisional value of the camera parameter to be used when generating a rendering image corresponding to the camera image by the rendering unit 120 (S201).

[0046] Specifically, the camera parameter estimation unit 130 sets a designed value on catalog, an actual value of past correction, an actual value of correction on equivalent products, and the like for the internal parameter of the camera 300 such as the focal length, the center position, and the strain. Further, the camera parameter estimation unit 130 calculates conversionWPCfrom the environment coordinate system to the coordinate system of the camera 300 by using the result of the sensor parameter estimation unit 110. The conversion from the environment coordinate system to the coordinate system of the camera 300 is represented by the following formula 2 usingL0PWfrom the coordinate system of the 0-th sensor 200 estimated by the sensor parameter estimation unit 110 into the environment coordinate system, conversionMPCfrom the vehicle center position into the position of the camera 300, and conversionL0PMfrom the position of the 0-th sensor 200 into the vehicle center position.

[0047]

[0048] Next, the rendering unit 120 generates a rendering image corresponding to the camera image taken by the camera 300 (S203) from the three-dimensional data measured by the 3D scanner 400 using the above provisional value of the camera parameter.

[0049] Subsequently, the camera parameter estimation unit 130 performs feature point matching between the camera image taken by the camera 300 and the generated rendering image (S205). For example, the camera parameter estimation unit 130 may perform feature point matching between the camera image and the rendering image using a classic algorithm such as scale-invariant feature transform (SIFT) or a feature point matching algorithm using a DNN, such as detector-free local feature matching with transformers (LoFTR). However, in the case where the strain of the camera 300 is large, it is possible to achieve better results using a feature point matching algorithm using a DNN.

[0050] Next, the camera parameter estimation unit 130 acquires the three-dimensional position of the matching point by the feature point matching from the rendering image (S207). Specifically, the camera parameter estimation unit 130 is capable of acquiring the three-dimensional position of the matching point by calculating the intersection between the ray (light beam) of the pixel of the matching point in the rendering image and the three-dimensional point cloud or three-dimensional mesh used for generating the rendering image.

[0051] Subsequently, the camera parameter estimation unit 130 estimates the conversionWPCfrom the environment coordinate system into the coordinate system of the camera 300 using the three-dimensional position of the matching point (S209). Specifically, in order to exclude outliers included in the matching point, the camera parameter estimation unit 130 performs robust estimation using a random sample consensus (RANSAC) algorithm or the like. This allows the camera parameter estimation unit 130 to roughly estimate the conversionWPCfrom the environment coordinate system into the coordinate system of the camera 300.

[0052] For example, first, the camera parameter estimation unit 130 acquires approximately four points at random from the matching points and calculates the position-orientation matrix for projecting the three-dimensional positions of the acquired matching points on the camera image. Next, the camera parameter estimation unit 130 projects all the matching points on the camera image using the calculated position-orientation matrix and measures the number of points where the reprojection error of the projected matching point is equal to or less than a threshold value. The camera parameter estimation unit 130 repeats the above acquisition of the matching point and projection a plurality of times and extracts the combination of matching points when the number of points where the reprojection error of the projected matching point is equal to or less than the threshold value is the largest. The combination of matching points extracted at this time is a combination from which outliers have been excluded. In this regard, the camera parameter estimation unit 130 uses the extracted combination of matching points to solve the perspective-n-point (PnP) problem such that the reprojection error is the smallest. As a result, the camera parameter estimation unit 130 is capable of roughly estimating the conversionWPCfrom the environment coordinate system into the coordinate system of the camera 300.

[0053] The above-estimated conversionWPCfrom the environment coordinate system into the coordinate system of the camera 300 is rough estimation. For this reason, the rendering unit 120 and the camera parameter estimation unit 130 perform generation of a rendering image and estimation of a camera parameter again using the estimated conversionWPCfrom the environment coordinate system into the coordinate system of the camera 300. At this time, the camera parameter estimation unit 130 is capable of estimating a camera parameter without including an error during detection of the feature point by detecting the difference between the camera image and the rendering image instead of feature point matching. However, in the case where the internal parameter of the camera 300 such as the focal length, the center position, and the strain is known, the camera parameter estimation unit 130 may exclude these internal parameters from the estimation.

[0054] In detail, as shown in Fig. 5, the rendering unit 120 generates a rendering image corresponding to the camera image taken by the camera 300 on the basis of the three-dimensional data measured by the 3D scanner 400 using the position of the camera 300 estimated in the processing shown in Fig. 4 (S211). Specifically, the rendering unit 120 generates a rendering image corresponding to the camera image taken by the camera 300 on the basis of the three-dimensional data measured by the 3D scanner 400 using the conversionWPCfrom the environment coordinate system into the coordinate system of the camera 300 estimated in the processing shown in Fig. 4 and the internal parameter of the camera 300.

[0055] Next, the camera parameter estimation unit 130 detects a difference L (D(image), D(rendering_image)) between the camera image taken by the camera 300 and the rendering image generated in Step S211 (S213). L represents a robust error function such as the L1 norm, the L2 norm, and the Geman-McClure function. D represents an image conversion function such as image normalization and a descriptor using a gradient (e.g., Robust 3D Tracking with Descriptor Fields). However, in the case where the three-dimensional data measured by the 3D scanner 400 is not color but data obtained by measuring reflection intensity, the camera parameter estimation unit 130 is capable of converging the difference L between the camera image and the rendering image better by using a descriptor using a DNN (Deep Lucas-Kanade Homography for Multimodal Image Alignment).

[0056] Subsequently, the camera parameter estimation unit 130 updates the camera parameter of the camera 300 such that the detected difference L is the smallest (S215). This allows the camera parameter estimation unit 130 to estimate a camera parameter with less error than the camera parameter used for generating a rendering image in the processing of Step S211.

[0057] Note that the camera parameter estimation unit 130 is also capable of estimating camera parameters in a stepwise manner from a low-resolution image to a high-resolution image by creating a Gaussian pyramid. Specifically, the camera parameter estimation unit 130 may generate an image with the resolution reduced to 1 / 2 and an image with the resolution reduced to 1 / 4 by blurring the rendering image and the camera image. The camera parameter estimation unit 130 estimates the camera parameter in the order of the image with the resolution of 1 / 4, the image with the resolution of 1 / 2, and the image before being blurred, making it possible to further improve the robustness of estimation of the camera parameter.

[0058] The camera parameter estimation unit 130 determines, when the processing of Step S215 ends, whether or not the termination condition has been satisfied (S217). The termination condition is, for example, that the detected difference L is a threshold value or less or the number of iterations, which is the number of times the processing of Steps S211 to S215 is repeated, has reached a predetermined number of times.

[0059] In the case where the termination condition has not been satisfied (S217 / No), the rendering unit 120 and the camera parameter estimation unit 130 repeat generation of a rendering image and updating of a camera parameter by repeatedly executing the processing of Steps S211 to S215.

[0060] Meanwhile, in the case where the termination condition has been satisfied (S217 / Yes), the camera parameter estimation unit 130 terminates the iteration of the processing of Steps S211 to S215 and acquires the camera parameter where the difference L is the smallest as an estimation result (S219).

[0061] In the above processing, the conversionWPCfrom the environment coordinate system into the coordinate system of the camera 300 is estimated. For this reason, the camera parameter estimation unit 130 is capable of estimating the position of each camera 300 relative to the moving object 10 by setting a reference on the moving object 10, similarly to the sensor parameter estimation unit 110 (S221).

[0062] For example, the conversionMPCfrom the vehicle center position into the position of the n-th camera 300 is represented by the following formula 3 using, as a reference, the conversionMPL0from the designed vehicle center position of the moving object 10 into the position of the 0-th sensor 200. This allows the camera parameter estimation unit 130 to estimate the position of each camera 300 with reference to the designed vehicle center position of the moving object 10.

[0063]

[0064] In accordance with the above processing, the information processing apparatus 100 is capable of estimating the positions of the sensor 200 and the camera 300 mounted on the moving object 10 with reference to the designed vehicle center position of the moving object 10, or the like. Further, the information processing apparatus 100 is capable of estimating the internal parameter of the camera 300 such as the focal length, the center position, and the strain with higher accuracy on the basis of the designed value.

[0065] In accordance with this, the moving object 10 is capable of integrating and handling the data of an image taken by the camera 300 and the point cloud data sensed by the sensor 200 and thus estimating the surrounding environment and the self-position with higher accuracy by SLAM or the like.

[0066] <3. Modified example> (3.1. First modified example) Fig. 6 is an explanatory diagram describing a first modified example. For example, in the case where the static environment SE is a garage, underground parking lot, or the like, there is a possibility that there are not enough textures to serve as landmarks in the static environment SE.

[0067] In such a case, as shown in Fig. 6, a texture Tx may be projected onto the wall surface or the like of the static environment SE by a projector 500. The texture Tx is, for example, an aperiodic pattern with no identical patterns. The information processing apparatus 100 is capable of more easily detecting the difference between the rendering image generated on the basis of the three-dimensional data measured by the 3D scanner 400 and the camera image taken by the camera 300 by projecting the texture Tx on the static environment SE.

[0068] Note that the texture Tx only needs to be projected only during data acquisition by the 3D scanner 400 and the camera 300 and does not necessarily need to be constantly projected. The three-dimensional data obtained by measuring, by the 3D scanner 400, the static environment SE on which the texture Tx has been projected and the data of an image of the static environment SE on which the texture Tx has been projected taken by the camera 300 can be continuously used as long as the position of the projector 500 and the static environment SE do not change.

[0069] (3.2. Second modified example) Fig. 7 is an explanatory diagram describing a second modified example. For example, the information processing apparatus 100 is also appliable to a camera alone.

[0070] Specifically, as shown in Fig. 7, in the case where the information processing apparatus 100 estimates a camera parameter of a camera 300a, a correction object Sea is prepared. The correction object Sea is an object for providing a static environment to be imaged by the camera 300a. The correction object Sea may be, for example, a hemisphere, polygonal pyramid, rectangular parallelepiped with an internal space that covers most of the imaging range of the camera 300a. Such a correction object Sea can be produced by, for example, a 3D printer.

[0071] A texture is attached to the internal space of the correction object Sea, and the internal space to which the texture is attached is measured by the 3D scanner 400 to measure three-dimensional data of the internal space of the correction object Sea. Further, the internal space to which the texture is attached is imaged by the camera 300a to acquire data of an image of the internal space of the correction object Sea. The information processing apparatus 100 is capable of estimating a camera parameter of the camera 300a by generating a rendering image of the internal space on the basis of the three-dimensional data of the internal space of the correction object Sea and comparing the generated rendering image with the camera image of the internal space of the correction object Sea.

[0072] The processing to be executed by the information processing apparatus 100 in order to estimate a camera parameter of the camera 300a is as described above with reference to Fig. 4 and Fig. 5. However, in the case where differences between n camera images taken by the camera 300a and n corresponding rendering images are detected, the difference L is the sum of the differences between the n camera images and the n rendering images.

[0073] (3.3. Third modified example) Fig. 8 is a flowchart showing a flow of the processing in the preceding stage executed by the camera parameter estimation unit 130 according to a third modified example. For example, in the case where a two-dimensional marker that can be individually identified is embedded in the texture of the static environment SE, the information processing apparatus 100 is capable of estimating the initial value of the internal parameter of the camera 300 using the detected two-dimensional marker.

[0074] For example, in the case where an appropriate designed value on catalog, actual value of past correction, actual value of correction on equivalent products, and the like, which can be initial values of the internal parameter of the camera 300, are not present, no initial value of the internal parameter of the camera 300 is set in some cases. In such a case, there is a possibility that estimation of the camera parameter is unstable and does not converge in the processing in the subsequent stage executed by the camera parameter estimation unit 130 shown in Fig. 5.

[0075] In the third modified example, the camera parameter estimation unit 130 is capable of estimating the initial value of the internal parameter of the camera 300 by using the two-dimensional marker embedded in the texture of the static environment SE. Examples of the two-dimensional marker embedded in the texture of the static environment SE include the ArUco marker of OpenCV (registered trademark).

[0076] Specifically, as shown in Fig. 8, first, the camera parameter estimation unit 130 detects the two-dimensional coordinate of the corner of the two-dimensional marker from the camera image of the camera 300 (S301).

[0077] Next, the camera parameter estimation unit 130 acquires the three-dimensional coordinate of the corner of the two-dimensional marker from which the two-dimensional coordinate has been detected (S303). For example, the rendering unit 120 generates, on the basis of the three-dimensional point cloud or three-dimensional mesh measured by the 3D scanner 400, a rendering image including the two-dimensional marker from which the two-dimensional coordinate has been detected. The camera parameter estimation unit 130 is capable of acquiring the three-dimensional position of the corner of the two-dimensional marker by calculating the intersection between the ray (light beam) of the two-dimensional coordinate of the corner of the two-dimensional marker of the generated rendering image and the three-dimensional point cloud or three-dimensional mesh used for generating the rendering image.

[0078] Subsequently, the camera parameter estimation unit 130 estimates a camera parameter including the internal parameter of the camera 300 by minimizing the reprojection error when the corner of the two-dimensional marker from which the three-dimensional position has been detected is projected on the camera image (S305).

[0079] At this time, the camera parameter estimation unit 130 may estimate all the internal parameters of the camera 300, but estimation of the internal parameter can be made more stable by reducing the number of internal parameters to be estimated. For example, by setting the center position of the camera 300 to be the same as the image center of the camera image, the center position of the camera 300 can be excluded from estimation. Further, regarding the focal length of the camera 300, by setting the focal length in the longitudinal direction and the focal length in the lateral direction to be the same, the number of parameters to be estimated can be reduced by one. Further, regarding the strain of the camera 300, the number of parameters indicating the strain of the camera 300 can be reduced to approximately 1 to 2 to make the estimation more stable.

[0080] The rendering unit 120 and the camera parameter estimation unit 130 is capable of estimating the camera parameter of the camera 300 with higher accuracy similarly by executing the processing in the subsequent stage shown in Fig. 5 using, as an initial value, the camera parameter including the estimated internal parameter of the camera 300.

[0081] <4. Hardware configuration example> A hardware configuration of the information processing apparatus 100 according to this embodiment will be described with reference to Fig. 9. Fig. 9 is a block diagram showing a hardware configuration example of the information processing apparatus 100 according to this embodiment.

[0082] The function of the information processing apparatus 100 according to this embodiment can be realized by cooperation of software and hardware described below. The functions of the sensor parameter estimation unit 110, the rendering unit 120, and the camera parameter estimation unit 130 may be executed by, for example, a CPU 901.

[0083] As shown in Fig. 9, the information processing apparatus 100 includes a central processing unit (CPU) 901, a read only memory (ROM) 902, and a random access memory (RAM) 903.

[0084] The information processing apparatus 100 may further include a host bus 904a, a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a connection port 910, and a communication device 911. The information processing apparatus 100 may include a processing circuit such as a digital signal processor (DSP) and an application specific integrated circuit (ASIC) instead of or in addition to the CPU 901.

[0085] The CPU 901 functions as an arithmetic processing unit or control device and controls the operation in the information processing apparatus 100 in accordance with various programs recorded on the ROM 902, the RAM 903, the storage device 908, or a removable recording medium attached to the drive 909. The ROM 902 stores a program, an operation parameter, and the like to be used by the CPU 901. The RAM 903 temporarily stores a program to be used in execution of the CPU 901, a parameter to be used at the time of the execution, and the like.

[0086] The CPU 901, the ROM 902, and the RAM 903 are connected to each other via the host bus 904a that is capable of performing high-speed data transmission. The host bus 904a is connected to the external bus 904b such as a peripheral component interconnect / interface (PCI) bus via the bridge 904. The external bus 904b is connected to various components via the interface 905.

[0087] The input device 906 is, for example, a device that receives an input from a user, such as a mouse, a keyboard, a touch panel, a button, a switch, and a lever. Note that the input device 906 may be a microphone that detects voice of a user, or the like. The input device 906 may be, for example, a remote control device that uses infrared rays or other radio waves, and may be an external connection device that is compatible with the operation of the information processing apparatus 100.

[0088] The input device 906 further includes an input control circuit that outputs the input signal generated on the basis of information input by a user to the CPU 901. The user can give instructions for inputting various types of data or processing operation to the information processing apparatus 100 by operating the input device 906.

[0089] The output device 907 is a device that is capable of visually or audibly presenting the information acquired or generated by the information processing apparatus 100 to a user. The output device 907 may be, for example, a display device such as a liquid crystal display (LCD), a plasma display panel (PDP), an organic light emitting diode (OLED) display, a hologram, and a projector, a sound output device such as a speaker and a headphone, or a print device such as a printer device. The output device 907 is capable of outputting, as video such as text and an image or a sound such as audio and acoustics, the information acquired by the processing of the information processing apparatus 100.

[0090] The storage device 908 is a data storage device configured as an example of the storage unit of the information processing apparatus 100. The storage device 908 may include, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 908 is capable of storing programs to be executed by the CPU 901, various types of data, various types of data acquired from the outside, and the like.

[0091] The drive 909 is a reading or writing device for a removable recording medium such as a magnetic disc, an optical disc, a magneto-optical disc, and a semiconductor memory, and is built into the information processing apparatus 100 or attached externally. For example, the drive 909 is capable of reading the information recorded on the attached removable recording medium and outputting the read information to the RAM 903. Further, the drive 909 is capable of writing information to the attached removable recording medium.

[0092] The connection port 910 is a port for directly connecting an external connection device to the information processing apparatus 100. The connection port 910 may be, for example, a universal serial bus (USB) port, an IEEE1394 port, a small computer system interface (SCSI) port, or the like. Further, the connection port 910 may be an RS-232C port, an optical audio terminal, a high-definition multimedia interface (HDMI) (registered trademark) port, or the like. The connection port 910 is capable of transmitting / receiving various types of data between the information processing apparatus 100 and the external connection device by being connected to the external connection device.

[0093] The communication device 911 is, for example, a communication interface configured by a communication device for connecting to a communication network 920. The communication device 911 may be, for example, a wired or wireless local area network (LAN), Wi-Fi (registered trademark), Bluetooth (registered trademark), or a communication card for wireless USB (WUSB). Further, the communication device 911 may be a router for optical communication, a router for asymmetric digital subscriber line (ADSL), a modem for various types of communication, or the like.

[0094] The communication device 911 is capable of transmitting / receiving signals and the like to / from, for example, the Internet or another communication device using a predetermined protocol such as TCP / IP. Further, the communication network 920 connected to the communication device 911 is a wired or wirelessly connected network and may be, for example, an Internet communication network, a home LAN, an infrared communication network, a radio wave communication network, or a satellite communication network.

[0095] Note that a program for performing functions equivalent to those of the information processing apparatus 100 may be created in hardware such as the CPU 901, the ROM 902, and the RAM 903 built in a computer. Further, a computer-readable recording medium recoding the program may be provided.

[0096] Although a suitable embodiment of the present disclosure has been described above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such an example. It is clear that a person with ordinary knowledge in the technical field of the present disclosure can conceive various changes or modifications within the scope of the technical idea described in the claims, and it is understood that these also fall within the technical scope of the present disclosure.

[0097] Further, the effects described in the present specification are merely explanatory or illustrative, and are not limitative. That is, the technology according to the present disclosure can have other effects that are obvious to those skilled in the art from the description of the present specification in addition to or instead of the above effects.

[0098] Note that the following configurations also belong to the technical scope of the present disclosure. (1) An information processing apparatus, including: a rendering unit that generates, on the basis of three-dimensional data of an environment measured by a 3D scanner and a camera parameter of a camera that images the environment, a rendering image corresponding to a camera image taken by the camera; and a camera parameter estimation unit that extracts a difference between the camera image and the rendering image and updates the camera parameter such that the difference is smaller. (2) The information processing apparatus according to (1) above, in which the rendering unit and the camera parameter estimation unit respectively repeat generation of the rendering image and updating of the camera parameter until the difference is equal to or less than a threshold value. (3) The information processing apparatus according to (1) or (2) above, in which the camera parameter includes an external parameter indicating a position of the camera and an internal parameter indicating a correspondence between the camera image and the environment. (4) The information processing apparatus according to (3) above, in which an initial value of the external parameter is estimated using a matching point extracted by feature point matching between the camera image and the rendering image generated using a provisional value of the external parameter. (5) The information processing apparatus according to (4) above, in which the camera is mounted on a device on which a sensor that is capable of detecting a position of an object is mounted, and the provisional value of the external parameter is determined on the basis of a position of the sensor. (6) The information processing apparatus according to (5) above, in which the device is a moving object. (7) The information processing apparatus according to (5) or (6) above, further including a sensor parameter estimation unit that estimates a sensor parameter including the position of the sensor. (8) The information processing apparatus according to (7) above, in which the sensor parameter estimation unit estimates the sensor parameter by aligning the three-dimensional data with point cloud data of the environment acquired by the sensor. (9) The information processing apparatus according to (8) above, in which the sensor parameter estimation unit performs the alignment in multiples stages. (10) The information processing apparatus according to any one of (1) to (9) above, in which the environment is a static environment closed on at least three sides. (11) The information processing apparatus according to (10) above, in which a predetermined texture is projected onto a surface of the environment. (12) The information processing apparatus according to (11) above, in which the texture includes a two-dimensional marker that can be individually identified. (13) An information processing method executed by a computer, including: generating, on the basis of three-dimensional data of an environment measured by a 3D scanner and a camera parameter of a camera that images the environment, a rendering image corresponding to a camera image taken by the camera; and extracting a difference between the camera image and the rendering image and updating the camera parameter such that the difference is smaller. (14) A program that causes a computer to function as: a rendering unit that generates, on the basis of three-dimensional data of an environment measured by a 3D scanner and a camera parameter of a camera that images the environment, a rendering image corresponding to a camera image taken by the camera; and a camera parameter estimation unit that extracts a difference between the camera image and the rendering image and updates the camera parameter such that the difference is smaller.

[0099] It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.

[0100] 10 moving object 100 information processing apparatus 110 sensor parameter estimation unit 120 rendering unit 130 camera parameter estimation unit 200 sensor 300 camera 400 3D scanner 500 projector SE static environment Tx texture

Claims

1. An information processing system, comprising: a rendering unit configured to generate, on a basis of three-dimensional data of an environment and a first parameter of a first device that images the environment, a rendering image corresponding to a first device image taken by the first device; and a parameter estimation unit configured to extract a difference between the first device image and the rendering image and update the first parameter such that the difference is smaller, wherein the rendering unit and the parameter estimation unit are each implemented via at least one processor.

2. The information processing system according to claim 1, wherein the rendering unit and the parameter estimation unit are further configured to respectively repeat generation of the rendering image and updating of the first parameter until the difference is equal to or less than a threshold value.

3. The information processing system according to claim 1, wherein the first parameter includes an external parameter indicating a position of the first device and an internal parameter indicating a correspondence between the first device image and the environment.

4. The information processing system according to claim 3, wherein an initial value of the external parameter is estimated using a matching point extracted by feature point matching between the first device image and the rendering image generated using a provisional value of the external parameter.

5. The information processing system according to claim 4, wherein the first device is mounted on a device on which a sensor that is capable of detecting a position of an object is mounted, and the provisional value of the external parameter is determined on a basis of a position of the sensor.

6. The information processing system according to claim 5, wherein the device is a moving object.

7. The information processing system according to claim 5, further comprising a sensor parameter estimation unit configured to estimate a sensor parameter including the position of the sensor, wherein the sensor parameter estimation unit is implemented via at least one processor.

8. The information processing system according to claim 7, wherein the sensor parameter estimation unit is further configured to estimate the sensor parameter by aligning the three-dimensional data with point cloud data of the environment acquired by the sensor.

9. The information processing system according to claim 8, wherein the sensor parameter estimation unit is further configured to perform the alignment in multiples stages.

10. The information processing system according to claim 1, wherein the environment is a static environment closed on at least three sides.

11. The information processing system according to claim 10, wherein a texture is projected onto a surface of the environment.

12. The information processing system according to claim 11, wherein the texture includes a marker that can be individually identified.

13. An information processing method executed by a computer, comprising: generating, on a basis of three-dimensional data of an environment and a first parameter of a first device that images the environment, a rendering image corresponding to a first device image taken by the first device; and extracting a difference between the first device image and the rendering image and updating the first parameter such that the difference is smaller.

14. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute an information processing method, the method comprising: generating, on a basis of three-dimensional data of an environment and a first parameter of a first device that images the environment, a rendering image corresponding to a first device image taken by the first device; and extracting a difference between the first device image and the rendering image and updating the first parameter such that the difference is smaller.

15. The information processing system according to claim 1, wherein the moving object performs automated driving by estimating the environment on a basis of the three-dimensional data.

16. The information processing system according to claim 5, wherein the sensor is LiDAR, a millimeter-wave radar, or an ultrasonic sensor.

17. The information processing system according to claim 1, wherein the first device is an RGB camera, a monochrome camera, or a stereo camera.

18. The information processing system according to claim 1, wherein the circuitry is further configured to estimate internal parameters included in the first parameter, and exclude internal parameters included in the first parameter that are known from the estimation.

19. The information processing system according to claim 18, wherein a stability of the estimation of the internal parameters is increased on a basis of the exclusion of the internal parameters included in the first parameter thar are known.

20. The information processing system according to claim 1, wherein the circuitry is further configured to repeat generation of the rendering image and updating of the first parameter until a number of times of the of the generation of the rendering image and updating of the first parameter has reached a predetermined number of times.

Images