Information processing apparatus, information processing method, and program

Abstract

To further easily correct a camera for imaging an environment.SOLUTION: An information processing apparatus is provided with: a rendering unit for generating a rendered image corresponding to a camera image picked up by a camera based on three-dimensional data of an environment measured by a 3D scanner and a camera parameter of the camera for picking up the environment; and a camera parameter estimation unit for extracting a difference between the camera image and the rendered image and updating the camera parameter so that the difference becomes smaller.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, technology for sensing the surrounding environment using a sensor capable of detecting the position of an object, such as LiDAR (Light Detection And Ranging), and a camera for capturing an image of the object has been developed. For example, in the case of automobiles, technology for autonomous driving is being actively developed by sensing the environment using LiDAR and a camera.

[0003] In such a technology for sensing the surrounding environment, it is important to appropriately integrate the data sensed by the sensor and the image data captured by the camera. Specifically, it is important to correct the positions between the sensor and the camera and the internal parameters of the camera, and to integrate the data sensed by the sensor and the image data captured by the camera.

[0004] For example, the following Patent Document 1 describes the preparation of a marker board for calibrating a camera capable of capturing an image of an object and a sensor capable of detecting the position of the object using a laser. The technology described in Patent Document 1 makes it possible to perform position correction between the camera and the sensor by detecting marker boards installed in multiple positions with a camera and a sensor attached to the same moving object. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-38939 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology described in Patent Document 1 requires a dedicated marker board, making it difficult for general users to implement. In particular, since the camera requires internal parameter correction in addition to position correction with other cameras or sensors, there has been a demand for technology that makes it easier to correct the camera.

[0007] Therefore, the present disclosure proposes a new and improved information processing device, information processing method, and program that can more easily correct a camera that captures an environment. [Means for solving the problem]

[0008] According to the present disclosure, there is provided an information processing device including: a rendering unit that generates a rendering image corresponding to a camera image captured by a camera based on three-dimensional data of an environment measured by a 3D scanner and camera parameters of the camera capturing the environment; and a camera parameter estimation unit that extracts a difference between the camera image and the rendering image and updates the camera parameters so as to reduce the difference.

[0009] In addition, according to the present disclosure, there is provided an information processing method by a computer, comprising: generating a rendering image corresponding to a camera image captured by a camera based on three-dimensional data of an environment measured by a 3D scanner and camera parameters of the camera capturing the environment; extracting a difference between the camera image and the rendering image; and updating the camera parameters so that the difference becomes smaller.

[0010] Furthermore, according to the present disclosure, a program is provided to cause a computer to function as a rendering unit that generates a rendering image corresponding to a camera image captured by a camera based on three-dimensional data of an environment measured by a 3D scanner and camera parameters of the camera capturing the environment, and a camera parameter estimation unit that extracts a difference between the camera image and the rendering image and updates the camera parameters so as to reduce the difference. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is an explanatory diagram illustrating functions realized by an information processing device according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a block diagram showing a functional configuration of the information processing device according to the embodiment. [Diagram 3] FIG. 4 is a flowchart showing the flow of operations executed by a sensor parameter estimation unit. [Figure 4] FIG. 11 is a flowchart showing the flow of the previous processing executed by the camera parameter estimation unit. [Diagram 5] FIG. 11 is a flowchart showing the flow of subsequent processing executed by the camera parameter estimation unit. [Figure 6] FIG. 11 is an explanatory diagram illustrating a first modified example. [Figure 7] FIG. 11 is an explanatory diagram illustrating a second modified example. [Figure 8] FIG. 13 is a flowchart showing the flow of the first stage of processing executed by a camera parameter estimation unit according to a third modified example. [Figure 9] 2 is a block diagram showing an example of a hardware configuration of an information processing device according to the embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0013] The explanation will be given in the following order. 1. Overview 2. Configuration Example 3. Variations 4. Hardware configuration example

[0014] <1. Overview> First, an overview of an information processing device according to an embodiment of the present disclosure will be described with reference to Fig. 1. Fig. 1 is an explanatory diagram illustrating functions realized by the information processing device according to the present embodiment.

[0015] 1, a moving body 10 equipped with a sensor 200 and a camera 300 is placed in a stationary environment SE in which three-dimensional data is measured by a 3D scanner 400. The information processing device according to this embodiment corrects the sensor 200 and the camera 300 based on the three-dimensional data of the stationary environment SE measured by the 3D scanner 400.

[0016] Here, correction of the sensor 200 refers to correcting the position (sensor parameters) of the sensor 200. Correction of the camera 300 refers to correcting the position (external parameters) of the camera 300 and correcting the effects on the image due to the focal length, center position, distortion, etc. (internal parameters) of the camera 300. The external parameters and internal parameters are collectively referred to as camera parameters.

[0017] The 3D scanner 400 is a three-dimensional measuring machine capable of measuring three-dimensional data of the stationary environment SE over 360°. Specifically, the 3D scanner 400 may be a laser scanner capable of measuring the depth and reflection intensity or color of the stationary environment SE over 360°. The 3D scanner 400 can acquire a three-dimensional point cloud or a three-dimensional mesh in which color or reflection intensity is recorded as the three-dimensional data of the stationary environment SE.

[0018] The stationary environment SE is a static environment in which no moving object exists. Specifically, the stationary environment SE may be an indoor environment in which at least three sides of the sensor 200 and the camera 300 mounted on the moving object 10 are enclosed by surfaces. In such a case, the information processing device according to the present embodiment can correct the sensor 200 and the camera 300 with higher accuracy. For example, the stationary environment SE may be an indoor environment such as a garage.

[0019] The moving body 10 is a device equipped with at least one or more sensors 200 and at least one or more cameras 300. The moving body 10 can perform autonomous driving by estimating the surrounding environment and its own position based on, for example, image data captured by the camera 300 and point cloud data sensed by the sensor 200.

[0020] The sensor 200 is a sensor capable of acquiring three-dimensional point cloud data indicating the positions of objects present in the stationary environment SE by irradiating a laser, a millimeter wave, or an ultrasonic wave. The sensor 200 may be, for example, a LiDAR, a millimeter wave radar, or an ultrasonic sensor. Furthermore, a plurality of sensors 200 may be mounted on the moving body 10.

[0021] The camera 300 is an imaging device capable of capturing an image of the still environment SE. The camera 300 may be, for example, an RGB camera, a monochrome camera, or a stereo camera. A plurality of cameras 300 may be mounted on the moving body 10.

[0022] In order to estimate the surrounding environment and self-position of the moving body 10 with higher accuracy, it is important to integrate image data captured by the camera 300 and point cloud data sensed by the sensor 200. For example, the moving body 10 can estimate the surrounding environment and self-position with higher accuracy by combining Visual-SLAM (Simultaneous Localization and Mapping) using image data captured by the camera 300 and LiDAR-SLAM using point cloud data sensed by the sensor 200.

[0023] The information processing device according to this embodiment can more easily correct the sensor 200 and the camera 300 by using three-dimensional data obtained by measuring the still environment SE with the 3D scanner 400. Specifically, the information processing device can more easily correct the position between the camera 300 and the sensor 200 (i.e., the sensor parameters and the external parameters of the camera 300), and the focal length, center position, and effects of distortion and the like on the image (i.e., the internal parameters of the camera 300) by using three-dimensional data obtained by measuring the still environment SE with the 3D scanner 400.

[0024] With this, the moving body 10 can estimate the surrounding environment and its own position with higher accuracy by integrating image data captured by the camera 300 and point cloud data sensed by the sensor 200 with high accuracy.

[0025] <2. Configuration example> Next, an example of the configuration of an information processing device according to this embodiment will be described with reference to Figs. 2 to 5. Fig. 2 is a block diagram showing the functional configuration of the information processing device 100 according to this embodiment. Fig. 3 is a flowchart showing the flow of operations executed by the sensor parameter estimation unit 110. Fig. 4 is a flowchart showing the flow of the first stage of processing executed by the camera parameter estimation unit 130. Fig. 5 is a flowchart showing the flow of the second stage of processing executed by the camera parameter estimation unit 130.

[0026] 2, the information processing device 100 according to this embodiment includes a sensor parameter estimation unit 110, a rendering unit 120, and a camera parameter estimation unit 130. Point cloud data of the still environment SE is input to the information processing device 100 from a plurality of sensors 200 mounted on the moving body 10, and image data of the still environment SE is input to the information processing device 100 from a plurality of cameras 300 mounted on the moving body 10. In addition, three-dimensional data of the still environment SE is input to the information processing device 100 from a 3D scanner 400.

[0027] (Sensor parameter estimation unit 110) The sensor parameter estimation unit 110 will be described with reference to FIG.

[0028] The sensor parameter estimation unit 110 estimates the position of the sensor 200 based on the point cloud data of the stationary environment SE acquired by each of the sensors 200 and the three-dimensional data of the stationary environment SE acquired by the 3D scanner 400. The sensor parameter estimation unit 110 may estimate the position of each of the sensors 200 by registering (aligning) the three-dimensional data of the stationary environment SE measured by the 3D scanner 400 and the point cloud data of the stationary environment SE acquired by the sensor 200.

[0029] For example, the sensor parameter estimation unit 110 may estimate a transformation from the coordinate system of the sensor 200 to an environmental coordinate system based on the 3D scanner 400 by registration between the three-dimensional data of the 3D scanner 400 and the point cloud data of the sensor 200. The coordinate system of the sensor 200 is based on the position of the sensor 200. Therefore, by estimating the transformation from the coordinate system of the sensor 200 to the environmental coordinate system, the sensor parameter estimation unit 110 can identify the positional relationship between the sensors 200 based on the environmental coordinate system. Note that the transformation from the coordinate system of the sensor 200 to the environmental coordinate system L P W is expressed, for example, by a 4×4 position and orientation matrix including rotation and translation.

[0030] 3, the sensor parameter estimation unit 110 first performs registration by point cloud matching (S101). As a result, the sensor parameter estimation unit 110 performs transformation from the coordinate system of the sensor 200 to the environmental coordinate system. L P W can be roughly estimated.

[0031] In the process of step S101, the sensor parameter estimation unit 110 first resamples the three-dimensional point cloud measured by the 3D scanner 400 and the three-dimensional point cloud acquired by the sensor 200 to a predetermined grid size in order to reduce the amount of calculation. Next, the sensor parameter estimation unit 110 calculates feature amounts of each of the resampled three-dimensional point clouds. For example, the sensor parameter estimation unit 110 may calculate a point cloud feature amount FPFH (Fast Point Feature Histogram) or a DNN (Deep Neural Network) feature amount SpinNet as the feature amount of the three-dimensional point cloud.

[0032] Next, the sensor parameter estimation unit 110 calculates the L2 norm of each point between each of the resampled three-dimensional point clouds using the calculated feature amount, and determines the point with the smallest calculated L2 norm as the matching point. Furthermore, the sensor parameter estimation unit 110 performs robust estimation using a RANSAC (RANdom SAmple Consensus) algorithm or the like to remove outliers included in the matching points. As a result, the sensor parameter estimation unit 110 performs a transformation from the coordinate system of the sensor 200 to the environmental coordinate system. L P W can be roughly estimated.

[0033] For example, the sensor parameter estimation unit 110 first randomly acquires about four matching points from the matching points of the sensor 200, and calculates a position and orientation matrix for projecting the acquired matching points onto the matching points of the 3D scanner 400 of the matching destination. Next, the sensor parameter estimation unit 110 projects all matching points using the calculated position and orientation matrix, and measures the number of points whose distance difference with the matching points of the matching destination is equal to or less than a threshold. The sensor parameter estimation unit 110 repeats the acquisition and projection of the above matching points multiple times, and extracts a combination of matching points in which the number of points equal to or less than the threshold is the largest. The combination of matching points extracted at this time is a combination of matching points from which outliers have been removed. Therefore, the sensor parameter estimation unit 110 uses the combination of matching points extracted to calculate a position and orientation matrix that minimizes the error when the matching points of the sensor 200 are projected onto the matching points of the 3D scanner 400. The calculated position and orientation matrix is ​​roughly estimated by converting the coordinate system of the sensor 200 into the environmental coordinate system. L P W It becomes.

[0034] Next, as shown in FIG. 3, the sensor parameter estimation unit 110 performs registration using an ICP (Iterative Closest Point) algorithm to estimate the transformation roughly estimated in the process of step S101. L P W The position and orientation matrix of is estimated in more detail (S103).

[0035] In the process of step S103, the sensor parameter estimation unit 110 performs the ICP algorithm using the 3D point cloud before resampling to obtain the transformation calculated in step S101. L P WThe sensor parameter estimation unit 110 estimates the position and orientation matrix of the sensor 200 with higher accuracy. Specifically, the sensor parameter estimation unit 110 searches for and associates each point of the three-dimensional point cloud of the sensor 200 with the nearest point of the three-dimensional point cloud of the 3D scanner 400, and calculates a position and orientation matrix that minimizes the difference between the associated points. The calculated position and orientation matrix is ​​converted from the coordinate system of the sensor 200 to the environmental coordinate system in a more detailed manner. L P W In the process of step S101, L P W By roughly estimating the position and orientation matrix of L P W This allows the calculation of the position and orientation matrix to converge more quickly.

[0036] Thereafter, the sensor parameter estimation unit 110 executes the above-mentioned processes of steps S101 and S103 for each of the sensors 200 mounted on the moving object 10.

[0037] Next, as shown in FIG. 3, the sensor parameter estimation unit 110 can estimate the position of each of the sensors 200 relative to the moving object 10 by using a transformation from the coordinate system of each of the sensors 200 to the environmental coordinate system (S105).

[0038] For example, conversion from the designed vehicle center position of the moving body 10 to the position of the 0th sensor 200 M P L0 By using as a reference, the conversion from the vehicle center position to the position of the n-th sensor 200 is performed. M P Ln is expressed by the following Equation 1. This allows the sensor parameter estimation unit 110 to estimate the position of each of the sensors 200 with the designed vehicle center position of the moving body 10 as a reference.

[0039]

number

[0040] (Rendering unit 120, camera parameter estimation unit 130) The rendering unit 120 and the camera parameter estimation unit 130 will be described with reference to FIGS.

[0041] The camera parameter estimation unit 130 estimates the camera parameters of the camera 300 (i.e., the position of the camera 300 and the internal parameters of the camera 300) based on the image data of the still environment SE acquired by the camera 300 and the three-dimensional data of the still environment SE acquired by the 3D scanner 400. Specifically, the camera parameter estimation unit 130 may estimate the camera parameters by comparing a rendering image rendered based on the three-dimensional data of the still environment SE acquired by the 3D scanner 400 with a camera image of the still environment SE acquired by the camera 300.

[0042] The rendering image used for comparison with the camera image is generated by the rendering unit 120. Specifically, the rendering unit 120 may generate a rendering image corresponding to the camera image acquired by the camera 300 based on the three-dimensional data of the 3D scanner 400 by referring to provisionally set camera parameters.

[0043] Specifically, the rendering unit 120 first generates a rendering image by referring to the position of the camera 300 estimated from the result of the sensor parameter estimation unit 110, the design values ​​of the internal parameters of the camera 300, and the like, as camera parameters. As a result, the camera parameter estimation unit 130 can estimate camera parameters with less error than the provisionally set camera parameters by comparing the generated rendering image with the camera image. Therefore, the rendering unit 120 can generate a rendering image with a smaller difference from the camera image by referring to the estimated camera parameters. The rendering unit 120 and the camera parameter estimation unit 130 can estimate camera parameters with less error by repeating the generation of the rendering image and the estimation of the camera parameters.

[0044] In detail, as shown in FIG. 4, the camera parameter estimation unit 130 first sets provisional values ​​of camera parameters used when the rendering unit 120 generates a rendering image corresponding to the camera image (S201).

[0045] Specifically, the camera parameter estimation unit 130 sets the design values ​​in the catalog, past correction results, or correction results of a similar product, as the internal parameters such as the focal length, center position, and distortion of the camera 300. In addition, the camera parameter estimation unit 130 uses the results of the sensor parameter estimation unit 110 to perform transformation from the environmental coordinate system to the coordinate system of the camera 300. W P C The transformation from the environmental coordinate system to the coordinate system of the camera 300 is performed by transforming the coordinate system of the 0th sensor 200 estimated by the sensor parameter estimation unit 110 into the environmental coordinate system. L0 P W , transformation from the vehicle center position to the camera 300 position M P C , and the transformation from the position of the 0th sensor 200 to the vehicle center position L0 P M Using this, it is expressed by the following Equation 2.

[0046]

number

[0047] Next, the rendering unit 120 uses the provisional values ​​of the camera parameters to generate a rendering image corresponding to the camera image captured by the camera 300 from the three-dimensional data measured by the 3D scanner 400 (S203).

[0048] Next, the camera parameter estimation unit 130 performs feature point matching between the camera image captured by the camera 300 and the generated rendering image (S205). For example, the camera parameter estimation unit 130 may perform feature point matching between the camera image and the rendering image using a classic algorithm such as SIFT (Scale-Invariant Feature Transform) or a feature point matching algorithm using DNN such as LoFTR (Detector-Free Local Feature Matching with Transformers). However, when the distortion of the camera 300 is large, it is possible to obtain better results by using a feature point matching algorithm using DNN.

[0049] Next, the camera parameter estimation unit 130 obtains the three-dimensional position of the matching point by feature point matching from the rendering image (S207). Specifically, the camera parameter estimation unit 130 can obtain the three-dimensional position of the matching point by calculating the intersection between the ray of the pixel of the matching point in the rendering image and the three-dimensional point cloud or three-dimensional mesh used to generate the rendering image.

[0050] Next, the camera parameter estimation unit 130 uses the three-dimensional positions of the matching points to perform a transformation from the environmental coordinate system to the coordinate system of the camera 300. W P C Specifically, the camera parameter estimation unit 130 performs robust estimation using a random sample consensus (RANSAC) algorithm or the like to remove outliers from the matching points. As a result, the camera parameter estimation unit 130 performs transformation from the environmental coordinate system to the coordinate system of the camera 300. W P C can be roughly estimated.

[0051] For example, the camera parameter estimation unit 130 first randomly acquires about four matching points, and calculates a position and orientation matrix for projecting the three-dimensional positions of the acquired matching points onto the camera image. Next, the camera parameter estimation unit 130 projects all matching points onto the camera image using the calculated position and orientation matrix, and counts the number of points for which the reprojection error of the projected matching points is equal to or less than a threshold. The camera parameter estimation unit 130 repeats the acquisition and projection of the above matching points multiple times, and extracts a combination of matching points for which the number of points equal to or less than the threshold is the largest. The combination of matching points extracted at this time is a combination of matching points from which outliers have been removed. Therefore, the camera parameter estimation unit 130 converts the environmental coordinate system into the coordinate system of the camera 300 by solving a Perspective-n-Point (PnP) problem using the extracted combination of matching points so as to minimize the reprojection error. W P C can be roughly estimated.

[0052] Transformation from the environment coordinate system estimated above to the coordinate system of the camera 300 W P C is a rough estimate. Therefore, the rendering unit 120 and the camera parameter estimation unit 130 perform a transformation from the estimated environment coordinate system to the coordinate system of the camera 300. W P C Using the above, the rendering image is generated and the camera parameters are estimated again. At this time, the camera parameter estimation unit 130 detects the image difference between the camera image and the rendering image, thereby estimating the camera parameters without including errors at the time of detecting the feature points. Note that, if the internal parameters of the camera 300, such as the focal length, center position, and distortion, are known, the camera parameter estimation unit 130 may exclude these internal parameters from the estimation targets.

[0053] 5, the rendering unit 120 generates a rendering image corresponding to the camera image captured by the camera 300 from the three-dimensional data measured by the 3D scanner 400, using the position of the camera 300 estimated in the process shown in Fig. 4 (S211). Specifically, the rendering unit 120 converts the environmental coordinate system estimated in the process shown in Fig. 4 into the coordinate system of the camera 300. W P C and the internal parameters of the camera 300, a rendering image corresponding to the camera image captured by the camera 300 is generated from the three-dimensional data measured by the 3D scanner 400.

[0054] Next, the camera parameter estimation unit 130 detects the difference L(D(image), D(redearing_image) between the camera image captured by the camera 300 and the rendering image generated in step S211 (S213). L is a robust error function such as the L1 norm, the L2 norm, or the Geman-McClure function. D is an image conversion function such as image normalization or a descriptor using a gradient (for example, Robust 3D Tracking with Descriptor Fields, etc.). However, if the three-dimensional data measured by the 3D scanner 400 is data that measures reflection intensity rather than color, the camera parameter estimation unit 130 can better converge the difference L between the camera image and the rendering image by using a descriptor using a DNN (Deep Lucas-Kanade Homography for Multimodal Image Alignment).

[0055] Next, the camera parameter estimation unit 130 updates the camera parameters of the camera 300 so as to minimize the detected difference L (S215). This allows the camera parameter estimation unit 130 to estimate camera parameters with less error than the camera parameters used to generate the rendering image in the process of step S211.

[0056] The camera parameter estimation unit 130 can also estimate the camera parameters in stages from a low-resolution image to a high-resolution image by creating a Gaussian pyramid. Specifically, the camera parameter estimation unit 130 may generate an image with a resolution reduced to 1 / 2 and an image with a resolution reduced to 1 / 4 by blurring the rendering image and the camera image. The camera parameter estimation unit 130 can further increase the robustness of the estimation of the camera parameters by estimating the camera parameters in the order of an image with a 1 / 4 resolution, an image with a 1 / 2 resolution, and an image before blurring.

[0057] Here, when the process of step S215 ends, the camera parameter estimation unit 130 determines whether or not a termination condition is satisfied (S217). The termination condition is, for example, that the detected difference L is equal to or smaller than a threshold value, or that the number of iterations, which is the number of times the processes of steps S211 to S215 are repeatedly executed, reaches a predetermined number.

[0058] If the end condition is not satisfied (S217 / No), the rendering unit 120 and the camera parameter estimation unit 130 repeatedly execute the processes of steps S211 to S215 to repeatedly generate a rendering image and update the camera parameters.

[0059] On the other hand, if the end condition is satisfied (S217 / Yes), the camera parameter estimation unit 130 ends the repetition of the processes of steps S211 to S215, and acquires the camera parameters that minimize the difference L as the estimation result (S219).

[0060] In the above process, the transformation from the environmental coordinate system to the coordinate system of the camera 300 W P C Therefore, similarly to the sensor parameter estimation unit 110, the camera parameter estimation unit 130 can estimate the position of each of the cameras 300 with respect to the moving object 10 by setting a reference on the moving object 10 (S221).

[0061] For example, conversion from the designed vehicle center position of the moving body 10 to the position of the 0th sensor 200 M P L0 By using as a reference, the position of the n-th camera 300 is transformed from the vehicle center position M P C is expressed by the following Equation 3. This allows the camera parameter estimation unit 130 to estimate the position of each of the cameras 300 with the designed vehicle center position of the moving body 10 as a reference.

[0062]

number

[0063] According to the above process, the information processing device 100 can estimate the positions of the sensor 200 and the camera 300 mounted on the moving body 10 based on the designed vehicle center position of the moving body 10. Furthermore, the information processing device 100 can estimate the internal parameters of the camera 300, such as the focal length, center position, and distortion, with high accuracy from the design values.

[0064] This enables the moving body 10 to handle image data captured by the camera 300 and point cloud data sensed by the sensor 200 in an integrated manner, thereby enabling the moving body 10 to estimate the surrounding environment and its own position with higher accuracy using SLAM, etc.

[0065] <3. Modifications> (3.1. First Modification) 6 is an explanatory diagram illustrating a first modified example. For example, when the still environment SE is an underground parking lot or a simple garage, there is a possibility that the still environment SE does not have a sufficient landmark texture.

[0066] In such a case, as shown in Fig. 6, a texture Tx may be projected onto a wall surface or the like of the still environment SE by a projector 500. The texture Tx is, for example, a non-periodic pattern in which no identical pattern exists. By projecting the texture Tx onto the still environment SE, the information processing device 100 can more easily detect the difference between a rendering image generated from three-dimensional data measured by the 3D scanner 400 and a camera image captured by the camera 300.

[0067] The texture Tx does not have to be constantly projected, but only needs to be projected when data is acquired by the 3D scanner 400 and the camera 300. The three-dimensional data obtained by measuring the still environment SE onto which the texture Tx is projected by the 3D scanner 400, and the image data obtained by capturing the still environment SE onto which the texture Tx is projected by the camera 300 can be used continuously as long as the position of the projector 500 and the still environment SE do not change.

[0068] (3.2. Second Modification) 7 is an explanatory diagram illustrating the second modified example. For example, the information processing device 100 can be applied to a camera alone.

[0069] Specifically, as shown in Fig. 7, in the second modified example, when the information processing device 100 estimates the camera parameters of the camera 300a, a correction object SEa is prepared. The correction object SEa is an object that provides a still environment captured by the camera 300a. The correction object SEa may be, for example, a hemisphere, a polygonal pyramid, or a rectangular parallelepiped having an internal space that covers most of the imaging range of the camera 300a. Such a correction object SEa can be manufactured, for example, by a 3D printer.

[0070] A texture is applied to the internal space of the correction object SEa, and the three-dimensional data of the internal space of the correction object SEa is measured by measuring the internal space with the texture applied by the 3D scanner 400. In addition, image data of the internal space of the correction object SEa is acquired by capturing an image of the internal space with the texture applied by the camera 300a. The information processing device 100 generates a rendering image of the internal space from the three-dimensional data of the internal space of the correction object SEa, and compares the generated rendering image with a camera image of the internal space of the correction object SEa, thereby estimating the camera parameters of the camera 300a.

[0071] The process executed by the information processing device 100 to estimate the camera parameters of the camera 300a is as described with reference to Figures 4 and 5. However, when detecting differences between n camera images captured by the camera 300a and the corresponding n rendering images, the difference L is the sum of the differences between the n images.

[0072] (3.3. Third Modification) 8 is a flowchart showing the flow of the first stage of processing executed by the camera parameter estimation unit 130 according to the third modified example. For example, when identifiable two-dimensional markers are embedded in the texture of the still environment SE, the information processing device 100 can estimate the initial values ​​of the internal parameters of the camera 300 by using the detected two-dimensional markers.

[0073] For example, if there are no suitable design values ​​in a catalog, past correction results, or correction results of a comparable product that can serve as initial values ​​of the internal parameters of camera 300, initial values ​​of the internal parameters of camera 300 may not be set. In such a case, the estimation of the camera parameters may not be stable and may not converge in the subsequent processing of camera parameter estimation unit 130 shown in FIG.

[0074] In the third modified example, the camera parameter estimation unit 130 can estimate initial values ​​of the internal parameters of the camera 300 by using two-dimensional markers embedded in the texture of the still environment SE. An example of the two-dimensional marker embedded in the texture of the still environment SE is the ArUco marker of OpenCV (registered trademark).

[0075] Specifically, as shown in FIG. 8, first, the camera parameter estimation unit 130 detects the two-dimensional coordinates of the corners of the two-dimensional marker from the camera image of the camera 300 (S301).

[0076] Next, the camera parameter estimation unit 130 acquires the three-dimensional coordinates of the corners of the two-dimensional markers whose two-dimensional coordinates have been detected (S303). For example, the rendering unit 120 generates a rendering image including the two-dimensional markers whose two-dimensional coordinates have been detected from a three-dimensional point cloud or a three-dimensional mesh measured by the 3D scanner 400. The camera parameter estimation unit 130 can acquire the three-dimensional positions of the corners of the two-dimensional markers by calculating the intersection between a ray of the two-dimensional coordinates of the corners of the two-dimensional markers in the generated rendering image and the three-dimensional point cloud or the three-dimensional mesh used to generate the rendering image.

[0077] Next, the camera parameter estimation unit 130 estimates camera parameters including the internal parameters of the camera 300 by minimizing the reprojection error when the corners of the two-dimensional markers whose three-dimensional positions have been obtained are projected onto the camera image (S305).

[0078] At this time, the camera parameter estimation unit 130 may estimate all of the internal parameters of the camera 300, but by reducing the number of internal parameters to be estimated, it is possible to make the estimation of the internal parameters more stable. For example, the camera parameter estimation unit 130 can exclude the center position of the camera 300 from the estimation by making the center position of the camera 300 the same as the image center of the camera image. Also, the camera parameter estimation unit 130 can reduce the number of parameters to be estimated by one by making the focal length of the camera 300 in the vertical direction and the focal length of the camera 300 the same. Furthermore, the camera parameter estimation unit 130 can stabilize the estimation by reducing the number of parameters representing the distortion of the camera 300 to about 1 to 2.

[0079] The rendering unit 120 and the camera parameter estimation unit 130 perform the subsequent processing shown in FIG. 5 using the estimated camera parameters, including the internal parameters of the camera 300, as initial values, thereby enabling the rendering unit 120 and the camera parameter estimation unit 130 to estimate the camera parameters of the camera 300 with high accuracy.

[0080] <4. Hardware configuration example> Furthermore, the hardware configuration of the information processing device 100 according to this embodiment will be described with reference to Fig. 9. Fig. 9 is a block diagram showing an example of the hardware configuration of the information processing device 100 according to this embodiment.

[0081] The functions of the information processing device 100 according to this embodiment can be realized by cooperation between software and hardware described below. The functions of the sensor parameter estimation unit 110, the rendering unit 120, and the camera parameter estimation unit 130 may be executed by, for example, the CPU 901.

[0082] As shown in FIG. 9, the information processing device 100 includes a central processing unit (CPU) 901, a read only memory (ROM) 902, and a random access memory (RAM) 903.

[0083] The information processing device 100 may further include a host bus 904a, a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a connection port 910, or a communication device 911. Instead of the CPU 901 or together with the CPU 901, the information processing device 100 may have a processing circuit such as a DSP (Digital Signal Processor) or an ASIC (Application Specific Integrated Circuit).

[0084] The CPU 901 functions as an arithmetic processing device or a control device, and controls operations within the information processing device 100 in accordance with various programs recorded in the ROM 902, the RAM 903, the storage device 908, or a removable recording medium attached to the drive 909. The ROM 902 stores programs used by the CPU 901, arithmetic parameters, etc. The RAM 903 temporarily stores programs used in the execution of the CPU 901, and parameters used during the execution, etc.

[0085] The CPU 901, ROM 902, and RAM 903 are connected to one another by a host bus 904a capable of high-speed data transmission. The host bus 904a is connected to an external bus 904b such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 904, and the external bus 904b is connected to various components via an interface 905.

[0086] The input device 906 is a device that accepts input from a user, such as a mouse, a keyboard, a touch panel, a button, a switch, or a lever. The input device 906 may be a microphone that detects the user's voice. The input device 906 may be, for example, a remote control device that uses infrared rays or other radio waves, or may be an externally connected device that supports the operation of the information processing device 100.

[0087] The input device 906 further includes an input control circuit that outputs an input signal generated based on information input by a user to the CPU 901. The user can input various data or instruct the information processing device 100 to perform a processing operation by operating the input device 906.

[0088] The output device 907 is a device capable of visually or aurally presenting information acquired or generated by the information processing device 100 to a user. The output device 907 may be, for example, a display device such as an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), an OLED (Organic Light Emitting Diode) display, a hologram, or a projector, a sound output device such as a speaker or headphones, or a printing device such as a printer. The output device 907 can output information acquired by processing of the information processing device 100 as a video such as text or an image, or a sound such as voice or audio.

[0089] The storage device 908 is a data storage device configured as an example of a storage unit of the information processing device 100. The storage device 908 may be configured, for example, by a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 908 can store programs executed by the CPU 901, various data, various data acquired from the outside, and the like.

[0090] The drive 909 is a device for reading or writing data from or to a removable recording medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and is built into or externally attached to the information processing device 100. For example, the drive 909 can read information recorded on a removable recording medium that is attached to the drive 909 and output the information to the RAM 903. The drive 909 can also write data to the removable recording medium that is attached to the drive 909.

[0091] The connection port 910 is a port for directly connecting an external device to the information processing device 100. The connection port 910 may be, for example, a Universal Serial Bus (USB) port, an IEEE 1394 port, or a Small Computer System Interface (SCSI) port. The connection port 910 may also be an RS-232C port, an optical audio terminal, or a High-Definition Multimedia Interface (HDMI) (registered trademark) port. The connection port 910 is connected to an external device, thereby enabling transmission and reception of various data between the information processing device 100 and the external device.

[0092] The communication device 911 is, for example, a communication interface configured with a communication device for connecting to the communication network 920. The communication device 911 may be, for example, a communication card for a wired or wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), or WUSB (Wireless USB). The communication device 911 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication.

[0093] The communication device 911 can transmit and receive signals, for example, via the Internet or between other communication devices, using a predetermined protocol such as TCP / IP. A communication network 920 connected to the communication device 911 is a wired or wirelessly connected network, and may be, for example, an Internet communication network, a home LAN, an infrared communication network, a radio wave communication network, or a satellite communication network.

[0094] It is also possible to create a program for causing hardware such as the CPU 901, ROM 902, and RAM 903 built into a computer to perform functions equivalent to those of the information processing device 100. It is also possible to provide a computer-readable recording medium on which the program is recorded.

[0095] Although the preferred embodiment of the present disclosure has been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modified or amended examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure.

[0096] In addition, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are apparent to a person skilled in the art from the description of this specification, in addition to or in place of the above effects.

[0097] Note that the following configurations also fall within the technical scope of the present disclosure. (1) A rendering unit that generates a rendering image corresponding to a camera image captured by the camera based on three-dimensional data of the environment measured by the 3D scanner and camera parameters of the camera that captures the environment; a camera parameter estimation unit that extracts a difference between the camera image and the rendering image and updates the camera parameters so that the difference becomes smaller; An information processing device comprising: (2) The information processing device according to (1), wherein the rendering unit and the camera parameter estimation unit repeatedly generate the rendering image and update the camera parameters until the difference becomes equal to or less than a threshold value. (3) The information processing device according to (1) or (2), wherein the camera parameters include external parameters representing a position of the camera and internal parameters indicating a correspondence between the camera image and the environment. (4) The information processing device described in (3), wherein the initial values ​​of the external parameters are estimated using matching points extracted by feature point matching between the camera image and the rendering image generated using provisional values ​​of the external parameters. (5) The camera is mounted on a device equipped with a sensor capable of detecting a position of an object, The information processing device according to (4), wherein the provisional value of the external parameter is determined based on a position of the sensor. (6) The information processing device according to (5), wherein the device is a mobile object. (7) The information processing device according to (5) or (6), further comprising a sensor parameter estimation unit that estimates sensor parameters including a position of the sensor. (8) The information processing device according to (7), wherein the sensor parameter estimation unit estimates the sensor parameters by aligning the three-dimensional data with point cloud data of the environment acquired by the sensor. (9) The information processing device according to (8), wherein the sensor parameter estimation unit performs the alignment in a plurality of stages. (10) The information processing device according to any one of (1) to (9), wherein the environment is a stationary environment enclosed by surfaces on at least three or more sides. (11) The information processing device according to (10), wherein a predetermined texture is projected onto a surface of the environment. (12) The information processing device according to (11), wherein the texture includes two-dimensional markers that are each identifiable. (13) generating a rendering image corresponding to a camera image captured by the camera based on three-dimensional data of the environment measured by the 3D scanner and camera parameters of the camera capturing the environment; extracting a difference between the camera image and the rendering image, and updating the camera parameters so that the difference becomes smaller; A method for processing information by a computer, comprising: (14) Computer, A rendering unit that generates a rendering image corresponding to a camera image captured by the camera based on three-dimensional data of the environment measured by the 3D scanner and camera parameters of the camera that captures the environment; a camera parameter estimation unit that extracts a difference between the camera image and the rendering image and updates the camera parameters so that the difference becomes smaller; A program to function as a [Explanation of symbols]

[0098] 10 Mobile 100 Information processing device 110 Sensor parameter estimation unit 120 Rendering Department 130 Camera parameter estimation unit 200 Sensors 300 Camera 400 3D Scanner 500 Projector SE static environment Tx Texture

Claims

1. A rendering unit that generates a rendering image corresponding to a camera image captured by the camera based on three-dimensional data of the environment measured by the 3D scanner and camera parameters of the camera that captures the environment; a camera parameter estimation unit that extracts a difference between the camera image and the rendering image and updates the camera parameters so that the difference becomes smaller; An information processing device comprising:

2. The information processing apparatus according to claim 1 , wherein the rendering section and the camera parameter estimation section repeatedly generate the rendering image and update the camera parameters until the difference becomes equal to or smaller than a threshold value.

3. The information processing apparatus according to claim 1 , wherein the camera parameters include external parameters representing a position of the camera and internal parameters indicating a correspondence between the camera image and the environment.

4. The information processing apparatus according to claim 3 , wherein the initial values ​​of the external parameters are estimated using matching points extracted by feature point matching between the camera image and the rendering image generated using the provisional values ​​of the external parameters.

5. The camera is mounted on a device equipped with a sensor capable of detecting a position of an object, The information processing apparatus according to claim 4 , wherein the provisional value of the external parameter is determined based on a position of the sensor.

6. The information processing apparatus according to claim 5 , wherein the device is a mobile object.

7. The information processing apparatus according to claim 5 , further comprising a sensor parameter estimation unit that estimates sensor parameters including a position of the sensor.

8. The information processing apparatus according to claim 7 , wherein the sensor parameter estimation unit estimates the sensor parameters by performing alignment between the three-dimensional data and point cloud data of the environment acquired by the sensor.

9. The information processing device according to claim 8 , wherein the sensor parameter estimation unit performs the alignment in a plurality of stages.

10. The information processing apparatus according to claim 1 , wherein the environment is a stationary environment enclosed by surfaces on at least three sides.

11. The information processing apparatus according to claim 10 , wherein a predetermined texture is projected onto a surface of the environment.

12. The information processing device according to claim 11 , wherein the texture includes two-dimensional markers that are individually identifiable.

13. generating a rendering image corresponding to a camera image captured by the camera based on three-dimensional data of the environment measured by the 3D scanner and camera parameters of the camera capturing the environment; extracting a difference between the camera image and the rendering image, and updating the camera parameters so that the difference becomes smaller; A method for processing information by a computer, comprising:

14. Computer, A rendering unit that generates a rendering image corresponding to a camera image captured by the camera based on three-dimensional data of the environment measured by the 3D scanner and camera parameters of the camera that captures the environment; a camera parameter estimation unit that extracts a difference between the camera image and the rendering image and updates the camera parameters so that the difference becomes smaller; A program to function as a

Images