Information processing device, information processing method, and program

Abstract

There is provided an information processing system including: a rendering unit configured to generate, based on three-dimensional data of an environment and a first parameter of a first device that images the environment, a rendered image corresponding to a first device image captured by the first device; and a parameter estimation unit configured to extract a difference between the first device image and the rendered image, and update the first parameter such that the difference is smaller.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of Japanese priority patent application JP 2023-171381, filed on October 2, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to information processing devices, information processing methods, and procedures. Background Technology

[0004] In recent years, advancements have been made in technologies that use sensors capable of detecting the position of objects (such as light detection and ranging (LiDAR)) and camera devices to image those objects to sense the surrounding environment. For example, technologies for autonomous driving in automobiles by using LiDAR and camera devices to sense the surrounding environment are being actively developed.

[0005] In such technologies for sensing the surrounding environment, it is important to correctly integrate the data sensed by the sensors and the image data captured by the camera device. Specifically, the important steps are: calibrating the position between the sensor and the camera device; calibrating the internal parameters of the camera device; and integrating the data sensed by the sensors and the image data captured by the camera device.

[0006] For example, Patent Document 1 below describes the preparation of a marking plate for calibrating each of a camera capable of imaging an object and a sensor capable of detecting the position of an object using a laser beam. In the technology described in Patent Document 1, the position between the camera and the sensor can be corrected by detecting marking plates installed at multiple locations using a camera and a sensor attached to the same moving object.

[0007] Citation List

[0008] Patent documents

[0009] Patent Document 1: Japanese Patent Application Publication No. 2021-38939 Summary of the Invention

[0010] Technical issues

[0011] However, because a dedicated marking plate is required, the technology described in Patent Document 1 is difficult for ordinary users to implement. In particular, since the internal parameters of the camera device also need to be calibrated in addition to the position between the camera device and other camera devices or sensors, there has been a need for a technology that can more easily perform the calibration of the camera device.

[0012] In this regard, this disclosure proposes a new and improved information processing apparatus, information processing method and program that can more easily perform corrections on camera devices that image the environment.

[0013] Solution to the problem

[0014] According to this disclosure, an information processing system is provided, comprising: a rendering unit configured to generate a rendered image corresponding to a first device image captured by the first device based on three-dimensional data of the environment and first parameters of a first device for imaging the environment; and a parameter estimation unit configured to extract the difference between the first device image and the rendered image and update the first parameters to make the difference smaller, wherein the rendering unit and the parameter estimation unit are implemented via at least one processor.

[0015] Furthermore, according to this disclosure, a computer-executed information processing method is provided, comprising: generating a rendered image corresponding to a first device image captured by the first device based on three-dimensional data of the environment and first parameters of a first device for imaging the environment; and extracting the difference between the first device image and the rendered image, and updating the first device parameters to make the difference smaller.

[0016] Furthermore, according to this disclosure, a non-transitory computer-readable medium is provided, on which a program is contained, which, when executed by a computer, causes the computer to perform an information processing method, the method comprising: generating a rendered image corresponding to a first device image captured by the first device based on three-dimensional data of an environment and first parameters of a first device for imaging the environment; and extracting a difference between the first device image and the rendered image, and updating the first parameters such that the difference is smaller. Attached Figure Description

[0017] [ Figure 1 ] Figure 1 This is an explanatory diagram illustrating the functions implemented by an information processing device according to an embodiment of the present disclosure.

[0018] [ Figure 2 ] Figure 2 This is a block diagram illustrating the functional configuration of an information processing apparatus according to an embodiment.

[0019] [ Figure 3 ] Figure 3 This is a flowchart illustrating the operation performed by the sensor parameter estimation unit.

[0020] [ Figure 4 ] Figure 4 This is a flowchart illustrating the process of the previous stage of processing performed by the camera device parameter estimation unit.

[0021] [ Figure 5 ] Figure 5 This is a flowchart illustrating the processing flow of subsequent stages performed by the camera device parameter estimation unit.

[0022] [ Figure 6 ] Figure 6 This is an explanatory diagram describing the first modified example.

[0023] [ Figure 7 ] Figure 7 This is an explanatory diagram describing the second modified example.

[0024] [ Figure 8 ] Figure 8 This is a flowchart illustrating the process of the previous stage of processing performed by the camera device parameter estimation unit according to the third modified example.

[0025] [ Figure 9 ] Figure 9 This is a block diagram illustrating an example of the hardware configuration of an information processing apparatus according to an embodiment. Detailed Implementation

[0026] Embodiments of this disclosure will now be described in detail with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration will be denoted by the same reference numerals, and redundant descriptions will be omitted.

[0027] Note that the descriptions will be presented in the following order.

[0028] 1. Overview

[0029] 2. Configuration Example

[0030] 3. Modify the example

[0031] 4. Hardware Configuration Example

[0032] <1. Overview>

[0033] First, refer to Figure 1 An overview of the information processing apparatus according to embodiments of the present disclosure is described. Figure 1 This is an explanatory diagram illustrating the functions implemented by the information processing device according to this embodiment.

[0034] like Figure 1 As shown, a moving object 10, on which sensors 200 and camera devices 300 are mounted, is placed in a static environment SE, the three-dimensional data of which is measured by a 3D scanner 400. The information processing apparatus according to this embodiment can perform calibration of the sensors 200 and camera devices 300 based on the three-dimensional data of the static environment SE measured by the 3D scanner 400.

[0035] The calibration of sensor 200 refers to correcting the position of sensor 200 (sensor parameters). Furthermore, the calibration of camera device 300 refers to correcting the position of camera device 300 (external parameters) and the effects of camera device 300's focal length, center position, distortion, etc. (internal parameters) on the image. External and internal parameters are also referred to as camera device parameters.

[0036] The 3D scanner 400 is a coordinate measuring machine capable of performing 360° complete coverage measurement of the three-dimensional data of a static environment SE. Specifically, the 3D scanner 400 can be a laser scanner capable of performing 360° complete coverage measurement of the depth and reflectance intensity or color of the static environment SE. The 3D scanner 400 can acquire three-dimensional point clouds or three-dimensional meshes that record color or reflectance intensity as three-dimensional data of the static environment SE.

[0037] The static environment SE is a static environment in which no moving object exists. Specifically, for each of the sensors 200 and the camera device 300 mounted on the moving object 10, the static environment SE can be an environment that is enclosed on at least three sides. In such a case, the information processing apparatus according to this embodiment can perform calibration of the sensors 200 and the camera device 300 with higher accuracy. For example, the static environment SE can be, for example, the indoor environment inside a garage.

[0038] The moving object 10 is a device on which at least one sensor 200 and at least one camera device 300 are mounted. The moving object 10 is capable of autonomous driving by estimating its surrounding environment and its own position based on, for example, image data captured by the camera device 300 and point cloud data sensed by the sensor 200.

[0039] Sensor 200 is a sensor capable of acquiring three-dimensional point cloud data representing the position of an object present in a static environment SE by applying a laser beam, millimeter wave, or ultrasonic wave. Sensor 200 may be, for example, a LiDAR, millimeter-wave radar, or an ultrasonic sensor. Additionally, sensor 200 may include multiple sensors mounted on the moving object 10.

[0040] The camera device 300 is an imaging device capable of capturing images of a static environment SE. The camera device 300 can be, for example, an RGB camera device, a monochrome camera device, or a stereo camera device. The camera device 300 may include multiple camera devices mounted on the moving object 10.

[0041] To estimate the surrounding environment and its own position with greater accuracy, it is important for the moving object 10 to integrate data from images captured by the camera device 300 and point cloud data sensed by the sensor 200. For example, the moving object 10 can estimate the surrounding environment and its own position with greater accuracy by combining visual-SLAM (simultaneous localization and mapping) data from images captured by the camera device 300 and LiDAR-SLAM data from point cloud data sensed by the sensor 200.

[0042] According to this embodiment, the information processing apparatus can more easily perform corrections on the sensor 200 and the camera device 300 by using three-dimensional data obtained from measuring the static environment SE using the 3D scanner 400. Specifically, the information processing apparatus can more easily perform corrections on the following by using three-dimensional data obtained from measuring the static environment SE using the 3D scanner 400: the position between the camera device 300 and the sensor 200 (i.e., sensor parameters and external parameters of the camera device 300) and the effects on the image caused by the focal length, center position, distortion, etc. of the camera device 300 (i.e., internal parameters of the camera device 300).

[0043] Based on this, the moving object 10 can estimate the surrounding environment and its own position with higher accuracy by integrating the data of the image captured by the camera device 300 and the point cloud data sensed by the sensor 200.

[0044] <2. Configuration Example>

[0045] Next, we will refer to Figures 2 to 5 An example configuration of the information processing apparatus according to this embodiment will be described. Figure 2 This is a block diagram illustrating the functional configuration of the information processing apparatus 100 according to this embodiment. Figure 3 This is a flowchart illustrating the operation performed by the sensor parameter estimation unit 110. Figure 4 This is a flowchart illustrating the process of the previous stage of processing performed by the camera device parameter estimation unit 130. Figure 5 This is a flowchart illustrating the processing flow of subsequent stages performed by the camera device parameter estimation unit 130.

[0046] like Figure 2As shown, the information processing apparatus 100 according to this embodiment includes a sensor parameter estimation unit 110, a rendering unit 120, and a camera device parameter estimation unit 130. The information processing apparatus 100 is, for example, a personal computer, laptop computer, smartphone, tablet computer, server, etc., but the information processing apparatus 100 according to aspects of this disclosure is not limited to such examples. For example, the information processing apparatus 100 can be implemented by an information processing system in which the sensor parameter estimation unit 110, the rendering unit 120, and the camera device parameter estimation unit 130 are included in different information processing devices, or the information processing system may include only one of the different information processing devices. Assume that point cloud data of the static environment SE is input to the information processing apparatus 100 from multiple sensors 200 mounted on the moving object 10, and image data of the static environment SE is input to the information processing apparatus 100 from multiple camera devices 300 mounted on the moving object 10. Furthermore, three-dimensional data of the static environment SE is input to the information processing apparatus 100 from a 3D scanner 400.

[0047] (Sensor parameter estimation unit 110)

[0048] Reference Figure 3 The sensor parameter estimation unit 110 is described.

[0049] The sensor parameter estimation unit 110 estimates the position of each sensor 200 based on point cloud data of the static environment SE acquired by each of the sensors 200 and 3D data of the static environment SE acquired by the 3D scanner 400. The sensor parameter estimation unit 110 can estimate the position of each sensor 200 by registering (aligning) the 3D data of the static environment SE measured by the 3D scanner 400 with the point cloud data of the static environment SE acquired by the sensors 200.

[0050] For example, the sensor parameter estimation unit 110 can estimate the transformation from the coordinate system of sensor 200 to the environmental coordinate system based on 3D scanner 400 by registering the 3D data of 3D scanner 400 and the point cloud data of sensor 200. The coordinate system of sensor 200 is based on the position of sensor 200. Therefore, the sensor parameter estimation unit 110 can identify the positional relationship between each sensor 200 by referencing the environmental coordinate system by estimating the transformation from the coordinate system of sensor 200 to the environmental coordinate system. Note the transformation from the coordinate system of sensor 200 to the environmental coordinate system. L P W For example, 4 including rotation and movement 4. Position orientation matrix representation.

[0051] Specifically, such as Figure 3As shown, the sensor parameter estimation unit 110 first performs registration (S101) through point cloud matching. Thus, the sensor parameter estimation unit 110 can roughly estimate the transformation from the coordinate system of the sensor 200 to the environmental coordinate system. L P W .

[0052] In step S101, the sensor parameter estimation unit 110 first resamples the 3D point cloud measured by the 3D scanner 400 and the 3D point cloud acquired by the sensor 200 to a predetermined grid size to reduce computational load. Next, the sensor parameter estimation unit 110 calculates the feature quantities of each resampled 3D point cloud. For example, the sensor parameter estimation unit 110 can calculate Fast Point Feature Histogram (FPFH) or Deep Neural Network (DNN) SpinNet feature quantities.

[0053] Subsequently, the sensor parameter estimation unit 110 uses the calculated feature quantities to calculate the L2 norm of each point in the resampled 3D point cloud, and determines the point with the smallest calculated L2 norm as the matching point. Furthermore, to exclude outliers from the matching points, the sensor parameter estimation unit 110 uses algorithms such as Random Sample Consensus (RANSAC) for robust estimation. This allows the sensor parameter estimation unit 110 to roughly estimate the transformation from the sensor 200's coordinate system to the environment's coordinate system. L P W .

[0054] For example, firstly, the sensor parameter estimation unit 110 randomly acquires approximately four points from the matching points of the sensor 200 and calculates a position orientation matrix for projecting the acquired matching points onto the matching points of the 3D scanner 400, which serves as the matching destination. Next, the sensor parameter estimation unit 110 uses the calculated position orientation matrix to project all the matching points and measures the number of points whose distance difference from the matching point is equal to or less than a threshold. The sensor parameter estimation unit 110 repeats the acquisition and projection of matching points multiple times and extracts the combination of matching points that maximizes the number of points whose distance difference from the matching point is equal to or less than the threshold. This extracted combination of matching points excludes outliers. In this respect, the sensor parameter estimation unit 110 uses the extracted combination of matching points to calculate the position orientation matrix with the minimum error when the matching points of the sensor 200 are projected onto the matching points of the 3D scanner 400. The calculated position orientation matrix is ​​used as a coarse estimation transformation from the coordinate system of the sensor 200 to the environmental coordinate system. L P W .

[0055] Next, as Figure 3As shown, the sensor parameter estimation unit 110 performs registration (S103) using the Iterative Closest Point (ICP) algorithm, which modulates the parameters that were roughly estimated in step S101. L P W The position orientation matrix is ​​estimated in more detail.

[0056] In step S103, the sensor parameter estimation unit 110 performs the ICP algorithm using the 3D point cloud before resampling to estimate the transformation calculated in step S101 with higher accuracy. L P W The position orientation matrix is ​​calculated. Specifically, the sensor parameter estimation unit 110 searches for points in the 3D point cloud of the sensor 200 that are closest to the 3D point cloud of the 3D scanner 400, correlates these points with each other, and calculates a position orientation matrix that minimizes the differences between the correlated points. The calculated position orientation matrix is ​​used as a transformation for a more detailed estimation from the coordinate system of the sensor 200 to the environmental coordinate system. L P W The conversion is performed through the process in step S101. L P W A rough estimate of the position orientation matrix can facilitate the transformation in step S103. L P W The calculation of the position orientation matrix converges faster.

[0057] Subsequently, the sensor parameter estimation unit 110 performs the above-described steps S101 and S103 for each of the sensors 200 mounted on the moving object 10.

[0058] Subsequently, as Figure 3 As shown, the sensor parameter estimation unit 110 is able to estimate the position of each of the sensors 200 relative to the moving object 10 by using a transformation from the coordinate system of each sensor 200 to the environmental coordinate system (S105).

[0059] For example, refer to the transformation from the design vehicle center position of the moving object 10 to the position of the 0th sensor 200. M P L0 The transition from the vehicle center position to the position of the nth sensor 200 M P Ln This is expressed by the following formula 1. Therefore, the sensor parameter estimation unit 110 can estimate the position of each sensor 200 by referring to the designed vehicle center position of the moving object 10.

[0060] [Mathematical Expression 1]

[0061]

[0062] (Rendering unit 120 and camera device parameter estimation unit 130)

[0063] Reference Figure 4 and Figure 5 The rendering unit 120 and the camera device parameter estimation unit 130 are described.

[0064] The camera device parameter estimation unit 130 estimates the camera device parameters (i.e., the position and internal parameters of the camera device 300) of the camera device 300 based on image data of the static environment SE acquired by each camera device 300 and three-dimensional data of the static environment SE acquired by the 3D scanner 400. Specifically, the camera device parameter estimation unit 130 can estimate the camera device parameters by comparing a rendered image based on the three-dimensional data of the static environment SE acquired by the 3D scanner 400 with the camera device image of the static environment SE acquired by the camera device 300.

[0065] The rendering unit 120 generates a rendered image for comparison with the image from the camera device. Specifically, the rendering unit 120 can generate a rendered image corresponding to the image from the camera device 300 based on the 3D data from the 3D scanner 400 and with reference to temporarily set camera device parameters.

[0066] Specifically, firstly, the rendering unit 120 generates a rendered image as camera device parameters, based on the position of the camera device 300 and the design values ​​of the internal parameters of the camera device 300 estimated according to the results of the sensor parameter estimation unit 110. Therefore, the camera device parameter estimation unit 130 can estimate camera device parameters with smaller errors than the temporarily set camera device parameters by comparing the generated rendered image with the camera device image. Thus, the rendering unit 120 can generate a rendered image with smaller differences from the camera device image by regenerating the rendered image with reference to the estimated camera device parameters. The rendering unit 120 and the camera device parameter estimation unit 130 can estimate camera device parameters with smaller errors by repeating the above-described process of generating the rendered image and estimating the camera device parameters.

[0067] Specifically, such as Figure 4 As shown, firstly, the camera device parameter estimation unit 130 sets provisional values ​​for the camera device parameters to be used when the rendering unit 120 generates a rendered image corresponding to the camera device image (S201).

[0068] Specifically, the camera device parameter estimation unit 130 sets the internal parameters of the camera device 300 (such as focal length, center position, and distortion) to design values ​​from the product catalog, actual values ​​calibrated in the past, and actual values ​​calibrated on the same product. Furthermore, the camera device parameter estimation unit 130 calculates the transformation from the environmental coordinate system to the coordinate system of the camera device 300 using the results from the sensor parameter estimation unit 110. W P C The transformation from the environmental coordinate system to the coordinate system of the camera device 300 uses the coordinate system estimated by the sensor parameter estimation unit 110 from the coordinate system of the 0th sensor 200 to the environmental coordinate system. L0 P W The transition from the center of the vehicle to the position of the camera device 300 M P C And the conversion from the position of sensor 0 200 to the vehicle center position. L0 P M It can be represented by the following formula 2.

[0069] [Mathematical Expression 2]

[0070]

[0071] Next, the rendering unit 120 uses the provisional values ​​of the camera device parameters mentioned above to generate a rendered image corresponding to the camera device image captured by the camera device 300 based on the three-dimensional data measured by the 3D scanner 400 (S203).

[0072] Subsequently, the camera device parameter estimation unit 130 performs feature point matching (S205) between the camera device image captured by the camera device 300 and the generated rendered image. For example, the camera device parameter estimation unit 130 can use classical algorithms (e.g., Scale Invariant Feature Transform (SIFT)) or feature point matching algorithms using DNNs (e.g., Transformer-based detectorless local feature matching (LoFTR)) to perform feature point matching between the camera device image and the rendered image. However, when the distortion of the camera device 300 is large, the feature point matching algorithm using DNNs can obtain better results.

[0073] Next, the camera device parameter estimation unit 130 obtains the three-dimensional position of the matching point from the rendered image through feature point matching (S207). Specifically, the camera device parameter estimation unit 130 can obtain the three-dimensional position of the matching point by calculating the intersection between the ray (beam) of the pixel of the matching point in the rendered image and the three-dimensional point cloud or three-dimensional mesh used to generate the rendered image.

[0074] Next, the camera device parameter estimation unit 130 uses the three-dimensional position of the matching point to estimate the transformation from the environmental coordinate system to the coordinate system of the camera device 300. W P C (S209). Specifically, in order to exclude outliers from the matching points, the camera device parameter estimation unit 130 uses a robust estimation method such as Random Sample Consensus (RANSAC). This enables the camera device parameter estimation unit 130 to roughly estimate the transformation from the environmental coordinate system to the coordinate system of the camera device 300. W P C .

[0075] For example, firstly, the camera device parameter estimation unit 130 randomly selects approximately four points from the matching points and calculates a position orientation matrix for projecting the three-dimensional positions of the selected matching points onto the camera device image. Next, the camera device parameter estimation unit 130 uses the calculated position orientation matrix to project all matching points onto the camera device image and measures the number of points whose reprojection error is equal to or less than a threshold. The camera device parameter estimation unit 130 repeats the acquisition and projection of matching points multiple times and extracts the combination of matching points that maximizes the number of points whose reprojection error is equal to or less than the threshold. This extracted combination of matching points excludes outliers. In this respect, the camera device parameter estimation unit 130 uses the extracted combination of matching points to solve the perspective n-point (PnP) problem, minimizing the reprojection error. Thus, the camera device parameter estimation unit 130 can roughly estimate the transformation from the environmental coordinate system to the coordinate system of the camera device 300. W P C .

[0076] The above-mentioned estimated transformation from the environmental coordinate system to the coordinate system of the camera device 300 W P C It is a rough estimate. Therefore, the rendering unit 120 and the camera device parameter estimation unit 130 use the estimated transformation from the environment coordinate system to the coordinate system of the camera device 300. W P C The generation of the rendered image and the estimation of the camera device parameters are then performed again. At this time, the camera device parameter estimation unit 130 can estimate the camera device parameters without including errors during feature point detection by detecting the difference between the camera device image and the rendered image, rather than by feature point matching. However, if the intrinsic parameters of the camera device 300 (such as focal length, center position, and distortion) are known, the camera device parameter estimation unit 130 can exclude these intrinsic parameters from the estimation.

[0077] Specifically, such as Figure 5 As shown, the rendering unit 120 is used in Figure 4The process shown estimates the position of the camera device 300 and generates a rendered image corresponding to the image of the camera device captured by the camera device 300 based on the three-dimensional data measured by the 3D scanner 400 (S211). Specifically, the rendering unit 120 uses... Figure 4 The transformation from the environmental coordinate system to the coordinate system of the camera device 300 estimated in the processing shown. W P C The internal parameters of the camera device 300 are used to generate a rendered image corresponding to the image captured by the camera device 300, based on the three-dimensional data measured by the 3D scanner 400.

[0078] Next, the camera device parameter estimation unit 130 detects the difference L(D(image), D(rendered image)) between the camera device image captured by the camera device 300 and the rendered image generated in step S211 (S213). L represents a robust error function, such as the L1 norm, L2 norm, and Geman-McClure function. D represents an image transformation function, such as image normalization and a descriptor using gradients (e.g., Robust 3D Tracking with Descriptor Fields). However, in cases where the 3D data measured by the 3D scanner 400 is not color but data obtained by measuring reflection intensity, the camera device parameter estimation unit 130 can better converge the difference L between the camera device image and the rendered image by utilizing a descriptor using a DNN (Deep Lucas-Kanade Homography for Multimodal Image Alignment).

[0079] Subsequently, the camera device parameter estimation unit 130 updates the camera device parameters of the camera device 300 to minimize the detected difference L (S215). This enables the camera device parameter estimation unit 130 to estimate camera device parameters with smaller errors than the camera device parameters used to generate the rendered image in step S211.

[0080] Note that the camera device parameter estimation unit 130 can also estimate camera device parameters in a stepwise manner from low-resolution images to high-resolution images by creating a Gaussian pyramid. Specifically, the camera device parameter estimation unit 130 can generate images with resolution reduced to 1 / 2 and 1 / 4 by blurring the rendered image and the camera device image. The camera device parameter estimation unit 130 estimates the camera device parameters in the order of the 1 / 4 resolution image, the 1 / 2 resolution image, and the image before blurring, which further improves the robustness of estimating the camera device parameters.

[0081] The camera device parameter estimation unit 130 determines whether a termination condition (S217) is met at the end of the processing in step S215. The termination condition is, for example, that the detected difference L is equal to or less than a threshold, or that the number of iterations (the number of times the processing in steps S211 to S215 is repeated) reaches a predetermined number.

[0082] If the termination condition is not met (S217 / No), the rendering unit 120 and the camera device parameter estimation unit 130 repeatedly generate the rendered image and update the camera device parameters by repeatedly executing the processing steps S211 to S215.

[0083] Meanwhile, if the termination condition is met (S217 / Yes), the camera device parameter estimation unit 130 terminates the iteration of the processing steps S211 to S215, and obtains the camera device parameter with the smallest difference L as the estimation result (S219).

[0084] In the above processing, the transformation from the environmental coordinate system to the coordinate system of the camera device 300 is estimated. W P C Therefore, the camera device parameter estimation unit 130 can estimate the position of each camera device 300 relative to the moving object 10 by setting a reference on the moving object 10, similar to the sensor parameter estimation unit 110 (S221).

[0085] For example, the transition from the center position of the vehicle to the position of the nth camera device 300. M P C Using the transformation from the design vehicle center position of the moving object 10 to the position of the 0th sensor 200. M P L0 For reference, this is expressed by the following formula 3. This enables the camera device parameter estimation unit 130 to estimate the position of each camera device 300 by referring to the design vehicle center position of the moving object 10.

[0086] [Mathematical Expression 3]

[0087]

[0088] Based on the above processing, the information processing device 100 can estimate the positions of the sensors 200 and the camera device 300 mounted on the moving object 10 by referring to the design vehicle center position, etc. Furthermore, the information processing device 100 can estimate the internal parameters of the camera device 300 (such as focal length, center position, and distortion) with higher accuracy based on design values.

[0089] Therefore, the moving object 10 is able to integrate and process data from images captured by the camera device 300 and point cloud data sensed by the sensor 200, and thus estimate its surrounding environment and its own position with higher accuracy through SLAM and the like.

[0090] <3. Modification Example>

[0091] (3.1. First Modification Example)

[0092] Figure 6 This is an explanatory diagram describing the first modified example. For example, in the case where the static environment SE is a garage, underground parking lot, etc., there is a possibility that there are not enough textures as landmarks in the static environment SE.

[0093] In such a situation, such as Figure 6 As shown, a texture Tx can be projected onto a wall or other surface of a static environment SE using a projector 500. The texture Tx is, for example, a non-periodic pattern without identical repeating patterns. By projecting the texture Tx onto the static environment SE, the information processing device 100 can more easily detect the difference between a rendered image generated based on three-dimensional data measured by the 3D scanner 400 and an image captured by the camera device 300.

[0094] Note that the texture Tx only needs to be projected during the data acquisition process of the 3D scanner 400 and the camera device 300, and does not necessarily need to be continuously projected. As long as the positions of the projector 500 and the static environment SE do not change, the data obtained by the 3D scanner 400 measuring the static environment SE on which the texture Tx is projected and the data of the image of the static environment SE on which the texture Tx is projected by the camera device 300 can be used continuously.

[0095] (3.2. Second Modification Example)

[0096] Figure 7 This is an explanatory diagram depicting a second modified example. For example, the information processing device 100 can also be applied to a standalone camera device.

[0097] Specifically, such as Figure 7As shown, when the information processing device 100 estimates the imaging device parameters of the imaging device 300a, a calibration object Sea is prepared. This calibration object Sea is an object used to provide a static environment for imaging by the imaging device 300a. The calibration object Sea can be, for example, a hemisphere, a pyramid, or a cuboid having an internal space covering most of the imaging range of the imaging device 300a. Such a calibration object Sea can be manufactured, for example, by a 3D printer.

[0098] A texture is attached to the interior space of the calibration object Sea, and the interior space of the attached texture is measured by a 3D scanner 400 to measure the three-dimensional data of the interior space of the calibration object Sea. Additionally, an imaging device 300a images the interior space of the attached texture to obtain image data of the interior space of the calibration object Sea. The information processing device 100 is able to generate a rendered image of the interior space based on the three-dimensional data of the interior space of the calibration object Sea and compare the generated rendered image with the image of the interior space of the calibration object Sea captured by the imaging device to estimate the imaging device parameters of the imaging device 300a.

[0099] The processing performed by the information processing device 100 to estimate the camera device parameters of the camera device 300a is as described above. Figure 4 and Figure 5 As described. However, when the difference between the n camera images captured by the camera device 300a and the n corresponding rendered images is detected, the difference L is the sum of the differences between the n camera images and the n rendered images.

[0100] (3.3. Third Modification Example)

[0101] Figure 8 This is a flowchart illustrating the process of the previous stage of processing performed by the camera device parameter estimation unit 130 according to the third modified example. For example, in the case where individually identifiable two-dimensional markers are embedded in the texture of a static environment SE, the information processing device 100 is able to use the detected two-dimensional markers to estimate the initial values ​​of the internal parameters of the camera device 300.

[0102] For example, in cases where there are no suitable design values ​​from a product catalog, past calibrated actual values, or calibrated actual values ​​from equivalent products that can serve as initial values ​​for the internal parameters of the camera device 300, initial values ​​for the internal parameters of the camera device 300 may not be set. In such cases, Figure 5 In the subsequent processing stage performed by the camera device parameter estimation unit 130 shown, the estimation of camera device parameters may be unstable and non-convergent.

[0103] In the third modified example, the camera device parameter estimation unit 130 is able to estimate the initial values ​​of the internal parameters of the camera device 300 by using two-dimensional markers embedded in the texture of the static environment SE. Examples of two-dimensional markers embedded in the texture of the static environment SE include the OpenCV (registered trademark) ArUco markers.

[0104] Specifically, such as Figure 8 As shown, firstly, the camera device parameter estimation unit 130 detects the two-dimensional coordinates of the corner point of the two-dimensional mark from the camera device image of the camera device 300 (S301).

[0105] Next, the camera device parameter estimation unit 130 acquires the three-dimensional coordinates of the corner points of the two-dimensional markers from which the two-dimensional coordinates have been detected (S303). For example, the rendering unit 120 generates a rendered image including the two-dimensional markers from which the two-dimensional coordinates have been detected, based on a three-dimensional point cloud or three-dimensional mesh measured by the 3D scanner 400. The camera device parameter estimation unit 130 is able to acquire the three-dimensional position of the corner points of the two-dimensional markers by calculating the intersection point between the ray (beam) of the two-dimensional coordinates of the corner points of the two-dimensional markers in the generated rendered image and the three-dimensional point cloud or three-dimensional mesh used to generate the rendered image.

[0106] Subsequently, the camera device parameter estimation unit 130 estimates the camera device parameters, including the internal parameters of the camera device 300, by minimizing the reprojection error when projecting the corner points of the two-dimensional markers that have been detected from the three-dimensional position onto the camera device image (S305).

[0107] At this time, the camera device parameter estimation unit 130 can estimate all the internal parameters of the camera device 300, but by reducing the number of internal parameters to be estimated, the estimation of the internal parameters can be made more stable. For example, by setting the center position of the camera device 300 to be the same as the image center of the image of the camera device, the center position of the camera device 300 can be excluded from the estimation. In addition, regarding the focal length of the camera device 300, by setting the vertical focal length and the horizontal focal length to be the same, the number of parameters to be estimated can be reduced by one. Furthermore, regarding the distortion of the camera device 300, the number of parameters representing the distortion of the camera device 300 can be reduced to about one or two, so as to make the estimation more stable.

[0108] The rendering unit 120 and the camera device parameter estimation unit 130 can perform operations by using camera device parameters, including the estimated internal parameters of the camera device 300, as initial values. Figure 5 The subsequent processing shown is used to similarly estimate the camera device parameters of the camera device 300 with greater accuracy.

[0109] <4. Hardware Configuration Example>

[0110] Reference Figure 9 The hardware configuration of the information processing apparatus 100 according to this embodiment will be described. Figure 9 This is a block diagram illustrating an example of the hardware configuration of the information processing apparatus 100 according to this embodiment.

[0111] The functions of the information processing apparatus 100 according to this embodiment can be achieved through the cooperation of software and hardware as described below. The functions of the sensor parameter estimation unit 110, the rendering unit 120, and the camera device parameter estimation unit 130 can be executed by, for example, a CPU 901.

[0112] like Figure 9 As shown, the information processing device 100 includes a central processing unit (CPU) 901, a read-only memory (ROM) 902, and a random access memory (RAM) 903.

[0113] The information processing device 100 may also include a host bus 904a, a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a driver 909, a connection port 910, and a communication device 911. Instead of CPU 901 or other components, the information processing device 100 may include processing circuitry such as a digital signal processor (DSP) or an application-specific integrated circuit (ASIC).

[0114] The CPU 901 functions as an arithmetic processing unit or control device, and controls various operations within the information processing apparatus 100 according to various programs recorded on ROM 902, RAM 903, storage device 908, or a removable recording medium attached to driver 909. ROM 902 stores programs, operating parameters, etc., to be used by the CPU 901. RAM 903 temporarily stores programs to be used during CPU 901 execution, parameters to be appropriately used during execution, etc.

[0115] CPU 901, ROM 902, and RAM 903 are interconnected via a host bus 904a capable of performing high-speed data transfer. The host bus 904a is connected via a bridge 904 to an external bus 904b, such as a Peripheral Component Interconnect / Interface (PCI) bus. The external bus 904b is connected to various components via an interface 905.

[0116] Input device 906 may be a device that receives input from the user (such as a mouse, keyboard, touchpad, button, switch, and joystick). Note that input device 906 may be a microphone or the like that that detects the user's voice. Input device 906 may be a remote control device that uses, for example, infrared or other radio waves, and may be an external connection device compatible with the operation of information processing device 100.

[0117] The input device 906 also includes an input control circuit that outputs an input signal generated based on user input to the CPU 901. The user can operate the input device 906 to give instructions to the information processing device 100 for inputting various types of data or performing processing operations.

[0118] Output device 907 is a device capable of visually or audibly presenting information acquired or generated by information processing device 100 to a user. Output device 907 may be, for example, a display device (such as a liquid crystal display (LCD), plasma display panel (PDP), organic light-emitting diode (OLED) display, hologram, and projector), a sound output device (such as a speaker and headphones), or a printing device (such as a printer). Output device 907 is capable of outputting information obtained through processing by information processing device 100 as video (such as text and images) or sound (such as audio and sound effects).

[0119] Storage device 908 is an example data storage device configured as a storage unit of information processing apparatus 100. Storage device 908 may include, for example, magnetic storage devices (e.g., hard disk drives (HDDs)), semiconductor storage devices, optical storage devices, or magneto-optical storage devices. Storage device 908 is capable of storing programs executed by CPU 901, various types of data, various types of data acquired from external sources, etc.

[0120] The drive 909 is a read / write device for removable recording media such as magnetic disks, optical disks, magneto-optical disks, and semiconductor memories, and is either built into the information processing device 100 or attached externally. For example, the drive 909 can read information recorded on the attached removable recording medium and output the read information to RAM 903. Furthermore, the drive 909 can write information to the attached removable recording medium.

[0121] Connection port 910 is a port used to directly connect external devices to information processing device 100. Connection port 910 can be, for example, a Universal Serial Bus (USB) port, an IEEE 1394 port, a Small Computer System Interface (SCSI) port, etc. Furthermore, connection port 910 can be an RS-232C port, an optical audio terminal, a High Definition Multimedia Interface (HDMI) (registered trademark) port, etc. Connection port 910 enables the transmission / reception of various types of data between information processing device 100 and external devices by connecting to them.

[0122] Communication device 911 is, for example, a communication interface configured by a communication device for connecting to communication network 920. For example, communication device 911 can be a wired or wireless local area network (LAN), Wi-Fi (registered trademark), Bluetooth (registered trademark), or a communication card for wireless USB (WUSB). Furthermore, communication device 911 can be a router for optical communication, a router for asymmetric digital subscriber line (ADSL), a modem for various types of communication, etc.

[0123] The communication device 911 is capable of sending signals to, for example, the Internet or another communication device and receiving signals from, for example, the Internet or another communication device using a predetermined protocol such as TCP / IP. Furthermore, the communication network 920 connected to the communication device 911 is a wired or wireless network, and can be, for example, an Internet communication network, a home LAN, an infrared communication network, a radio wave communication network, or a satellite communication network.

[0124] Note that a program for performing functions equivalent to those of the information processing device 100 can be generated in the hardware built into the computer (such as CPU 901, ROM 902, and RAM 903). Furthermore, a computer-readable recording medium containing the program can be provided.

[0125] Although embodiments of the present disclosure have been described above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It will be apparent to those skilled in the art that various changes or modifications will occur to them within the scope of the technical concept described in the claims, and it is understood that such changes and modifications also fall within the technical scope of the present disclosure.

[0126] Furthermore, the effects described in this specification are merely illustrative or exemplary and are therefore not limiting. That is, in addition to or in lieu of the effects described above, the technology according to this disclosure may have other effects that are apparent to those skilled in the art based on the description herein.

[0127] Note that the following configurations also fall within the technical scope of this disclosure. (1)

[0129] An information processing apparatus, comprising:

[0130] A rendering unit, based on three-dimensional data of the environment measured by a 3D scanner and camera parameters of a camera imaging device imaging the environment, generates a rendered image corresponding to a camera image captured by the camera device; and

[0131] A camera device parameter estimation unit extracts the difference between the camera device image and the rendered image, and updates the camera device parameters to make the difference smaller. (2)

[0133] According to the information processing device described in (1) above, wherein

[0134] The rendering unit and the camera device parameter estimation unit respectively repeat the generation of the rendered image and the updating of the camera device parameters until the difference is equal to or less than the threshold. (3)

[0136] According to the information processing apparatus described in (1) or (2) above, wherein

[0137] The camera device parameters include external parameters indicating the position of the camera device and internal parameters indicating the correspondence between the image of the camera device and the environment. (4)

[0139] The information processing apparatus according to any one of (1) to (3) above, wherein

[0140] The initial value of the extrinsic parameter is estimated using matching points extracted by feature point matching between the image from the camera device and the rendered image generated using provisional values ​​of the extrinsic parameter. (5)

[0142] The information processing apparatus according to any one of (1) to (4) above, wherein

[0143] The camera device is mounted on a device equipped with a sensor capable of detecting the position of an object, and

[0144] The provisional value of the external parameter is determined based on the position of the sensor. (6)

[0146] The information processing apparatus according to any one of (1) to (5) above, wherein

[0147] The device is a mobile object. (7)

[0149] The information processing apparatus according to any one of (1) to (6) above further includes

[0150] A sensor parameter estimation unit estimates sensor parameters, including the position of the sensor. (8)

[0152] The information processing apparatus according to any one of (1) to (7) above, wherein

[0153] The sensor parameter estimation unit estimates the sensor parameters by aligning the three-dimensional data with the point cloud data of the environment acquired by the sensor. (9)

[0155] The information processing apparatus according to any one of (1) to (8) above, wherein

[0156] The sensor parameter estimation unit performs the alignment in multiple stages. (10)

[0158] The information processing apparatus according to any one of (1) to (9) above, wherein

[0159] The environment is a static environment that is closed on at least three sides. (11)

[0161] The information processing apparatus according to any one of (1) to (10) above, wherein

[0162] A predetermined texture is projected onto the surface of the environment. (12)

[0164] The information processing apparatus according to any one of (1) to (11) above, wherein

[0165] The texture includes two-dimensional markers that can be individually identified. (13)

[0167] An information processing method executed by a computer, comprising:

[0168] Based on the three-dimensional data of the environment measured by a 3D scanner and the camera parameters of the camera device imaging the environment, a rendered image corresponding to the camera image captured by the camera device is generated; and

[0169] Extract the difference between the image from the camera device and the rendered image, and update the parameters of the camera device to make the difference smaller. (14)

[0171] A program that causes a computer to function as:

[0172] A rendering unit, based on three-dimensional data of the environment measured by a 3D scanner and camera parameters of a camera imaging device imaging the environment, generates a rendered image corresponding to a camera image captured by the camera device; and

[0173] A camera device parameter estimation unit extracts the difference between the camera device image and the rendered image, and updates the camera device parameters to make the difference smaller. (15)

[0175] An information processing system, comprising:

[0176] A rendering unit configured to generate a rendered image corresponding to a first device image captured by the first device, based on three-dimensional data of the environment and first parameters of a first device imaging the environment; and

[0177] A parameter estimation unit is configured to extract the difference between the first device image and the rendered image, and update the first parameters to make the difference smaller.

[0178] The rendering unit and the parameter estimation unit are each implemented via at least one processor. (16)

[0180] According to the information processing system described in (15), wherein

[0181] The rendering unit and the parameter estimation unit are further configured to repeat the generation of the rendered image and the update of the first parameter, respectively, until the difference is equal to or less than a threshold. (17)

[0183] According to the information processing system described in (15) or (16), wherein

[0184] The first parameter includes external parameters indicating the location of the first device and internal parameters indicating the correspondence between the image of the first device and the environment. (18)

[0186] The information processing system according to any one of (15) to (17), wherein

[0187] The initial value of the extrinsic parameter is estimated using matching points extracted by feature point matching between the first device image and the rendered image generated using provisional values ​​of the extrinsic parameter. (19)

[0189] The information processing system according to any one of (15) to (18), wherein

[0190] The first device is installed on a device equipped with a sensor capable of detecting the position of an object, and

[0191] The provisional value of the external parameter is determined based on the position of the sensor. (20)

[0193] The information processing system according to any one of (15) to (19), wherein

[0194] The device is a mobile object. (twenty one)

[0196] The information processing system according to any one of (15) to (20) further includes

[0197] A sensor parameter estimation unit, configured to estimate sensor parameters including the position of the sensor.

[0198] The sensor parameter estimation unit is implemented via at least one processor. (twenty two)

[0200] The information processing system according to any one of (15) to (21), wherein

[0201] The sensor parameter estimation unit is also configured to estimate the sensor parameters by aligning the three-dimensional data with point cloud data of the environment acquired by the sensor. (twenty three)

[0203] The information processing system according to any one of (15) to (22), wherein

[0204] The sensor parameter estimation unit is also configured to perform the alignment in multiple stages. (twenty four)

[0206] The information processing system according to any one of (15) to (23), wherein

[0207] The environment is a static environment that is closed on at least three sides. (25)

[0209] The information processing system according to any one of (15) to (24), wherein

[0210] The texture is projected onto the surface of the environment. (26)

[0212] The information processing system according to any one of (15) to (25), wherein

[0213] The texture includes markers that can be identified individually. (27)

[0215] An information processing method executed by a computer, comprising:

[0216] Based on the three-dimensional data of the environment and the first parameters of the first device for imaging the environment, a rendered image corresponding to the first device image captured by the first device is generated; and

[0217] Extract the difference between the first device image and the rendered image, and update the first parameter to make the difference smaller. (28)

[0219] A non-transitory computer-readable medium having a program thereon that, when executed by a computer, causes the computer to perform an information processing method, the method comprising:

[0220] Based on the three-dimensional data of the environment and the first parameters of the first device for imaging the environment, a rendered image corresponding to the first device image captured by the first device is generated; and

[0221] Extract the difference between the first device image and the rendered image, and update the first parameter to make the difference smaller. (29)

[0223] The information processing system according to any one of (15) to (25), wherein

[0224] The moving object performs autonomous driving by estimating the environment based on the three-dimensional data. (30)

[0226] The information processing system according to any one of (15) to (25) or (29), wherein

[0227] The sensor is a LiDAR, millimeter-wave radar, or ultrasonic sensor. (31)

[0229] The information processing system according to any one of (15) to (25), (29) or (30), wherein

[0230] The first device is an RGB camera, a monochrome camera, or a stereo camera. (32)

[0232] The information processing system according to any one of (15) to (25) or (29) to (31), wherein

[0233] The circuit system is also configured to estimate the intrinsic parameters included in the first parameter, and to exclude known intrinsic parameters included in the first parameter from the estimation. (33)

[0235] The information processing system according to any one of (15) to (25) or (29) to (32), wherein

[0236] The stability of estimating the intrinsic parameters is increased by excluding the known intrinsic parameters that are included in the first parameter. (34)

[0238] The information processing system according to any one of (15) to (25) or (29) to (33), wherein

[0239] The circuit system is also configured to repeat the generation of the rendered image and the updating of the first parameter until the number of times the rendered image is generated and the first parameter is updated reaches a predetermined number.

[0240] Those skilled in the art will understand that various modifications, combinations, sub-combinations, and alterations can be made according to design requirements and other factors, as long as such modifications, combinations, sub-combinations, and alterations are within the scope of the appended claims or their equivalents.

[0241] Reference Symbol List

[0242] 10 Moving Objects

[0243] 100 Information Processing Device

[0244] 110 Sensor Parameter Estimation Unit

[0245] 120 rendering units

[0246] 130 Camera Device Parameter Estimation Unit

[0247] 200 sensors

[0248] 300 camera devices

[0249] 400 3D Scanner

[0250] 500 projector

[0251] SE static environment

[0252] Tx texture

Claims

1. An information processing system, comprising: A rendering unit is configured to generate a rendered image corresponding to a first device image captured by the first device, based on three-dimensional data of the environment and first parameters of a first device imaging the environment. as well as A parameter estimation unit is configured to extract the difference between the first device image and the rendered image, and update the first parameters to make the difference smaller. Each of the rendering unit and the parameter estimation unit is implemented via at least one processor.

2. The information processing system according to claim 1, wherein, The rendering unit and the parameter estimation unit are further configured to repeat the generation of the rendered image and the update of the first parameter, respectively, until the difference is equal to or less than a threshold.

3. The information processing system according to claim 1, wherein, The first parameter includes external parameters indicating the location of the first device and internal parameters indicating the correspondence between the image of the first device and the environment.

4. The information processing system according to claim 3, wherein, The initial values ​​of the extrinsic parameters are estimated using matching points extracted by feature point matching between the first device image and the rendered image, and the rendered image is generated using provisional values ​​of the extrinsic parameters.

5. The information processing system according to claim 4, wherein, The first device is installed on a device equipped with a sensor capable of detecting the position of an object, and The provisional values ​​of the external parameters are determined based on the position of the sensor.

6. The information processing system according to claim 5, wherein, The device is a mobile object.

7. The information processing system according to claim 5, further comprising: A sensor parameter estimation unit, configured to estimate sensor parameters including the position of the sensor. in, The sensor parameter estimation unit is implemented via at least one processor.

8. The information processing system according to claim 7, wherein, The sensor parameter estimation unit is further configured to estimate the sensor parameters by aligning the three-dimensional data with point cloud data of the environment acquired by the sensor.

9. The information processing system according to claim 8, wherein, The sensor parameter estimation unit is also configured to perform the alignment in multiple stages.

10. The information processing system according to claim 1, wherein The environment is a static environment that is closed on at least three sides.

11. The information processing system according to claim 10, wherein, The texture is projected onto the surface of the environment.

12. The information processing system according to claim 11, wherein, The texture includes markers that can be individually identified.

13. An information processing method executed by a computer, comprising: Based on the three-dimensional data of the environment and the first parameters of the first device for imaging the environment, a rendered image corresponding to the first device image captured by the first device is generated. as well as Extract the difference between the first device image and the rendered image, and update the first parameter to make the difference smaller.

14. A non-transitory computer-readable medium having a program thereon, the program causing the computer to perform an information processing method when executed by a computer, the method comprising: Based on the three-dimensional data of the environment and the first parameters of the first device for imaging the environment, a rendered image corresponding to the first device image captured by the first device is generated. as well as Extract the difference between the first device image and the rendered image, and update the first parameter to make the difference smaller.

15. The information processing system according to claim 1, wherein, The moving object performs autonomous driving by estimating the environment based on the three-dimensional data.

16. The information processing system according to claim 5, wherein, The sensor is a LiDAR, millimeter-wave radar, or ultrasonic sensor.

17. The information processing system according to claim 1, wherein, The first device is an RGB camera, a monochrome camera, or a stereo camera.

18. The information processing system according to claim 1, wherein, The circuit system is also configured to estimate the intrinsic parameters included in the first parameter, and to exclude known intrinsic parameters included in the first parameter from the estimate.

19. The information processing system according to claim 18, wherein, The stability of estimating the intrinsic parameters is improved by excluding the known intrinsic parameters included in the first parameters.

20. The information processing system according to claim 1, wherein, The circuit system is also configured to repeat the generation of the rendered image and the updating of the first parameter until the number of times the rendered image is generated and the first parameter is updated reaches a predetermined number.

Images