Method and system for performing self-calibration

Abstract

This invention provides a method for self-calibration using joint optimization of 2D and 3D differential data. [Solution] The present invention provides a method for performing self-calibration by optimizing imaging parameters using both the 2D difference data and the 3D difference data as variables, so as to reduce the difference between the target value and the current value of a function whose variables include data comprising 2D difference data, which is data composed of the difference components of at least two images contained in a video, and 3D difference data, which is data composed of the difference components of information obtained by adding a depth component to at least two images contained in the video.

Description

Technical Field

[0001] The present invention relates to a method for performing self - calibration and a system for performing self - calibration.

Background Art

[0002] Patent Document 1 discloses a method for self - calibration of at least one camera.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, as will be described later, during monotonous camera movement, the gradient for optimization becomes zero, and optimization based on the gradient descent method cannot be performed and learning cannot be carried out. Also, in order to handle steep camera movements, acquisition of a pose estimator (PoseNet) by supervised learning is required, so "acquiring videos assuming the usage environment in advance" is necessary. Since it is non - convex optimization, convergence to a global solution is not guaranteed unless the initial value of the optimization is appropriate. On the other hand, the PoseNet required by the solution of Kanai et al. must share the scale with the depth estimator used in combination [Reference 1 below]. As a result, supervised learning using in - domain data [Reference 2 below] for matching the scales of the depth estimator and PoseNet has been required.

[0005] Therefore, one object of the present disclosure is to provide a method for performing self - calibration using joint optimization of 2D difference data and 3D difference data.

Means for Solving the Problems

[0006] The method for performing self-calibration in this disclosure is: This method involves optimizing imaging parameters using both the 2D difference data and the 3D difference data as variables to minimize the difference between the target value and the current value of a function whose variables include data comprising: 2D difference data, which is data composed of the difference components of at least two images contained in a video; and 3D difference data, which is data composed of the difference components of information obtained by adding a depth component to at least two images contained in the video.

[0007] The above configuration provides a method for performing self-calibration using joint optimization of 2D difference data and 3D difference data.

[0008] The system that performs self-calibration in this disclosure is An imaging device for capturing video, The system performs self-calibration and includes an optimization unit that optimizes imaging parameters using both the 2D difference data and the 3D difference data as variables to minimize the difference between the target value and the current value of a function whose variables include data comprising: 2D difference data, which is data composed of the difference components of at least two images included in the video; and 3D difference data, which is data composed of the difference components of information obtained by adding a depth component to at least two images included in the video.

[0009] The above configuration provides a system that performs self-calibration using joint optimization of 2D differential data and 3D differential data. [Effects of the Invention]

[0010] This disclosure provides a method for performing self-calibration using joint optimization of 2D difference data and 3D difference data. [Brief explanation of the drawing]

[0011] [Figure 1]This is a block diagram showing the configuration of a system that performs self-calibration according to an embodiment. [Figure 2] This is a flowchart of the method for performing self-calibration according to the embodiment. [Figure 3] This is a schematic diagram showing a detailed flow of the method for performing self-calibration according to the embodiment. [Modes for carrying out the invention]

[0012] Embodiment Embodiments of the present invention will be described below with reference to the drawings. However, the invention claimed is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential for solving the problem. For clarity of explanation, the following descriptions and drawings have been omitted and simplified as appropriate. In each drawing, the same elements are denoted by the same reference numerals, and redundant explanations have been omitted where necessary.

[0013] (Description of a system for performing self-calibration according to an embodiment) Figure 1 is a block diagram showing the configuration of a self-calibration system according to the embodiment. The self-calibration system according to the embodiment will be described with reference to Figure 1. Self-calibration is the automatic calibration of the focus, field of view, magnification, etc., when a camera that images the surroundings moves.

[0014] As shown in Figure 1, the self-calibration system 100 comprises an imaging device 101 and an optimization unit 102.

[0015] The imaging device 101 is a camera that captures video. The imaging device 101 can be any type of camera, such as an RGB camera, an RGBD camera, an infrared camera, or a fisheye camera. The imaging device 101 can acquire two consecutive images.

[0016] The optimization unit 102 acquires 2D difference data and 3D difference data from two images captured by the imaging device, and optimizes the imaging parameters based on the 2D difference data and 3D difference data. The 2D difference data is data composed of the difference components of at least two images included in the video. The 3D difference data is data composed of the difference components of information obtained by adding the depth component to at least two images included in the video.

[0017] The optimization unit 102 optimizes the imaging parameters using both the 2D difference data and the 3D difference data as variables, so as to reduce the difference between the target value and the current value of a function that includes the 2D difference data and the 3D difference data as variables.

[0018] If imaging parameters can be acquired, self-calibration of a single camera, including focus, field of view, and magnification, can be performed.

[0019] 2D difference data and 3D difference data can be optimized separately or combined. For example, imaging parameters can be adjusted to minimize the difference in 2D difference data, and then adjusted to minimize the difference in 3D difference data. Alternatively, imaging parameters can be adjusted to minimize the difference between 2D and 3D difference data by performing arithmetic operations such as adding, subtracting, multiplying, and dividing.

[0020] The optimization unit 102 is executed by an information processing device. The information processing device comprises a processor that executes and processes a program, and a memory that stores the program. The information processing device may consist of one device or multiple devices. The information processing device may also be a cloud server that processes some or all of its functions in a distributed manner.

[0021] Some or all of the processing in an information processing device can be implemented as a computer program. Such programs can be stored and supplied to a computer using various types of non-temporary computer-readable media. Non-temporary computer-readable media include various types of tangible recording media. Examples of non-temporary computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). Programs may also be supplied to a computer using various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable media can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0022] The above configuration provides a system that performs self-calibration using joint optimization of 2D differential data and 3D differential data.

[0023] (Description of the method for performing self-calibration according to the embodiment) Figure 2 is a flowchart of the self-calibration method according to the embodiment. Figure 3 is a block diagram showing a detailed flowchart of the self-calibration method according to the embodiment. The self-calibration method according to the embodiment will be explained with reference to Figures 2 and 3.

[0024] As shown in Figures 2 and 3, trained Artificial Neural Networks (ANNs) 302 are prepared (step S201). The trained ANNs 302 estimate depth from at least one image. The trained ANNs 302 also estimate optical flow from at least two images. The trained ANNs 302 consists of two inference models. The trained ANNs 302 is further equipped with an optical flow separator 303.

[0025] The optical flow separator 303 separates the optical flow into camera attitude and camera intrinsic parameters.

[0026] Next, a video is recorded while the imaging device 101 is moved (step S202). At this time, at least two images are captured, as shown in Figure 3.

[0027] Finally, the self-calibration algorithm is executed (step S203). In this way, self-calibration is performed.

[0028] Figure 3 shows the details of the self-calibration algorithm. As shown in Figure 3, the two images captured by the imaging device 101 are input to the provisional target flow estimator 304 and ANNs 302.

[0029] S-1 (Acquisition of machine learning models) The provisional target flow estimator 304 and ANNs302 use existing OSS (operating systems), such as the one proposed by Hagemann et al. The provisional target flow estimator 304 is a model that estimates the update amount and its weights to obtain a more accurate optical flow from image feature vectors and optical flow.

[0030] S-2 (Estimation of initial values ​​for the optimization variables) ANNs302 uses at least one (inverse) depth image z of the paired images. 0 and optical flow f ijEstimate it. That is, the machine-learned information processing device inputs at least one image and outputs the depth, and inputs at least two images including one image and outputs the optical flow. The optical flow separator 303 separates the optical flow f ijを into the camera pose G to be optimized 0 and the camera internal parameter θ 0 . That is, the optical flow separator separates the camera pose and the camera internal parameter from the optical flow. For the optical flow separator 303, for example, the application of Procrustes Analysis (Eqn. 1) that does not require linearity in the internal parameter representation can be considered. [Number] [Number]

[0031] Here, G ij is the relative camera movement from the image I i to I j , x i is the projection of the observation point group of I i onto the world coordinate system (Eqn. 2). W 1 / 2 is the weight for the optimization coordinates that can be used arbitrarily. STN b (x j , u ij ) is an operation that applies bilinear interpolation to obtain the coordinate value from the coordinates (x ij j tl , x j tr , ···) of the four nearest neighbor points (top-left, that is, tl, bottom-right, that is, br, etc.) on the image grid based on the optical flow estimate u ij

[0032] Note that π -1 (·) is the reprojection function of the point group, and the grid point u i on the image coordinates is expressed as follows using the depth estimate 1 / z iIt is projected onto three-dimensional space via [a specific method / framework].

number

[0033] u in equation (1) ij This directly uses the estimation results of ANNs to u i +f ij Alternatively, the set of optimization variables {G} may be separated once. 0 ,z 0 ,θ 0 It may be reconstructed from} (Eqn. 2).

[0034] Unlike the method of Hagemann et al. in Non-Patent Document 1, the initial estimate given by ANNs {G 0 ,z 0 ,θ 0 This reduces the sensitivity to initial values, which is a challenge in gradient descent-based methods. In this way, initial values ​​are obtained for the optimization calculation.

[0035] S-3 (Provisional optical flow estimation) The group of variables to be optimized {G 0 ,z 0 ,θ 0 Based on} and image information, the provisional target value u of optical flow ij * =u ij +Δu ij The amount of update required to obtain Δu ij And, update weight lol ij This is estimated using the provisional target flow estimator 304 (Eqn. 3). For example, the same mechanism as DROID-SLAM, namely the use of UpdateModule, can be considered [see reference 4 below].

number

[0036] S-4 (joint optimization) The optimization unit 102 performs nonlinear optimization that makes it difficult to make the gradient zero during the camera's linear motion. For example, the objective function L used by Hagemann et al.Opt.1 In addition to (Eqn. 4), there is the objective function L proposed by Smith et al. Opt.2 The combined use of (Eqn. 5) is a possibility. The former is an optical flow, which is a residual on a two-dimensional representation, while the latter, which is proposed to be used in combination, is a scene flow, that is, an objective function based on a residual on a three-dimensional representation.

number

number

number

[0037] Here, π(·) is the projection function onto the image plane, and (i,j) and (i,j) are sets of keyframe pairs, ||·|| Σ This is the Mahalanobis environment, Σ ij =diagw ij It is composed of.

[0038]

number

number

[0039] Unlike S-2, u ij This directly uses the estimation results of ANNs to u i +f ij This is used reliably. This ensures that the differential term that may cause the gradient to vanish is, i.e., du ij Avoid generating / dθ during the first derivative process. Also, L Opt.2 Optimization of W 1 / 2 lol ij This proposal also includes forms that use w for assistance. In particular, ij to W 1 / 2This approach, applying the formula to the equation (5), is considered to play an important role in more reliably performing the optimization of equation (5), which is assumed to be susceptible to the adverse effects of estimation noise.

[0040] Unlike the method of Hagemann et al. in Non-Patent Document 1, the optimization described by equation (5) is expected to mitigate learning failures during monotonic camera movement.

[0041] S-5({G K ,z K ,θ K} Acquisition 308) The set of variables {G} that underwent the optimization described in equations (4) and (5) a specified number of times. 1 ,z 1 ,θ 1 We obtain {G} (306). If the error is small enough (Yes in 306), the estimated value {G} K ,z K ,θ K Obtain {G} (308). If the error is not small enough (No. 306), increment the count by 1 (307) and set {G} to the provisional target flow estimator 304. 1 ,z 1 ,θ 1 Input}. That is, repeat S-2 to S-4 across multiple keyframes until the objective function error is minimized, and the final estimated value {G K ,z K ,θ K This yields}. Following Teed et al.'s implementation, this is repeated at each time step within a defined keyframe pair [see reference 4 below].

[0042] (Other embodiments) In S-1, to improve the performance of the provisional target flow estimator 304, it is conceivable to generate training data that simulates various camera parameters, such as fisheye lenses, and to apply this to the training process. In other words, data captured by one camera can be applied to data captured by another camera.

[0043] In S-1, if video data can be acquired in the expected operating environment, ANNs can be acquired separately through self-supervised learning. For example, the method by Fang et al. can be considered [see reference 5 below].

[0044] In S-2, if information that does not require estimation, such as depth or self-position, can be obtained, those sensor values ​​may be used instead of ANNs. For example, LiDAR, GPS, and RADAR can be used.

[0045] In S-4, the optimization function {L Opt.1 ,L Opt.2 There are various ways to use and express the}. For example, you can perform 2D difference data optimization and 3D difference data optimization separately. Alternatively, you can combine 2D difference data optimization and 3D difference data optimization using arithmetic operations.

[0046] This disclosure enables rapid automatic camera recalibration. In the event of a malfunction in an (autonomous) driving vehicle equipped with at least one camera, after safety has been confirmed, the minimum camera parameters can be restored simply by moving the vehicle forward or backward.

[0047] This disclosure enables the scaling of training data for robots. Even when training data for making robots intelligent requires 3D reconstruction from video as a preprocessing step, it is possible to collect and integrate the data without worrying about individual differences between cameras.

[0048] (Proof of vanishing gradient) The objective function L used by Hagemann et al. Opt.1 If we assume that (1) it is a pinhole camera and (2) the camera movement is monotonous, the residual r ij We show that the Jacobian matrix for θ is zero. The Jacobian matrix is ​​given by equation (6) at this time.

number

number

[0049] Here, the flow x of the 3D point cloud ij The slope of with respect to θ is as follows:

number

[0050] In addition to equation (6), the camera model, i.e., θ = [f x f y c x c y ] T Assuming (Eqn. 8), each term has the following structure.

number

number

number

[0051] If the camera's movement is monotonous, that is, if it consists only of "translational motion in the z-axis direction that does not involve rotation," then the following approximation holds:

number

[0052] From equations (8) and (9), the following approximation holds.

number

[0053] Therefore, substituting equation (10) into equation (6), we get dr ij / dθ=0 can be found immediately. This makes it difficult to find θ using nonlinear optimization based on gradient descent. Also, equation (6)

number

[0054] On the other hand, Smith et al.'s method (Eqn. 5) assumes that the optical flow estimation is constant (it uses the output of the trained model as is), so du ij It does not explicitly have a component of / dθ. In this case, the deviation e with respect to θ ij The gradient is the coefficient γ of the constant given by the optical flow. k Using this, it is given as in equation (11).

number

number

number

number

[0055] For simplification, f x Focusing only on the component of and applying the approximation of equation (9), we obtain the following approximate formula.

number

number

[0056] Thus, even if the camera movement is monotonous, L Opt.2 The gradient can be derived, and therefore, gradient descent-based optimization can be facilitated.

[0057] For detailed formula manipulations, please refer to the supplementary materials of Hagemann et al.'s paper [see reference 6 below].

[0058] It should be noted that the present invention is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention.

[0059] References [Reference 1] T. Kanai et al. “Self-supervised geometry-guided initialization for robust monocular visual odometry,” arXiu, 2024. [Reference 2] T. Zou et al. “Unsupervised learning of depth and ego-motion from video,” in CVPR, 2017. [Reference 3] M. Jaderberg et al. “Spatial transformer networks,” in Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, Eds., vol. 28. Curran Associates, Inc., 2015. [Reference 4] Z. Teed et al. “DROID-SLAM: Deep Visual SLAM for monocular, Stereo, and RGB-D Cameras,” Advances in neural information processing systems, 2021. [Reference 5] J. Fang et al. “Self-supervised camera self-calibration from video,” in ICRA, 2022, pp.8468-8475. [Reference 6] P. Hagemann et al. “Deep geometru-aware camera pose learning (supplemental),” 2023, accessed: 2023-10-05.[Online]. Available: https: / / openaccess.thecvf.com / comtent / ICCV2023 / supplemental / Hagemann_Deep_Geometry_Aware_Camera_ICCV_2023_supplemental.pdf [Explanation of symbols]

[0060] 100 System, 101 Imaging device, 102 Optimization unit, 302 ANNs, 303 Optical flow separator, 304 Provisional target flow estimator

Claims

1. A method for performing self-calibration by optimizing imaging parameters using both the 2D difference data and the 3D difference data as variables, so as to reduce the difference between the target value and the current value of a function whose variables include data comprising: 2D difference data, which is data composed of the difference components of at least two images contained in a video; and 3D difference data, which is data composed of the difference components of information obtained by adding a depth component to at least two images contained in the video.

2. A machine learning-based information processing device takes at least one image as input and outputs depth, and takes at least two images including the one image as input and outputs optical flow, A method for performing self-calibration according to claim 1, wherein the optical flow separator separates the optical flow into camera attitude and camera internal parameters to obtain initial values ​​for the optimization calculation.

3. A method for performing self-calibration according to claim 1, wherein the optimization of the 2D difference data and the optimization of the 3D difference data are performed separately.

4. A method for performing self-calibration according to claim 1, comprising combining the optimization of the 2D difference data and the optimization of the 3D difference data by performing arithmetic operations.

5. An imaging device for capturing video, A self-calibration system comprising: an optimization unit that optimizes imaging parameters using both the 2D difference data and the 3D difference data as variables to reduce the difference between the target value and the current value of a function whose variables include data comprising: 2D difference data which is data composed of the difference components of at least two images included in the video; and 3D difference data which is data composed of the difference components of information obtained by adding a depth component to at least two images included in the video.

Images