Neural network parameter learning method, camera parameter calculation method, and program

Abstract

In the present invention, an information processing device acquires a learning image, acquires a real camera parameter relating to the learning image, calculates real two-dimensional coordinate points by projecting, to a predetermined plane, three-dimensional coordinate points on a unit sphere by using the real camera parameter, calculates estimated two-dimensional coordinate points by projecting, to the predetermined plane, the three-dimensional coordinate points by using an estimated camera parameter estimated by a neural network, and performs learning of a network parameter of the neural network on the basis of a distance between the real two-dimensional coordinate points and the estimated two-dimensional coordinate points.

Description

[Technical Field]

[0001] The present disclosure relates to a method for learning network parameters of a neural network, a method for calculating camera parameters, and a program. [Background technology]

[0002] Camera parameter calculation devices according to the background art are disclosed in the following Non-Patent Documents 1 and 2.

[0003] However, the background art disclosed in Non-Patent Document 1 does not allow for easy calculation of camera parameters. Also, the background art disclosed in Non-Patent Document 2 does not provide sufficient accuracy in calculating camera parameters. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] RY Tsai. “A versatile camera calibration technique for high accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. IEEE Journal of Robotics and Automation, Volume 3, Number 4, pages 323-344, 1987 [Non-patent document 2] M. Lopez, R. Mari, P. Gargallo, Y. Kuang, J. Gonzalez-Jimenez, and G. Haro. “Deep single image camera calibration with radial distortion”, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 11809-11817, 2019 Summary of the Invention

[0005] An object of the present disclosure is to provide a neural network parameter learning method, a camera parameter calculation method, and a program that are capable of easily and highly accurately calculating camera parameters. [Means for solving the problem]

[0006] A method for learning network parameters of a neural network according to one aspect of the present disclosure includes an information processing device acquiring a learning image, acquiring true camera parameters related to the learning image, calculating true two-dimensional coordinate points by projecting three-dimensional coordinate points on a unit sphere onto a predetermined plane using the true camera parameters, calculating estimated two-dimensional coordinate points by projecting the three-dimensional coordinate points onto the predetermined plane using estimated camera parameters estimated by the neural network, and learning network parameters of the neural network based on the distance between the true two-dimensional coordinate points and the estimated two-dimensional coordinate points. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a diagram showing a simplified configuration of a camera parameter calculation device according to a first embodiment of the present disclosure. [Figure 2] 10 is a flowchart showing the flow of processing executed by the camera parameter calculation device. [Figure 3] 1 is a flowchart showing the flow of a method for learning network parameters in a DNN. [Figure 4] 10 is a flowchart showing details of a loss calculation process. [Figure 5] 10 is a flowchart showing details of a loss calculation process. [Figure 6] FIG. 2 is a diagram for explaining the difference between the first embodiment of the present disclosure and the background art. [Figure 7] 10 is a flowchart showing details of a loss calculation process according to a second embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0008] (Findings that formed the basis of this disclosure) In order to calibrate a sensing camera or the like, geometry-based methods require associating three-dimensional coordinate values ​​in three-dimensional space with pixel positions in a two-dimensional image. To achieve this, a repeating pattern with a known shape is photographed, and the positions of intersections or the center positions of circles are detected, thereby associating three-dimensional coordinate values ​​with pixel positions in a two-dimensional image (Non-Patent Document 1).

[0009] Furthermore, a deep learning-based method has been proposed as a learning method that uses a single input image and is robust to image brightness, subject, etc. (Non-Patent Document 2).

[0010] However, the method of Non-Patent Document 1 requires photographing a repeating pattern of known shape, detecting the positions of intersections or the center positions of circles, and associating three-dimensional coordinate values ​​with pixel positions in a two-dimensional image, which are cumbersome tasks.

[0011] Furthermore, in the method of Non-Patent Document 2, lens distortion is expressed by a simple polynomial using one first parameter that infers lens distortion through deep learning and a second parameter that is calculated as a quadratic function of the first parameter. As a result, large lens distortion cannot be expressed appropriately, and the accuracy of calculating camera parameters is insufficient when applied to calibrating a camera with large lens distortion, such as a fisheye camera.

[0012] In order to solve such problems, the present inventor discovered that camera parameters can be calculated easily and with high accuracy by devising a method for projecting three-dimensional coordinate points on a unit sphere and two-dimensional coordinate points on a specified plane, and came up with the present disclosure.

[0013] Next, each aspect of the present disclosure will be described.

[0014] A method for learning network parameters of a neural network according to one aspect of the present disclosure includes an information processing device acquiring a learning image, acquiring true camera parameters related to the learning image, calculating true two-dimensional coordinate points by projecting three-dimensional coordinate points on a unit sphere onto a predetermined plane using the true camera parameters, calculating estimated two-dimensional coordinate points by projecting the three-dimensional coordinate points onto the predetermined plane using estimated camera parameters estimated by the neural network, and learning network parameters of the neural network based on the distance between the true two-dimensional coordinate points and the estimated two-dimensional coordinate points.

[0015] According to this aspect, the learning of the network parameters of the neural network can be performed easily and with high accuracy, and as a result, the camera parameters can be calculated easily and with high accuracy.

[0016] A method for learning network parameters of a neural network according to one aspect of the present disclosure includes an information processing device acquiring a learning image, acquiring true camera parameters related to the learning image, calculating true two-dimensional coordinate points by projecting three-dimensional coordinate points on a unit sphere onto a predetermined plane using the true camera parameters, calculating estimated three-dimensional coordinate points by projecting the true two-dimensional coordinate points onto the unit sphere using estimated camera parameters estimated by the neural network, and learning network parameters of the neural network based on the distance between the three-dimensional coordinate points and the estimated three-dimensional coordinate points.

[0017] According to this aspect, the learning of the network parameters of the neural network can be performed easily and with high accuracy, and as a result, the camera parameters can be calculated easily and with high accuracy.

[0018] In the above aspect, the three-dimensional coordinate points are each a plurality of three-dimensional coordinate points generated with a uniform distribution with respect to the incident angle of the camera.

[0019] According to this aspect, by using a plurality of three-dimensional coordinate points, it is possible to further improve the accuracy of learning the network parameters.

[0020] In the above aspect, the camera parameters include a plurality of parameters, and the estimated camera parameters are composite camera parameters in which one parameter of the plurality of parameters is an estimated parameter and another parameter of the plurality of parameters is a true parameter.

[0021] According to this aspect, by using composite parameters, it is possible to further improve the learning accuracy of network parameters.

[0022] In the above aspect, in learning the network parameters, the information processing device learns the network parameters so as to minimize the distance.

[0023] According to this aspect, by performing learning to minimize the distance between the true coordinate points and the estimated coordinate points, it is possible to further improve the learning accuracy of the network parameters.

[0024] In a method for calculating camera parameters according to one aspect of the present disclosure, an information processing device acquires a target image, calculates camera parameters of the target image based on a neural network in which network parameters have been learned, and the network parameters are learned by a method for learning network parameters of a neural network according to the above aspect, and outputs the camera parameters.

[0025] According to this aspect, the learning of the network parameters of the neural network can be performed easily and with high accuracy, and as a result, the camera parameters can be calculated easily and with high accuracy.

[0026] A program according to one aspect of the present disclosure is a program for causing an information processing device to function as an acquisition means and a calculation means, wherein the acquisition means acquires a training image and acquires true camera parameters related to the training image, the calculation means calculates true two-dimensional coordinate points by projecting three-dimensional coordinate points on a unit sphere onto a predetermined plane using the true camera parameters, calculates estimated two-dimensional coordinate points by projecting the three-dimensional coordinate points onto the predetermined plane using estimated camera parameters estimated by a neural network, and learns network parameters of the neural network based on the distance between the true two-dimensional coordinate points and the estimated two-dimensional coordinate points.

[0027] According to this aspect, the learning of the network parameters of the neural network can be performed easily and with high accuracy, and as a result, the camera parameters can be calculated easily and with high accuracy.

[0028] A program according to one aspect of the present disclosure is a program for causing an information processing device to function as an acquisition means and a calculation means, wherein the acquisition means acquires a training image and acquires true camera parameters related to the training image, the calculation means calculates true two-dimensional coordinate points by projecting three-dimensional coordinate points on a unit sphere onto a predetermined plane using the true camera parameters, calculates estimated three-dimensional coordinate points by projecting the true two-dimensional coordinate points onto the unit sphere using estimated camera parameters estimated by a neural network, and learns network parameters of the neural network based on the distance between the three-dimensional coordinate points and the estimated three-dimensional coordinate points.

[0029] According to this aspect, the learning of the network parameters of the neural network can be performed easily and with high accuracy, and as a result, the camera parameters can be calculated easily and with high accuracy.

[0030] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Elements with the same reference numerals in different drawings indicate the same or corresponding elements.

[0031] Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept are described as optional components. Furthermore, in all embodiments, the respective contents can be combined.

[0032] (First embodiment) 1 is a diagram showing a simplified configuration of a camera parameter calculation device 101 according to a first embodiment of the present disclosure. The camera parameter calculation device 101 is configured to include an input unit 102, a storage unit 103 such as a frame memory, a calculation unit 104 such as a CPU, and an output unit 105. The input unit 102, calculation unit 104, and output unit 105 can be realized as functions obtained by a processor such as a CPU executing a program read from a recording medium such as a CD-ROM to a ROM or RAM. Note that the input unit 102, calculation unit 104, and output unit 105 may also be configured using dedicated hardware.

[0033] 2 is a flowchart showing the flow of processing executed by the camera parameter calculation device 101. First, in step S201, the input unit 102 acquires image data of an image (target image) captured by a camera that is the target of camera parameter calibration from the camera or an arbitrary recording medium, etc. The input unit 102 stores the acquired image data in the storage unit 103.

[0034] Next, in step S202, the calculation unit 104 reads image data of the target image from the storage unit 103. The calculation unit 104 inputs the image data of the target image into a trained deep neural network (DNN) to calculate camera parameters of the target image. Details of a method for training network parameters in the DNN will be described later.

[0035] Next, in step S203, the output unit 105 outputs the camera parameters calculated by the calculation unit 104.

[0036] 3 is a flowchart showing the flow of a method for learning network parameters in a DNN. First, in step S301, the calculation unit 104 inputs image data of learning images to be used in learning the DNN. The learning images are images captured in advance using a fisheye camera or the like. However, the learning images may also be generated by computer graphics (CG) processing from panoramic images using a fisheye camera model.

[0037] Next, in step S302, the calculation unit 104 inputs the true camera parameters Ω hat. The true camera parameters Ω hat are camera parameters related to the camera that captured the learning images. However, if the learning images are generated by CG processing, the true camera parameters Ω hat are the camera parameters used in the CG processing. The camera parameters include external parameters that are parameters related to the camera's attitude (rotation and translation relative to the world coordinate base) and internal parameters that are parameters related to the focal length, lens distortion, etc.

[0038] Next, in step S303, calculation unit 104 estimates (infers) camera parameters Ω by inputting the learning image into the DNN. The DNN extracts image features from a convolution layer or the like, and finally outputs each estimated camera parameter. For example, it outputs three estimated camera parameters Ω: the tilt angle θ, roll angle ψ, and focal length f of the camera. For simplicity of explanation, an example of estimating the above three camera parameters (θ, ψ, f) will be described below.

[0039] Next, in step S304, the calculation unit 104 calculates the loss L, which is the error of the estimation result of the DNN, for learning the network parameters of the DNN. total The process of step S304 will be described in detail later.

[0040] Next, in step S305, the calculation unit 104 updates the network parameters of the DNN by backpropagation. As an optimization algorithm in backpropagation, for example, stochastic gradient descent can be used.

[0041] Next, in step S306, the calculation unit 104 determines whether the learning of the DNN is completed. If the number of updates of the network parameters of the DNN exceeds a threshold (for example, 10,000 times), or if the loss L calculated in step S304 total When the value of becomes less than a threshold value (for example, 3 pixels), it is determined that learning is complete.

[0042] If the learning is complete (step S306: YES), the process ends. If the learning is not complete (step S306: NO), the process from step S301 onwards is repeated.

[0043] FIG. 4 shows the loss L in step S304. total 4 is a flowchart showing details of the calculation process of Ω hat. First, in step S401, the calculation unit 104 inputs the true camera parameters Ω hat acquired in step S302.

[0044] Next, in step S402, the calculation unit 104 receives the estimated camera parameter Ω estimated in step S303.

[0045] Next, in step S403, the calculation unit 104 calculates the loss L according to the following formula (1): total Calculate.

[0046]

number

[0047] w θ , w ψ , w f are the weights for the tilt angle, roll angle, and focal length, respectively. For example, the weight w θ , w ψ , w f However, if you want to differentiate the importance of each camera parameter, you can use the weight w θ , w ψ , w f may be different values. θ , L ψ , L f are the loss L for the tilt angle, roll angle, and focal length, respectively.

[0048] Next, in step S404, the calculation unit 104 calculates the loss L calculated in step S403. total Output.

[0049] FIG. 5 shows the loss L in step S403. total 1 is a flowchart showing the details of the calculation process of . First, in step S501, the calculation unit 104 inputs the true camera parameter Ω hat and the estimated camera parameter Ω. The estimated camera parameter Ω is generated as a composite camera parameter in which only one of the multiple parameters θ, ψ, and f is replaced with an estimated parameter and the remaining two parameters are used as true parameters. For example, when replacing only the tilt angle θ with an estimated parameter, an estimated parameter by DNN is used for the tilt angle θ, and true parameters are used for the roll angle ψ and the focal length f. As a result, the loss L, which is an error related to the tilt angle θ, is calculated. θ is expressed.

[0050] Next, in step S502, the calculation unit 104 defines a spherical surface of a unit circle with the camera position as the origin, and cuts out a hemispherical surface S with an incident angle of 90° or less. Note that in the case of a fisheye camera model (for example, stereographic projection) that can handle incident angles of 90° or more, the incident angle may be set to 90° or more. The calculation unit 104 defines three-dimensional coordinate points P uniformly distributed on the hemispherical surface S. w Generate N hats. This uniform distribution can be generated by applying a uniform random number to each of the two angles in the three-dimensional polar coordinate representation (radius, angle 1, angle 2). The value of N is, for example, 10,000.

[0051] Next, in step S503, the calculation unit 104 calculates the true three-dimensional coordinate point P w By projecting the hat onto a predetermined image plane (hereinafter referred to as the "predetermined plane"), a true two-dimensional coordinate point P i The camera parameters are parameters for projecting from world coordinates to image coordinates. In the case of stereoscopic projection, which is an example of a fisheye camera model, this projection is expressed by the following equations (2) to (5).

[0052]

number

[0053]

number

[0054]

number

[0055]

number

[0056] Here, (X, Y, Z) are the true three-dimensional coordinates of the point P w The hat's world coordinates, (x, y) are the true two-dimensional coordinates of the point P iare the image coordinate values ​​of the hat. f is the focal length of the camera, and (C x ,C y ) is the camera's principal point image coordinate. 11 ~r 33 are the elements of a 3x3 rotation matrix that represents the rotation relative to the world coordinate system, and T X ,T Y ,T Z represents a translation relative to the world coordinate reference.

[0057] Next, in step S504, the calculation unit 104 calculates the true three-dimensional coordinate point P w By projecting the hat onto a predetermined plane, the estimated two-dimensional coordinate point P i Calculate.

[0058] Next, in step S505, the calculation unit 104 calculates the true two-dimensional coordinate point P i Hat and estimated 2D coordinate point P i The loss L is calculated based on the error between the true two-dimensional coordinate point P i Hat and estimated 2D coordinate point P i It can be defined as the square of the Euclidean distance between , and the average of N points generated by uniform distribution is calculated as shown in the following equation (6).

[0059]

number

[0060] The error function for calculating the loss L is not limited to the example of equation (6), and the Huber loss shown in equation (7) below may also be used.

[0061]

number

[0062] Next, in step S506, the calculation unit 104 outputs the loss L calculated in step S505.

[0063] FIG. 6 is a diagram illustrating the difference between this embodiment and the above-mentioned Non-Patent Document 2. Non-Patent Document 2, like this embodiment, uses a DNN to estimate camera parameters, and deep learning is performed using the loss described in the document (called Bearing Loss in the document). Unlike the loss L used in this embodiment, Bearing Loss selects pixel values ​​for all pixels in an image (grid points on the image), projects each grid point onto a unit sphere of world coordinates using camera parameters, and defines the distance on the unit sphere as the error. As shown in FIG. 6, the grid points on image 200 in Non-Patent Document 2 are not uniform in distance (image height) from principal point 300, nor are they uniform in angle of incidence (the angle of incidence depends on the image height). For example, grid point 301 is located on circle C1 at a first distance close to principal point 300, and grid point 302 is located on circle C2 at a second distance far from principal point 300. Therefore, when grid points are selected from a rectangular image, the selected pixels are non-uniform, as seen in grid point 303, where part of circle C2, which is far from the principal point, extends outside image 200 and does not exist on image 200. Furthermore, when the image height is large (corresponding to large circle C2 in FIG. 6), the number of selected grid points increases (increases in proportion to the square of the image height) compared to when the image height is small (corresponding to small circle C1 in FIG. 6). Also, in a camera model symmetric about the optical axis, there is a one-to-one correspondence between image height and incident angle, and when the image height is large, the incident angle increases. In other words, when the incident angle is large, biased sampling occurs. As described above, bearing loss based on points using an image grid results in non-uniform sampling and is not suitable for fisheye camera models with significant lens distortion.

[0064] On the other hand, the loss L according to this embodiment uses points that are uniformly distributed with respect to the angle of incidence as the projection source, and projects the points onto image coordinates using the camera parameters to calculate the error. Therefore, it is suitable for learning the camera parameters not only of normal cameras with small lens distortion, but also of fisheye cameras with large lens distortion.

[0065] (Second embodiment) The second embodiment of the present disclosure will be described below, focusing on the differences from the first embodiment.

[0066] FIG. 7 corresponds to FIG. 5 and shows the loss L total 5 is a flowchart showing details of the calculation process of Ω hat and Ω estimated camera parameters ... hat and Ω estimated camera parameters Ω hat and Ω estimated camera parameters Ω hat and

[0067] Next, in step S502, the calculation unit 104 defines a spherical surface of a unit circle with the camera position as the origin, cuts out a hemisphere S with an incident angle of 90° or less, and calculates uniformly distributed three-dimensional coordinate points P w Generate N hats.

[0068] Next, in step S503, the calculation unit 104 calculates the true three-dimensional coordinate point P w By projecting the hat onto a given plane, the true two-dimensional coordinate point P i Calculate the hat.

[0069] Next, in step S704, the calculation unit 104 calculates the true two-dimensional coordinate point P i By projecting the hat onto the hemisphere S, the estimated 3D coordinate point P w The above equations (2) to (5) are used to calculate the three-dimensional coordinate point P w the two-dimensional coordinate point P i is not only a mathematical formula for projecting the two-dimensional coordinate point P in the image coordinates using the camera parameters Ω. i is the three-dimensional coordinate point P in world coordinates. w Since image coordinates are two-dimensional and world coordinates are three-dimensional, the two-dimensional coordinate point P i the three-dimensional coordinate point P w When projecting onto a surface, unique world coordinates can be obtained by limiting the world coordinates to those on the unit sphere (hemisphere S).

[0070] Next, in step S705, the calculation unit 104 calculates the true three-dimensional coordinate point P w Hat and estimated 3D coordinate point P w The loss L is calculated based on the error between the true three-dimensional coordinate point Pw Hat and estimated 3D coordinate point P w It can be defined as the square of the Euclidean distance between , and the average of N points generated by uniform distribution is calculated as shown in the following equation (8).

[0071]

number

[0072] The error function for calculating the loss L is not limited to the example of equation (8), and the Huber loss shown in equation (9) below may also be used.

[0073]

number

[0074] Next, in step S506, the calculation unit 104 outputs the loss L calculated in step S705.

[0075] As with the first embodiment, this embodiment also allows for simple and highly accurate learning of neural network parameters, thereby enabling simple and highly accurate calculation of camera parameters. Since the maximum error value is kept within the diameter (=1) of a unit sphere in this embodiment, learning is less likely to fail than in the first embodiment in the early stages of learning when the network parameters are not yet determined. Meanwhile, according to the first embodiment, learning is performed to minimize errors in two-dimensional images, so the effect of removing image distortion by calibrating the camera parameters is greater than in this embodiment. [Industrial Applicability]

[0076] The present disclosure is particularly useful when applied to a camera parameter calculation device for a camera with large lens distortion, such as a fisheye camera.

Claims

1. The information processing device Obtain training images, obtaining true camera parameters for the training images; Calculating true two-dimensional coordinate points by projecting three-dimensional coordinate points on the unit sphere onto a predetermined plane using the true camera parameters; calculating estimated two-dimensional coordinate points by projecting the three-dimensional coordinate points onto the predetermined plane using estimated camera parameters estimated by a neural network; A method for learning network parameters of a neural network, which learns network parameters of the neural network based on the distance between the true two-dimensional coordinate point and the estimated two-dimensional coordinate point.

2. The information processing device Obtain training images, obtaining true camera parameters for the training images; Calculating true two-dimensional coordinate points by projecting three-dimensional coordinate points on the unit sphere onto a predetermined plane using the true camera parameters; calculating estimated three-dimensional coordinate points by projecting the true two-dimensional coordinate points onto the unit sphere using estimated camera parameters estimated by a neural network; A method for learning network parameters of a neural network, the method learning network parameters of the neural network based on the distance between the three-dimensional coordinate point and the estimated three-dimensional coordinate point.

3. 3. The method for learning network parameters of a neural network according to claim 1, wherein the three-dimensional coordinate points are each of a plurality of three-dimensional coordinate points generated with a uniform distribution with respect to the incident angle of the camera.

4. the camera parameters include a plurality of parameters; 4. The method for learning network parameters of a neural network according to claim 1, wherein the estimated camera parameters are composite camera parameters in which one parameter of the plurality of parameters is an estimated parameter and the other parameters of the plurality of parameters are true parameters.

5. 5. The neural network parameter learning method according to claim 1, wherein in learning the network parameters, the information processing device learns the network parameters so as to minimize the distance.

6. The information processing device Acquire the target image, Calculating camera parameters of the target image based on the neural network whose network parameters have been trained; The network parameters are learned by the neural network parameter learning method according to any one of claims 1 to 5, A camera parameter calculation method that outputs the camera parameters.

7. An information processing device Acquisition means; A calculation means; A program for functioning as The acquisition means Obtain training images, obtaining true camera parameters for the training images; The calculation means Calculating true two-dimensional coordinate points by projecting three-dimensional coordinate points on the unit sphere onto a predetermined plane using the true camera parameters; calculating estimated two-dimensional coordinate points by projecting the three-dimensional coordinate points onto the predetermined plane using estimated camera parameters estimated by a neural network; A program that learns network parameters of the neural network based on the distance between the true two-dimensional coordinate points and the estimated two-dimensional coordinate points.

8. An information processing device Acquisition means; A calculation means; A program for functioning as The acquisition means Obtain training images, obtaining true camera parameters for the training images; The calculation means Calculating true two-dimensional coordinate points by projecting three-dimensional coordinate points on the unit sphere onto a predetermined plane using the true camera parameters; calculating estimated three-dimensional coordinate points by projecting the true two-dimensional coordinate points onto the unit sphere using estimated camera parameters estimated by a neural network; A program that learns network parameters of the neural network based on the distance between the three-dimensional coordinate point and the estimated three-dimensional coordinate point.

Images