Learning device, learning method, learning program, camera parameter calculation device, camera parameter calculation method, and camera parameter calculation program

Abstract

By inputting an image to a deep neural network, a training unit of this training device: estimates distortion parameters which represent the distortion of a lens which assumes optical axis symmetry by means of a projection function that is a 4th order function or lower pertaining to an incident angle to a camera, and camera parameters including orientation parameters that indicate the orientation of the camera; calculates an inverse function of the projection function in a closed form; calculates a plurality of true values obtained by projecting a plurality of reference points on the image into world coordinates by using the inverse function and true camera parameters; calculates a plurality of estimation values obtained by projecting the plurality of reference points into the world coordinates by using the inverse function and the estimated camera parameters; calculates, on the basis of each combination of each of the plurality of true values and each of the plurality of estimation values, network errors that indicate errors between the true camera parameters and the estimation results of the camera parameters from the deep neural network; and learns with the parameters of the deep neural network so as to minimize the network errors.

Description

[Technical Field]

[0001] This disclosure relates to a technique for training a deep neural network to calculate camera parameters from images, and a technique for calculating camera parameters from images. [Background technology]

[0002] To calibrate cameras such as sensing cameras, geometry-based methods require associating 3D coordinate values ​​in 3D space with pixel positions in a 2D image. Conventionally, a repeating pattern with a known shape is captured, and the correspondence between 3D coordinates and pixel positions in the 2D image is achieved by detecting intersections or the centers of circles from the obtained image.

[0003] Furthermore, deep learning-based methods have been proposed to perform robust camera calibration using a single input image, either for image brightness or for adjusting the subject. Camera calibration, in this context, refers to the calculation of camera parameters.

[0004] For example, in Non-Patent Document 1, camera parameters are calculated using a geometry-based method that associates 3D coordinate values ​​in 3D space with pixel positions in a 2D image using calibration indices.

[0005] Furthermore, for example, Non-Patent Document 2 discloses a deep learning-based method for performing camera calibration from a single image.

[0006] The method described in Non-Patent Document 1 requires the process of photographing a repeating pattern with a known shape, detecting intersections or the center of a circle from the obtained image, and mapping 3D coordinates to pixel positions in the 2D image. Therefore, the camera calibration process is complex and may not be easily performed.

[0007] Furthermore, in the method described in Non-Patent Document 2, lens distortion is expressed by a simple polynomial using a distortion parameter k1 and a distortion parameter k2 calculated as a quadratic function of the distortion parameter k1. Therefore, the method described in Non-Patent Document 2 has difficulty appropriately representing large lens distortions such as those found in fisheye cameras, and is not suitable for camera calibration of cameras with large lens distortions like fisheye cameras. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] RYTsai, “A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses,” IEEE Journal of Robotics and Automation, Volume 3, Number 4, pages 323-344, August 1987. [Non-Patent Document 2] M.Lopez, R.Mari, P.Gargallo, Y.Kuang, J.Gonzalez-Jimenez, and G.Haro, "Deep Single Image Camera Calibration with Radial Distortion", 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR), pages 11817-11825, June 2019. [Overview of the project]

[0009] This disclosure was made to solve the above-mentioned problems and aims to provide a technology that can calculate camera parameters with high accuracy from a single image.

[0010] The learning device according to this disclosure comprises: an image acquisition unit that acquires images captured by a camera; a camera parameter acquisition unit that acquires true camera parameters of the camera; a learning unit that learns a deep neural network by deep learning using the images acquired by the image acquisition unit and the true camera parameters acquired by the camera parameter acquisition unit; and an output unit that outputs the deep neural network learned by the learning unit. The learning unit inputs the images into the deep neural network and includes a distortion parameter representing the distortion of the lens assuming optical axis symmetry with respect to the angle of incidence to the camera as a projection function of a fourth order or lower, and a posture parameter representing the posture of the camera. The system estimates camera parameters, calculates the inverse function of the projection function in closed form, calculates multiple true values ​​obtained by projecting multiple reference points on the image onto world coordinates using the inverse function and the true camera parameters, calculates multiple estimated values ​​obtained by projecting the multiple reference points onto world coordinates using the inverse function and the estimated camera parameters, calculates a network error indicating the error between the true camera parameters and the camera parameter estimation result by the deep neural network based on each combination of each of the multiple true values ​​and each of the multiple estimated values, and learns the parameters of the deep neural network to minimize the calculated network error.

[0011] According to this disclosure, camera parameters can be calculated with high accuracy from a single image. [Brief explanation of the drawing]

[0012] [Figure 1] This block diagram shows an example of the configuration of the camera parameter calculation system in the embodiment of the present disclosure. [Figure 2] This flowchart shows an example of the camera parameter calculation process by the camera parameter calculation device of the embodiment of the present disclosure. [Figure 3] This is a block diagram showing an example of the configuration of a learning device in an embodiment of the present disclosure. [Figure 4] This is a flowchart showing an example of learning processing by a learning device according to an embodiment of the present disclosure. [Figure 5] This is a flowchart showing an example of DNN learning processing in step S13 of FIG. 4.

Embodiments for Carrying Out the Invention

[0013] (Findings on which the present disclosure is based) In recent years, sensing by cameras has been carried out. However, in order to perform image recognition with high accuracy, camera calibration is required. However, in camera calibration of a camera with large lens distortion such as a fisheye camera, it is difficult to calculate camera parameters with high accuracy from a single image by conventional deep learning-based camera calibration.

[0014] In order to solve the above problems, the following techniques are disclosed.

[0015] (1) A learning device according to one aspect of the present disclosure includes: an image acquisition unit that acquires an image captured by a camera; a camera parameter acquisition unit that acquires the true camera parameters of the camera; a learning unit that learns a deep neural network by deep learning using the image acquired by the image acquisition unit and the true camera parameters acquired by the camera parameter acquisition unit; and an output unit that outputs the deep neural network learned by the learning unit, wherein the learning unit inputs the image to the deep neural network and obtains a distortion parameter representing the distortion of the lens assuming optical axis symmetry with respect to the angle of incidence to the camera and a pose parameter representing the pose of the camera. The system estimates camera parameters including the above, calculates the inverse function of the projection function in closed form, calculates a plurality of true values ​​obtained by projecting a plurality of reference points on the image onto world coordinates using the inverse function and the true camera parameters, calculates a plurality of estimated values ​​obtained by projecting the plurality of reference points onto world coordinates using the inverse function and the estimated camera parameters, calculates a network error indicating the error between the true camera parameters and the camera parameter estimation result by the deep neural network based on each combination of each of the plurality of true values ​​and each of the plurality of estimated values, and learns the parameters of the deep neural network to minimize the calculated network error.

[0016] In this configuration, the distortion parameter representing lens distortion is expressed by a projection function of a fourth degree or lower with respect to the angle of incidence to the camera, allowing the inverse function of the projection function to be calculated in closed form. Then, using the inverse function and the true camera parameters, the reference point of the image coordinates is projected onto the world coordinates, and the reference point of the image coordinates is projected onto the world coordinates using the inverse function and the estimated camera parameters, and the deep neural network can be trained using the network error based on the world coordinates. Therefore, by inputting a single image into a deep neural network trained by deep learning, camera parameters can be calculated with high accuracy from a single image.

[0017] (2) In the learning device described in (1) above, the angle of view of the camera may be 180° or more.

[0018] With this configuration, cameras with a field of view of 180° or more exhibit significant lens distortion. However, by inputting a single image captured by a camera with high lens distortion into a deep neural network trained through deep learning, it becomes possible to calculate camera parameters with high accuracy from a single image captured by such a camera.

[0019] (3) In the learning device described in (1) or (2) above, the projection function may be represented by a first-order term of the angle of incidence and a third-order term of the angle of incidence.

[0020] With this configuration, the projection function is expressed as a cubic polynomial, which facilitates deep learning of deep neural networks. Furthermore, while it is difficult to calculate the inverse function of a projection function in closed form when its degree is greater than 4, with the above configuration, the degree of the projection function is 3, so the inverse function of the projection function can be calculated in closed form.

[0021] (4) In the learning device described in (3) above, the coefficient of the first term of the angle of incidence is 1, and the coefficient of the third term of the angle of incidence may be in the range of -1 / 6 or more and 1 / 3 or less.

[0022] With this configuration, if the coefficient of the third-order term of the angle of incidence is in the range of -1 / 6 to 1 / 3, all projection schemes of a fisheye lens can be represented, and the range of coefficients (distortion parameters) to be estimated by deep learning can be set to the range of -1 / 6 to 1 / 3.

[0023] (5) In the learning device described in any one of (1) to (4) above, the camera parameters further include the focal length of the camera, and the focal length estimated by the deep neural network may be in the range of 1 / 4 to 1 / 2 of the vertical length of the image sensor provided by the camera.

[0024] According to this configuration, the focal length of the fisheye lens is in the range of 1 / 4 to 1 / 2 of the vertical length of the image sensor, so the range of focal lengths to be estimated by deep learning included in the camera parameters of the fisheye camera is The vertical length of the image sensor It can be in the range of 1 / 4 to 1 / 2.

[0025] (6) In the learning device described in any one of (1) to (5) above, the plurality of true values ​​are a plurality of first world coordinate points obtained by projecting the plurality of reference points onto the world coordinate using the inverse function and the true camera parameters, the plurality of estimated values ​​are a plurality of second world coordinate points obtained by projecting the plurality of reference points onto the world coordinate using the inverse function and the estimated camera parameters, and the learning unit may calculate the network error based on the distance between each combination of the plurality of first world coordinate points and each of the plurality of second world coordinate points.

[0026] With this configuration, multiple first-world coordinate points are calculated by projecting multiple reference points on the image onto world coordinates using an inverse function and true camera parameters, and multiple second-world coordinate points are calculated by projecting multiple reference points on the image onto world coordinates using an inverse function and estimated camera parameters. Then, the network error can be calculated based on the distance between each combination of the multiple first-world coordinate points and each of the multiple second-world coordinate points.

[0027] (7) In the learning device described in any one of (1) to (5) above, the plurality of true values ​​are a plurality of first unit line-of-sight vectors obtained by projecting the plurality of reference points onto the world coordinates using the inverse function and the true camera parameters, the plurality of estimated values ​​are a plurality of second unit line-of-sight vectors obtained by projecting the plurality of reference points onto the world coordinates using the inverse function and the estimated camera parameters, and the learning unit may calculate the network error based on the angle of each combination of each of the plurality of first unit line-of-sight vectors and each of the plurality of second unit line-of-sight vectors.

[0028] With this configuration, multiple first unit line-of-sight vectors are calculated by projecting multiple reference points on the image onto world coordinates using an inverse function and true camera parameters, and multiple second unit line-of-sight vectors are calculated by projecting multiple reference points on the image onto world coordinates using an inverse function and estimated camera parameters. Then, the network error can be calculated based on the angle of each combination of each of the multiple first unit line-of-sight vectors and each of the multiple second unit line-of-sight vectors.

[0029] (8) In the learning device described in any one of (1) to (5) above, the plurality of true values ​​are a plurality of first unit line-of-sight vectors obtained by projecting the plurality of reference points onto the world coordinates using the inverse function and the true camera parameters, the plurality of estimated values ​​are a plurality of second unit line-of-sight vectors obtained by projecting the plurality of reference points onto the world coordinates using the inverse function and the estimated camera parameters, and the learning unit may calculate the network error based on the distances of each combination of the plurality of first unit line-of-sight vectors and a plurality of first intersections between each of the plurality of second unit line-of-sight vectors and a plurality of second intersections between each of the plurality of second unit line-of-sight vectors and the unit sphere.

[0030] With this configuration, multiple first unit line-of-sight vectors are calculated by projecting multiple reference points on the image onto world coordinates using an inverse function and true camera parameters, and multiple second unit line-of-sight vectors are calculated by projecting multiple reference points on the image onto world coordinates using an inverse function and estimated camera parameters. Then, the network error can be calculated based on the distances of each combination of multiple first intersection points between each of the multiple first unit line-of-sight vectors and the unit sphere, and each of the multiple second intersection points between each of the multiple second unit line-of-sight vectors and the unit sphere.

[0031] Furthermore, this disclosure can be implemented not only as a learning device having the characteristic configuration described above, but also as a learning method that performs characteristic processing corresponding to the characteristic configuration of the learning device. It can also be implemented as a computer program that causes a computer to execute the characteristic processing included in such a learning method. Therefore, the same effects as the learning device described above can be achieved in the following other embodiments.

[0032] (9) A learning method relating to another aspect of the present disclosure is a learning method in a computer, which includes acquiring an image captured by a camera, acquiring true camera parameters of the camera, training a deep neural network by deep learning using the acquired image and the acquired true camera parameters, outputting the trained deep neural network, and in training the deep neural network, inputting the image into the deep neural network to estimate camera parameters including a distortion parameter representing the distortion of the lens assuming optical axis symmetry with respect to the angle of incidence to the camera and an attitude parameter representing the attitude of the camera, The inverse function of the projection function is calculated in closed form, multiple true values ​​are calculated by projecting multiple reference points on the image onto world coordinates using the inverse function and the true camera parameters, multiple estimated values ​​are calculated by projecting the multiple reference points onto world coordinates using the inverse function and the estimated camera parameters, a network error is calculated based on each combination of each of the multiple true values ​​and each of the multiple estimated values, the error between the true camera parameters and the estimate of the camera parameters by the deep neural network is calculated, and the parameters of the deep neural network are learned to minimize the calculated network error.

[0033] (10) A learning program according to another aspect of the present disclosure includes an image acquisition unit that acquires images captured by a camera, a camera parameter acquisition unit that acquires true camera parameters of the camera, a learning unit that learns a deep neural network by deep learning using the images acquired by the image acquisition unit and the true camera parameters acquired by the camera parameter acquisition unit, and a computer that functions as an output unit that outputs the deep neural network learned by the learning unit, wherein the learning unit inputs the images into the deep neural network to obtain a distortion parameter that represents the distortion of the lens assuming optical axis symmetry with respect to the angle of incidence to the camera and a projection function of a fourth-degree or lower degree, and the pose of the camera. Camera parameters, including pose parameters, are estimated; the inverse function of the projection function is calculated in closed form; multiple true values ​​are calculated by projecting multiple reference points on the image onto world coordinates using the inverse function and the true camera parameters; multiple estimated values ​​are calculated by projecting the multiple reference points onto world coordinates using the inverse function and the estimated camera parameters; a network error is calculated based on each combination of each of the multiple true values ​​and each of the multiple estimated values, indicating the error between the true camera parameters and the camera parameter estimation results by the deep neural network; and the parameters of the deep neural network are learned to minimize the calculated network error.

[0034] (11) A camera parameter calculation device according to another aspect of the present disclosure comprises: an image acquisition unit that acquires an image captured by a camera; an estimation unit that estimates the camera parameters of the camera by inputting the image acquired by the image acquisition unit into a deep neural network trained by deep learning; and an output unit that outputs the camera parameters estimated by the estimation unit, wherein during the training of the deep neural network, a training image is acquired, the true camera parameters of the camera that captured the training image are acquired, and the training image is input into the deep neural network, thereby generating a distortion parameter that represents the distortion of the lens assuming optical axis symmetry with respect to the angle of incidence to the camera that captured the training image, and the camera Camera parameters, including pose parameters representing the pose, are estimated; the inverse function of the projection function is calculated in closed form; multiple true values ​​are calculated by projecting multiple reference points on the training image onto world coordinates using the inverse function and the true camera parameters; multiple estimated values ​​are calculated by projecting the multiple reference points onto world coordinates using the inverse function and the estimated camera parameters; a network error is calculated based on each combination of each of the multiple true values ​​and each of the multiple estimated values, indicating the error between the true camera parameters and the camera parameter estimation result by the deep neural network; and the parameters of the deep neural network are learned so as to minimize the calculated network error.

[0035] In this configuration, the distortion parameter representing lens distortion is expressed by a projection function of a fourth degree or lower with respect to the angle of incidence to the camera, allowing the inverse function of the projection function to be calculated in closed form. Then, using the inverse function and the true camera parameters, the reference point of the image coordinates is projected onto the world coordinates, and the reference point of the image coordinates is projected onto the world coordinates using the inverse function and the estimated camera parameters, and the deep neural network can be trained using the network error based on the world coordinates. Therefore, by inputting a single image into a deep neural network trained by deep learning, camera parameters can be calculated with high accuracy from a single image.

[0036] Furthermore, this disclosure can be implemented not only as a camera parameter calculation device having the characteristic configuration described above, but also as a camera parameter calculation method that performs characteristic processing corresponding to the characteristic configuration of the camera parameter calculation device. It can also be implemented as a computer program that causes a computer to execute the characteristic processing included in such a camera parameter calculation method. Therefore, the same effects as the above-described camera parameter calculation device can be achieved in the following other embodiments.

[0037] (12) A camera parameter calculation method in another aspect of the present disclosure is a camera parameter calculation method in a computer, comprising: acquiring an image captured by a camera; inputting the image acquired by the image acquisition unit into a deep neural network trained by deep learning to estimate the camera parameters of the camera; outputting the estimated camera parameters by the estimation unit; during the training of the deep neural network, a training image is acquired, the true camera parameters of the camera that captured the training image are acquired, and the training image is input into the deep neural network to obtain a distortion parameter representing the distortion of the lens assuming optical axis symmetry with respect to the angle of incidence to the camera that captured the training image, and Camera parameters, including pose parameters representing the camera's orientation, are estimated; the inverse function of the projection function is calculated in closed form; multiple true values ​​are calculated by projecting multiple reference points on the training image onto world coordinates using the inverse function and the true camera parameters; multiple estimated values ​​are calculated by projecting the multiple reference points onto world coordinates using the inverse function and the estimated camera parameters; a network error is calculated based on each combination of each of the multiple true values ​​and each of the multiple estimated values, indicating the error between the true camera parameters and the camera parameter estimation result by the deep neural network; and the parameters of the deep neural network are learned so as to minimize the calculated network error.

[0038] (13) A camera parameter calculation program according to another aspect of the present disclosure includes an image acquisition unit that acquires an image captured by a camera, an estimation unit that estimates the camera parameters of the camera by inputting the image acquired by the image acquisition unit into a deep neural network trained by deep learning, and an output unit that outputs the camera parameters estimated by the estimation unit, wherein during the training of the deep neural network, a training image is acquired, the true camera parameters of the camera that captured the training image are acquired, and the training image is input into the deep neural network, thereby generating a distortion parameter that represents the distortion of the lens assuming optical axis symmetry with respect to the angle of incidence to the camera that captured the training image and a projection function of a fourth order or lower. Camera parameters, including pose parameters representing the camera's orientation, are estimated; the inverse function of the projection function is calculated in closed form; multiple true values ​​are calculated by projecting multiple reference points on the training image onto world coordinates using the inverse function and the true camera parameters; multiple estimated values ​​are calculated by projecting the multiple reference points onto world coordinates using the inverse function and the estimated camera parameters; a network error is calculated based on each combination of each of the multiple true values ​​and each of the multiple estimated values, indicating the error between the true camera parameters and the camera parameter estimation result by the deep neural network; and the parameters of the deep neural network are learned so as to minimize the calculated network error.

[0039] Furthermore, this disclosure allows computer programs to be distributed via computer-readable non-temporary recording media such as CD-ROMs or communication networks such as the Internet. Therefore, the same effects as the learning device or camera parameter calculation device described above can be achieved in the following other embodiments.

[0040] (14) A computer-readable non-temporary recording medium recording a learning program according to another aspect of the present disclosure includes an image acquisition unit that acquires images captured by a camera, a camera parameter acquisition unit that acquires true camera parameters of the camera, a learning unit that learns a deep neural network by deep learning using the images acquired by the image acquisition unit and the true camera parameters acquired by the camera parameter acquisition unit, and an output unit that outputs the deep neural network learned by the learning unit, wherein the learning unit inputs the images into the deep neural network, thereby generating distortion parameters that represent the distortion of the lens assuming optical axis symmetry with respect to the angle of incidence to the camera using a projection function of a fourth-order or lower degree. Camera parameters are estimated, including the camera's orientation and orientation parameters representing the camera's pose. The inverse function of the projection function is calculated in closed form. Multiple true values ​​are calculated by projecting multiple reference points on the image onto world coordinates using the inverse function and the true camera parameters. Multiple estimated values ​​are calculated by projecting the multiple reference points onto world coordinates using the inverse function and the estimated camera parameters. A network error is calculated based on each combination of each of the multiple true values ​​and each of the multiple estimated values, indicating the error between the true camera parameters and the camera parameter estimation result by the deep neural network. The parameters of the deep neural network are then trained to minimize the calculated network error.

[0041] (15) A computer-readable non-temporary recording medium recording a camera parameter calculation program according to another aspect of the present disclosure includes an image acquisition unit that acquires an image taken by a camera, an estimation unit that estimates the camera parameters of the camera by inputting the image acquired by the image acquisition unit into a deep neural network trained by deep learning, and an output unit that outputs the camera parameters estimated by the estimation unit, wherein during the training of the deep neural network, a training image is acquired, the true camera parameters of the camera that took the training image are acquired, and the training image is input into the deep neural network, thereby assuming optical axis symmetry with respect to the angle of incidence to the camera that took the training image. Camera parameters are estimated, including a distortion parameter representing the distortion of the image and a posture parameter representing the pose of the camera; the inverse function of the projection function is calculated in closed form; multiple true values ​​are calculated by projecting multiple reference points on the training image onto world coordinates using the inverse function and the true camera parameters; multiple estimated values ​​are calculated by projecting the multiple reference points onto world coordinates using the inverse function and the estimated camera parameters; a network error is calculated based on each combination of each of the multiple true values ​​and each of the multiple estimated values, indicating the error between the true camera parameters and the camera parameter estimation result by the deep neural network; and the parameters of the deep neural network are learned so as to minimize the calculated network error.

[0042] The embodiments described below are all specific examples of this disclosure. The numerical values, shapes, components, steps, and order of steps shown in the following embodiments are examples only and are not intended to limit this disclosure. Furthermore, among the components in the following embodiments, those not described in the independent claim representing the highest-level concept will be described as optional components. In addition, the contents of each embodiment can be combined.

[0043] (Embodiment) The embodiments of this disclosure will be described below with reference to the drawings.

[0044] Figure 1 is a block diagram showing an example of the configuration of a camera parameter calculation system in an embodiment of the present disclosure.

[0045] The camera parameter calculation system comprises a camera parameter calculation device 1 and a camera 4.

[0046] In this embodiment, camera 4 is a fixed camera installed inside the house where the user to be recognized by sensing resides. Camera 4 captures the user at a predetermined frame rate and inputs the captured image to the camera parameter calculation device 1 at a predetermined frame rate. Camera 4 is, for example, a fisheye camera (ultra-wide-angle camera) with a field of view of 180° or more. Camera 4 may also be a wide-angle camera with a field of view of 60° or more, or a narrow-angle camera with a field of view of less than 25°.

[0047] The camera parameter calculation device 1 consists of a computer including a processor 2, memory 3, and interface circuitry (not shown in the diagram). The processor 2 is, for example, a central processing unit. The memory 3 is, for example, a non-volatile, rewritable storage device such as flash memory, a hard disk drive, or a solid-state drive. The interface circuitry is, for example, a communication circuit.

[0048] The camera parameter calculation device 1 may consist of an edge server installed in the house, a smart speaker installed in the house, or a cloud server. When the camera parameter calculation device 1 consists of an edge server or a smart speaker, the camera 4 and the camera parameter calculation device 1 are connected via a local area network. When the camera parameter calculation device 1 consists of a cloud server, the camera 4 and the camera parameter calculation device 1 are connected via a wide-area communication network such as the internet. Furthermore, a portion of the camera parameter calculation device 1 may be located on the edge side, with the remainder located on the cloud side.

[0049] The processor 2 includes an acquisition unit 21, a camera parameter estimation unit 22, and an output unit 23. The acquisition unit 21 to the output unit 23 may be implemented by a central processing unit executing a camera parameter calculation program, or they may be composed of dedicated hardware circuits such as an ASIC (Application Specific Integrated Circuit).

[0050] The acquisition unit 21 acquires images captured by the camera 4. The acquisition unit 21 stores the acquired images in the frame memory 31.

[0051] The camera parameter estimation unit 22 estimates the camera parameters of camera 4 by inputting the images acquired by the acquisition unit 21 into a deep neural network (hereinafter also referred to as DNN) trained by deep learning. The camera parameter estimation unit 22 reads the DNN from the DNN memory unit 32. The camera parameter estimation unit 22 calculates the camera parameters from the images read from the frame memory 31 using the DNN trained by deep learning. An example of a DNN is a convolutional neural network that includes convolutional layers and pooling layers.

[0052] During DNN training, training images are acquired. Next, the true camera parameters of the camera that captured the training images are acquired. Then, by inputting the training images into the DNN, camera parameters are estimated, including a distortion parameter representing the lens distortion assuming optical axis symmetry using a projection function of a fourth degree or lower with respect to the angle of incidence to the camera that captured the training images, and an attitude parameter representing the camera's orientation. Next, the inverse function of the projection function is calculated in closed form. Next, multiple true values ​​are calculated by projecting multiple reference points on the training images onto world coordinates using the inverse function and the true camera parameters. Next, multiple estimated values ​​are calculated by projecting multiple reference points onto world coordinates using the inverse function and the estimated camera parameters. Next, a network error is calculated, representing the error between the true camera parameters and the DNN's estimation of the camera parameters, based on each combination of the multiple true values ​​and each of the multiple estimated values. Finally, the DNN parameters are trained to minimize the calculated network error.

[0053] Note that the true camera parameters are the correct camera parameters.

[0054] The output unit 23 outputs the camera parameters estimated by the camera parameter estimation unit 22.

[0055] Memory 3 includes frame memory 31 and DNN storage unit 32.

[0056] The frame memory 31 stores images acquired by the acquisition unit 21 from the camera 4. The frame memory 31 stores time-series images acquired by the acquisition unit 21.

[0057] The DNN storage unit 32 pre-stores the DNN to be used by the camera parameter estimation unit 22. The DNN storage unit 32 also stores the DNN generated by the learning device 5, which will be described later. The DNN may be stored in the DNN storage unit 32 when the camera parameter calculation device 1 is manufactured, or it may be received from an external server and stored in the DNN storage unit 32.

[0058] The camera parameter calculation device 1 does not necessarily have to be implemented by a single computer device, but may be implemented by a distributed processing system (not shown) including a terminal device and a server. For example, the acquisition unit 21 and frame memory 31 may be provided in the terminal device, and the DNN storage unit 32, camera parameter estimation unit 22, and output unit 23 may be provided in the server. In this case, data is exchanged between the components via a communication line connected to the terminal device and the server.

[0059] Figure 2 is a flowchart showing an example of the camera parameter calculation process by the camera parameter calculation device 1 of the embodiment of this disclosure. The operation of the camera parameter calculation device 1 will be described below in accordance with Figure 2. The camera parameter calculation process is performed when the camera 4 is installed, and thereafter is performed periodically, for example, every week or every month.

[0060] First, in step S1, the acquisition unit 21 acquires an image (fisheye image) captured by the camera 4. The acquisition unit 21 stores the acquired image in the frame memory 31.

[0061] Next, in step S2, the camera parameter estimation unit 22 reads an image from the frame memory 31 and inputs the read image into a pre-trained DNN to estimate the camera parameters. The DNN training method will be described later.

[0062] Next, in step S3, the output unit 23 outputs the camera parameters estimated by the camera parameter estimation unit 22.

[0063] The camera 4 for sensing can be calibrated using the procedure described above. In particular, this embodiment is useful for camera calibration of cameras 4 with large distortion, such as fisheye cameras.

[0064] Next, we will describe the learning device for training the DNN used in the camera parameter estimation unit 22.

[0065] Figure 3 is a block diagram showing an example of the configuration of the learning device 5 in an embodiment of this disclosure.

[0066] The learning device 5 consists of a computer including a processor 6, memory 7, and interface circuitry (not shown in the diagram). The processor 6 is, for example, a central processing unit. The memory 7 is, for example, a non-volatile, rewritable storage device such as flash memory, a hard disk drive, or a solid-state drive. The interface circuitry is, for example, a communication circuit.

[0067] The learning device 5 may consist of a cloud server or a personal computer.

[0068] The processor 6 includes an image acquisition unit 60, a camera parameter acquisition unit 61, a learning unit 62, and an output unit 63. The image acquisition unit 60 to the output unit 63 may be implemented by a central processing unit executing a learning program, or they may be composed of dedicated hardware circuits such as an ASIC.

[0069] Memory 7 includes a learning image storage unit 71, a camera parameter storage unit 72, and a DNN storage unit 73.

[0070] The learning image storage unit 71 pre-stores multiple learning images captured by the camera. These learning images are used when training the DNN. The camera used to obtain the learning images is the same as camera 4. The learning images are fisheye images, obtained by capturing them in advance with a fisheye camera. Alternatively, the learning images may be generated by performing computer graphics (CG) processing on a panoramic image using the camera parameters of the fisheye camera.

[0071] The camera parameter storage unit 72 stores the true camera parameters of the camera in advance. These true camera parameters are used when training the DNN. The true camera parameters are the camera parameters of the camera used to obtain the training images. Alternatively, the true camera parameters may be the camera parameters used when performing CG processing on the training images. The camera parameter storage unit 72 stores the true camera parameters corresponding to the training images. The camera parameters consist of external parameters related to the camera's orientation (rotation and translation relative to the world coordinate system) and internal parameters related to the focal length and lens distortion.

[0072] The image acquisition unit 60 acquires training images captured by the camera. The image acquisition unit 60 reads training images from the training image storage unit 71. In this embodiment, the image acquisition unit 60 acquires training images that have been stored in advance from the training image storage unit 71, but this disclosure is not limited to this. The image acquisition unit 60 may acquire training images from an external server. In this case, the image acquisition unit 60 may receive training images from an external server. The image acquisition unit 60 may also acquire training images from a camera connected to the learning device 5.

[0073] The camera parameter acquisition unit 61 acquires the true camera parameters of the camera. The camera parameter acquisition unit 61 reads the true camera parameters from the camera parameter storage unit 72. In this embodiment, the camera parameter acquisition unit 61 acquires true camera parameters that have been stored in advance from the camera parameter storage unit 72, but this disclosure is not limited to this. The camera parameter acquisition unit 61 may acquire true camera parameters from an external server. In this case, the camera parameter acquisition unit 61 may receive true camera parameters from an external server. The camera parameter acquisition unit 61 may also acquire true camera parameters input by an operator.

[0074] The learning unit 62 trains the DNN using deep learning with training images acquired by the image acquisition unit 60 and true camera parameters acquired by the camera parameter acquisition unit 61.

[0075] The learning unit 62 inputs a training image into the DNN and estimates camera parameters, including a distortion parameter representing the lens distortion assuming optical axis symmetry with respect to the angle of incidence to the camera, and an attitude parameter representing the camera's orientation. The learning unit 62 calculates the inverse function of the projection function in closed form. The learning unit 62 calculates multiple true values ​​obtained by projecting multiple reference points on the training image onto world coordinates using the inverse function and the true camera parameters. The learning unit 62 calculates multiple estimated values ​​obtained by projecting multiple reference points onto world coordinates using the inverse function and the estimated camera parameters. Based on each combination of the multiple true values ​​and each of the multiple estimated values, the learning unit 62 calculates a network error that shows the error between the true camera parameters and the camera parameter estimation results by the DNN.

[0076] Here, the multiple true values ​​are multiple first world coordinate points obtained by projecting multiple reference points onto world coordinates using the inverse function and the true camera parameters. The multiple estimated values ​​are multiple second world coordinate points obtained by projecting multiple reference points onto world coordinates using the inverse function and the estimated camera parameters. The learning unit 62 calculates the network error based on the distance between each combination of the multiple first world coordinate points and each of the multiple second world coordinate points.

[0077] The learning unit 62 learns the DNN parameters to minimize the calculated network error.

[0078] In this embodiment, the multiple true values ​​may be multiple first unit line-of-sight vectors obtained by projecting multiple reference points onto world coordinates using an inverse function and true camera parameters. Alternatively, the multiple estimated values ​​may be multiple second unit line-of-sight vectors obtained by projecting multiple reference points onto world coordinates using an inverse function and estimated camera parameters. In this case, the learning unit 62 may calculate the network error based on the angles of each combination of each of the multiple first unit line-of-sight vectors and each of the multiple second unit line-of-sight vectors.

[0079] Furthermore, in this embodiment, the multiple true values ​​may be multiple first unit line-of-sight vectors obtained by projecting multiple reference points onto world coordinates using an inverse function and true camera parameters. Also, the multiple estimated values ​​may be multiple second unit line-of-sight vectors obtained by projecting multiple reference points onto world coordinates using an inverse function and estimated camera parameters. In this case, the learning unit 62 may calculate the network error based on the distances of each combination of multiple first intersection points between each of the multiple first unit line-of-sight vectors and the unit sphere, and multiple second intersection points between each of the multiple second unit line-of-sight vectors and the unit sphere.

[0080] The output unit 63 outputs the DNN learned by the learning unit 62. The output unit 63 outputs the DNN to the DNN storage unit 73.

[0081] The DNN storage unit 73 stores the DNN learned by the learning unit 62. In this embodiment, the output unit 63 stores the DNN learned by the learning unit 62 in the DNN storage unit 73, but this disclosure is not limited to this. The output unit 63 may output the DNN learned by the learning unit 62 to an external server. In this case, the output unit 63 may transmit the DNN to an external server.

[0082] Next, the learning process performed by the learning device 5 will be explained with reference to the diagram.

[0083] Figure 4 is a flowchart showing an example of the learning process by the learning device 5 in the embodiment of this disclosure. The operation of the learning device 5 will be described below in accordance with Figure 4.

[0084] First, in step S11, the image acquisition unit 60 acquires training images to be used for training the DNN.

[0085] Next, in step S12, the camera parameter acquisition unit 61 acquires the true camera parameter Ω'.

[0086] Next, in step S13, the learning unit 62 learns the DNN using the training images and the true camera parameters Ω' (DNN learning process).

[0087] The learning unit 62 calculates the error between the true camera parameters and the estimated camera parameters. Camera parameters represent the relationship between projecting world coordinates onto image coordinates, or projecting image coordinates onto world coordinates. Note that the world coordinates converted from image coordinates are synonymous with the unit line of sight vector or the world coordinates on a unit sphere, since the scale is unknown.

[0088] The learning unit 62 projects a reference point (e.g., one pixel) on the training image onto a first world coordinate point on the unit sphere using true camera parameters. The learning unit 62 also projects the reference point on the training image onto a second world coordinate point on the unit sphere using estimated camera parameters. The learning unit 62 then calculates the distance between the first world coordinate point and the second world coordinate point. The learning unit 62 calculates the distance between the first world coordinate point and the second world coordinate point for all reference points on the training image, and calculates the average of the multiple calculated distances as the network error (loss).

[0089] Using the method for calculating Bearing loss described in Non-Patent Document 2 as an example, we will explain how to calculate network error (loss).

[0090] Here, we will explain the DNN training process in step S13 of Figure 4.

[0091] Figure 5 is a flowchart showing an example of the DNN training process in step S13 of Figure 4.

[0092] First, in step S21, the learning unit 62 estimates camera parameters by inputting training images into the DNN. The DNN extracts image features from convolutional layers, etc., and finally outputs the estimated camera parameters. For example, the DNN outputs four camera parameters: the camera tilt angle θ, roll angle ψ, focal length f, and distortion parameter k1. Details of the camera parameters will be described later. For simplicity, in the following explanation, the learning unit 62 estimates the above four camera parameters.

[0093] Furthermore, camera parameters other than those estimated (camera parameters that were not estimated) may be generated using the true camera parameters. That is, if the number of true camera parameters is greater than the number of estimated camera parameters, the values ​​of the true camera parameters may be used for the camera parameters that were not estimated.

[0094] Next, in step S22, the learning unit 62 selects one reference point P' from among the N reference points on the training image. i Extract the following. If the reference point is 1 pixel, N is the same as the number of pixels in the training image and is expressed as the product of the number of pixels in the height direction of the image and the number of pixels in the width direction of the image.

[0095] Next, in step S23, the learning unit 62 determines the reference point P' i The first world coordinate point P' on the unit sphere using the true camera parameter Ω'. w Project onto it.

[0096] Next, in step S24, the learning unit 62 determines the reference point P' i Then, using the estimated camera parameter Ω, the second world coordinate point P on the unit sphere is determined. w Project onto it.

[0097] Next, in step S25, the learning unit 62 projects the first world coordinate point P' onto the unit sphere. w and the second world coordinate point P w Calculate the Euclidean distance to [the specified location].

[0098] Next, in step S26, the learning unit 62 determines whether all reference points on the training image have been extracted. If it is determined that not all reference points have been extracted (NO in step S26), the process returns to step S22. The learning unit 62 then identifies the other reference points P' on the training image that have not been extracted. i Extract it.

[0099] On the other hand, if it is determined that all reference points have been extracted (YES in step S26), in step S27, the learning unit 62 calculates the average of the calculated distances as the network error. Alternatively, the learning unit 62 may calculate the square of the average of the calculated distances as the network error. Or, the learning unit 62 may calculate the Huber loss of the calculated distances as the network error. The Huber loss is an error function that is a squared error for absolute errors less than 0.5 and a first-power error for absolute errors of 0.5 or greater.

[0100] Next, in step S28, the learning unit 62 updates the DNN parameters by backpropagation using the calculated network error. Stochastic gradient descent or the like is used to optimize backpropagation.

[0101] The learning unit 62 may convert the reference points on the training image into a first unit view vector using true camera parameters. Alternatively, the learning unit 62 may convert the reference points on the training image into a second unit view vector using estimated camera parameters. The learning unit 62 may calculate the angle between the first unit view vector and the second unit view vector. The learning unit 62 may calculate the angle between the first unit view vector and the second unit view vector for all reference points on the training image and calculate the average of the multiple calculated angles as the network error (loss).

[0102] The learning unit 62 may also calculate a first intersection point between the first unit line of sight vector and the unit sphere centered on the camera. The learning unit 62 may also calculate a second intersection point between the second unit line of sight vector and the unit sphere. The learning unit 62 may calculate the Euclidean distance between the first and second intersection points on the unit sphere. The learning unit 62 may calculate the Euclidean distance between the first and second intersection points for all reference points on the training image and calculate the average of the multiple calculated Euclidean distances as the network error (loss). This network error is called Bearing loss in Non-Patent Literature 2.

[0103] Furthermore, the learning unit 62 may project world coordinate points on a unit sphere (for example, 10,000 points) onto first image coordinate points on the image using true camera parameters. Alternatively, the learning unit 62 may project world coordinate points on a unit sphere onto second image coordinate points on the image using estimated camera parameters. The learning unit 62 may calculate the distance between the first image coordinate points and the second image coordinate points. The learning unit 62 may calculate the distance between the first image coordinate points and the second image coordinate points for all world coordinate points on the unit sphere and calculate the average of the multiple calculated distances as the network error (loss).

[0104] Returning to Figure 4, in step S14, the learning unit 62 determines whether the DNN training is complete. For example, the learning unit 62 determines that the DNN training is complete if the number of DNN parameter updates exceeds a threshold, and determines that the DNN training is not complete if the number of DNN parameter updates is less than or equal to the threshold. The threshold is, for example, 10,000 times.

[0105] Note that the learning unit 62 may determine that the learning of the DNN is completed when the network error is smaller than the threshold value, and may determine that the learning of the DNN is not completed when the network error is greater than or equal to the threshold value. When the network error is the average of a plurality of distances on the unit sphere, the threshold value is, for example, 0.1. Also, when the network error is the average of the angles of the unit line-of-sight vectors, the threshold value is, for example, 0.01. Also, when the network error is the average of a plurality of distances of the image coordinates point the threshold value is, for example, 3 pixels.

[0106] Here, when it is determined that the learning of the DNN is not completed (NO in step S14), the process returns to step S11. Then, in step S11, the image acquisition unit 60 acquires another learning image.

[0107] On the other hand, when it is determined that the learning of the DNN is completed (YES in step S14), in step S15, the output unit 63 outputs the DNN learned by the learning unit 62. The output unit 63 stores the DNN in the DNN storage unit 73.

[0108] Subsequently, an example of the camera parameters in the present disclosure will be described below. The conversion formula from the world coordinate system to the image coordinate system is represented by the following mathematical formulas (1) to (4). The camera parameters are the parameters of the projection formula that projects the world coordinates onto the image coordinates. Γ(η) in the mathematical formula (3) is a projection function representing lens distortion. Here, η is the incident angle.

[0109] [Number]

[0110] Here, (X, Y, Z) are the world coordinate values, and (x, y) are the image coordinate values. (C x , C y ) are the principal point image coordinates of the camera, and r 11 ~r 33 are the components of the 3x3 rotation matrix R representing the rotation with respect to the reference of the world coordinates, and (T X , T Y , TZ ) is a translation vector relative to the world coordinate system, and d x and d y is the pixel pitch in the horizontal and vertical directions of the camera's image sensor. In equations (1) to (4), d x d y , C x , C y , r 11 ~r 33 , T X , T Y , T Z These are the camera parameters.

[0111] Formulas (1) to (4) represent the transformation from (X,Y,Z) to (x,y). When transforming from (x,y) to (X,Y,Z) on the unit sphere, the inverse function or inverse matrix of formulas (1) to (4) is used. Note that rotation matrices are invertible, and their inverses can always be calculated, as can 4x4 matrices containing translation vectors. Furthermore, even when formulas (1) to (4) include arctan and arithmetic operations, the inverse functions of formulas (1) to (4) can be calculated. Inverse trigonometric functions such as arctan can be calculated within the appropriate domain. Therefore, if the inverse function of Γ can be calculated, it becomes possible to calculate the transformation from image coordinates to world coordinates.

[0112] An example of the γ (projection function) of an optically axially symmetric lens is expressed as a function of the angle of incidence η by the following equations (5) to (9).

[0113] γ = fsin(η)···(5) γ = 2fsin(η / 2)···(6) γ = fη···(7) γ = 2ftan(η / 2)···(8) γ = ftan(η)···(9)

[0114] Equation (5) represents the projection function of orthogonal projection, equation (6) represents the projection function of equisolid angle projection, equation (7) represents the projection function of equidistant projection, equation (8) represents the projection function of stereoscopic projection, and equation (9) represents the projection function of a pinhole camera. f represents the focal length, and η represents the angle of incidence.

[0115] The general camera model described in Non-Patent Document 3 (DBGennery, "Generalized Camera Calibration Including Fish-Eye Lenses," International Journal of Computer Vision, 68, 239-266, 2006) is represented by the following formula (10).

[0116] γ = k1η + k2η 3 +k2η 5 +··· ···(10)

[0117] In the above equation (10), η represents the angle of incidence, and k1 and k2 represent the distortion parameters (distortion coefficients), which are one of the camera parameters.

[0118] In Non-Patent Document 3 mentioned above, the general camera model is expressed as an arbitrary polynomial. Equation (10) above is an Nth-degree polynomial, and calculating its inverse function requires finding the solution to an Nth-degree equation. However, equations that can be calculated in closed form are limited to fourth-degree equations. Therefore, the general solution to the inverse function of equation (10) above cannot be obtained. For this reason, the conversion from image coordinates to line-of-view vectors cannot be applied to deep learning necessary for calculating network errors (losses). In view of the above, the inventors of this disclosure propose a general fisheye camera model shown in equation (11) below.

[0119] γ = f(η + k1η) 3 )···(11)

[0120] As shown in equation (11) above, the projection function is expressed by a first-order term of the angle of incidence and a third-order term of the angle of incidence. In equation (11) above, f represents the focal length, η represents the angle of incidence, and k1 represents the distortion parameter (distortion coefficient), which is one of the camera parameters.

[0121] Unlike Non-Patent Document 3 mentioned above, formula (11) explicitly defines the focal length f. The focal length f has a physical meaning as the scale of the image, and f is easy to estimate in deep learning. On the other hand, Non-Patent Document 3 assumes that the focal length f is a fixed value of 1 mm, and the coefficients of the polynomial implicitly include the focal length.

[0122] Since equation (11) above is a cubic function in terms of η, the inverse function requires solving a cubic equation. Cardano's formula is a known closed-form solution, and three complex solutions can be obtained using the cube roots of unity. Here, since equation (11) is a camera model, the focal length f is positive and k1 is a real number. Under these conditions, the solution to equation (11) will be one of the following (i) or (ii).

[0123] (i) One real solution and two imaginary solutions.

[0124] (ii) Three real solutions.

[0125] The signs of the discriminant used to calculate the three complex solutions of Cardano's formula distinguish between (i) and (ii). If the solution to equation (11) is the solution to (i), then the real solution η should be selected. On the other hand, if the solution to equation (11) is the solution to (ii), the relative magnitudes of the three real solutions can be calculated, and when the three solutions are arranged in ascending order, the middle value η should be selected. The remaining two solutions are false angles of incidence η that exist in the formula. The reason the middle solution is suitable is that, under the above conditions, if there are three real solutions, one real solution will be negative and the remaining two real solutions will be positive. Since the angle of incidence is greater than or equal to 0, negative real solutions are unsuitable. Furthermore, for the function of equation (11) and its inverse function to correspond one-to-one and be bijective, the function of equation (11) and its inverse function must be monotonically increasing. The function is monotonically increasing only if the two positive real solutions are closer to zero. If the larger solution is selected, the function will not be monotonically increasing and will not be bijective.

[0126] Furthermore, while it is possible to estimate real values ​​in deep learning, the space of all real numbers is infinite, and estimation without specifying a range is extremely inefficient. Here, equation (11) can be approximated by a cubic Taylor expansion and expressed as equations (12) to (16) below.

[0127] γ = f(η - 1 / 6·η) 3 )···(12) γ = f(η - 1 / 24·η) 3 )···(13) γ = fη···(14) γ = f(η + 1 / 12·η) 3 )···(15) γ = f(η + 1 / 3·η) 3 )···(16)

[0128] Equation (12) represents the projection function of orthogonal projection, equation (13) represents the projection function of equisolid angle projection, equation (14) represents the projection function of equidistant projection, equation (15) represents the projection function of stereoscopic projection, and equation (16) represents the projection function of a pinhole camera.

[0129] From equations (12) to (16) above, if the distortion parameter k1, which is the coefficient of the third-order term of the angle of incidence, satisfies -1 / 6 ≤ k1 ≤ 1 / 3, then all approximate projection schemes can be represented. That is, the range of k1 to be estimated by deep learning is between -1 / 6 and 1 / 3. In the projection function shown in equation (11) above, the coefficient of the first-order term of the angle of incidence is 1, and the coefficient k1 of the third-order term of the angle of incidence is in the range of -1 / 6 to 1 / 3. The learning unit 62 sets the scale of the output value of the sigmoid function, which is the output of the DNN, from 0 to 1. The learning unit 62 sets the range of the output value of the distortion parameter k1 from the DNN to be between -1 / 6 and 1 / 3.

[0130] Furthermore, when the camera's pose, such as rotation angle or focal length, is estimated by deep learning, efficient learning is possible if the focal length is set as follows. The focal length f estimated by the DNN is preferably in the range of 1 / 4 to 1 / 2 of the vertical length of the image sensor equipped in the camera. The focal length is the distance between the image sensor and the principal point of the lens. The focal length of most commercially available fisheye lenses corresponds to a range of 6 mm to 15 mm when converted to a full-frame image sensor with a vertical length of 24 mm. In addition, regarding the camera's pose parameters, the tilt angle and roll angle should be estimated within a range of ±90° (Euler angle). In camera calibration from a single image, the pan angle is always treated as 0°. This is because there is no standard for the pan angle.

[0131] A general camera model is defined as shown in equation (11), and by performing calculations according to the procedure described above, world coordinates on a unit sphere can be calculated from image coordinates, and a DNN can be trained using the network error (loss) based on world coordinates. As a result, camera parameters can be calculated with high accuracy from a single distorted image taken with a fisheye camera.

[0132] Furthermore, since the distortion parameter representing lens distortion is expressed by a projection function of a fourth degree or lower with respect to the angle of incidence to the camera, the inverse function of the projection function can be calculated in closed form. Then, using the inverse function and the true camera parameters, the reference point of the image coordinates is projected onto the world coordinates, and the reference point of the image coordinates is projected onto the world coordinates using the inverse function and the estimated camera parameters, and the DNN can be trained using the network error based on the world coordinates. Therefore, by inputting a single image into a DNN trained by deep learning, camera parameters can be calculated with high accuracy from a single image.

[0133] (modified version) The camera parameter calculation device and learning device according to one or more embodiments of this disclosure have been described above based on embodiments, but this disclosure is not limited to these embodiments. Without departing from the spirit of this disclosure, various modifications that a person skilled in the art can conceive of may be applied to these embodiments, and forms constructed by combining components from different embodiments may also be included within the scope of one or more embodiments of this disclosure.

[0134] In each of the above embodiments, each component may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0135] Some or all of the functions of the apparatus according to the embodiments of this disclosure are typically implemented as an integrated circuit, or LSI (Large Scale Integration). These may be individually integrated onto a single chip, or some or all of them may be integrated onto a single chip. Furthermore, the integration is not limited to LSIs, but may also be implemented using dedicated circuits or general-purpose processors. An FPGA (Field Programmable Gate Array) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells inside the LSI may also be used.

[0136] Furthermore, some or all of the functions of the apparatus according to the embodiments of this disclosure may be realized by a processor such as a CPU executing a program.

[0137] Furthermore, all figures used above are illustrative examples provided to illustrate this disclosure, and this disclosure is not limited to these illustrative figures.

[0138] Furthermore, the order in which the steps shown in the flowchart above are performed is illustrative for the purpose of specifically illustrating this disclosure, and other orders are acceptable as long as similar effects are achieved. Also, some of the steps above may be performed simultaneously (in parallel) with other steps. [Industrial applicability]

[0139] The technology disclosed herein can calculate camera parameters with high accuracy from a single image, and is therefore useful as a technology for training a deep neural network for calculating camera parameters from an image, and as a technology for calculating camera parameters from an image.

Claims

1. An image acquisition unit that acquires images captured by a camera, A camera parameter acquisition unit that acquires the true camera parameters of the aforementioned camera, A learning unit that trains a deep neural network by deep learning using the image acquired by the image acquisition unit and the true camera parameters acquired by the camera parameter acquisition unit, An output unit that outputs the deep neural network learned by the learning unit, Equipped with, The aforementioned learning unit, By inputting the aforementioned image into the deep neural network, camera parameters are estimated, including a distortion parameter representing the lens distortion assuming optical axis symmetry with respect to the angle of incidence to the camera using a projection function of a fourth degree or lower, and a posture parameter representing the orientation of the camera. The inverse function of the aforementioned projection function is calculated in closed form, Multiple true values ​​are calculated by projecting multiple reference points on the aforementioned image onto world coordinates using the inverse function and the true camera parameters. A plurality of estimated values ​​are calculated by projecting the plurality of reference points onto the world coordinates using the inverse function and the estimated camera parameters. Based on each combination of the multiple true values ​​and each of the multiple estimated values, a network error is calculated that represents the error between the true camera parameters and the estimates of the camera parameters by the deep neural network. The parameters of the deep neural network are learned to minimize the calculated network error. Learning device.

2. The field of view of the aforementioned camera is 180° or more. The learning device according to claim 1.

3. The projection function is expressed by a first-order term of the angle of incidence and a third-order term of the angle of incidence. The learning device according to claim 1.

4. The coefficient of the first term for the angle of incidence is 1. The coefficient of the third term for the angle of incidence is in the range of -1 / 6 or more and 1 / 3 or less. The learning device according to claim 3.

5. The camera parameters further include the focal length of the camera, The focal length estimated by the deep neural network is in the range of 1 / 4 to 1 / 2 of the vertical length of the image sensor provided by the camera. A learning device according to any one of claims 1 to 4.

6. The aforementioned plurality of true values ​​are a plurality of first world coordinate points obtained by projecting the plurality of reference points onto the world coordinates using the inverse function and the true camera parameters, The aforementioned plurality of estimated values ​​are a plurality of second world coordinate points projected onto the world coordinates using the plurality of reference points and the estimated camera parameters as inverse functions. The learning unit calculates the network error based on the distance between each combination of the plurality of first world coordinate points and each of the plurality of second world coordinate points. A learning device according to any one of claims 1 to 4.

7. The plurality of true values ​​are a plurality of first unit line-of-sight vectors obtained by projecting the plurality of reference points onto the world coordinates using the inverse function and the true camera parameters. The aforementioned multiple estimates are multiple second unit line-of-sight vectors obtained by projecting the aforementioned multiple reference points onto the world coordinates using the inverse function and the estimated camera parameters. The learning unit calculates the network error based on the angle of each combination of each of the plurality of first unit line-of-sight vectors and each of the plurality of second unit line-of-sight vectors. A learning device according to any one of claims 1 to 4.

8. The plurality of true values ​​are a plurality of first unit line-of-sight vectors obtained by projecting the plurality of reference points onto the world coordinates using the inverse function and the true camera parameters. The aforementioned multiple estimates are multiple second unit line-of-sight vectors obtained by projecting the aforementioned multiple reference points onto the world coordinates using the inverse function and the estimated camera parameters. The learning unit calculates the network error based on the distances between each of the plurality of first unit line-of-sight vectors and the plurality of first intersection points with the unit sphere, and each of the plurality of second unit line-of-sight vectors and the plurality of second intersection points with the unit sphere. A learning device according to any one of claims 1 to 4.

9. A learning method in computers, The camera captures an image, Obtain the true camera parameters of the aforementioned camera, A deep neural network is trained using deep learning with the acquired image and the acquired true camera parameters. The trained deep neural network is output, In the training of the deep neural network described above, By inputting the aforementioned image into the deep neural network, camera parameters are estimated, including a distortion parameter representing the lens distortion assuming optical axis symmetry with respect to the angle of incidence to the camera using a projection function of a fourth degree or lower, and a posture parameter representing the orientation of the camera. The inverse function of the aforementioned projection function is calculated in closed form, Multiple true values ​​are calculated by projecting multiple reference points on the aforementioned image onto world coordinates using the inverse function and the true camera parameters. A plurality of estimated values ​​are calculated by projecting the plurality of reference points onto the world coordinates using the inverse function and the estimated camera parameters. Based on each combination of the multiple true values ​​and each of the multiple estimated values, a network error is calculated that represents the error between the true camera parameters and the estimates of the camera parameters by the deep neural network. The parameters of the deep neural network are learned to minimize the calculated network error. Learning methods.

10. An image acquisition unit that acquires images captured by a camera, A camera parameter acquisition unit that acquires the true camera parameters of the aforementioned camera, A learning unit that trains a deep neural network by deep learning using the image acquired by the image acquisition unit and the true camera parameters acquired by the camera parameter acquisition unit, The computer is made to function as an output unit that outputs the deep neural network learned by the learning unit. The aforementioned learning unit, By inputting the aforementioned image into the deep neural network, camera parameters are estimated, including a distortion parameter representing the lens distortion assuming optical axis symmetry with respect to the angle of incidence to the camera using a projection function of a fourth degree or lower, and a posture parameter representing the orientation of the camera. The inverse function of the aforementioned projection function is calculated in closed form, Multiple true values ​​are calculated by projecting multiple reference points on the aforementioned image onto world coordinates using the inverse function and the true camera parameters. A plurality of estimated values ​​are calculated by projecting the plurality of reference points onto the world coordinates using the inverse function and the estimated camera parameters. Based on each combination of the multiple true values ​​and each of the multiple estimated values, a network error is calculated that represents the error between the true camera parameters and the estimates of the camera parameters by the deep neural network. The parameters of the deep neural network are learned to minimize the calculated network error. Learning program.

11. An image acquisition unit that acquires images captured by a camera, An estimation unit estimates the camera parameters of the camera by inputting the image acquired by the image acquisition unit into a deep neural network trained by deep learning, An output unit that outputs the camera parameters estimated by the estimation unit, Equipped with, During the training of the deep neural network, Training images have been acquired. The true camera parameters of the camera that captured the aforementioned training images are obtained. When the training image is input to the deep neural network, camera parameters are estimated, including a distortion parameter representing the lens distortion assuming optical axis symmetry with respect to the angle of incidence to the camera that captured the training image, and a posture parameter representing the camera's orientation. The inverse function of the aforementioned projection function is calculated in closed form, Multiple true values ​​are calculated by projecting multiple reference points on the training image onto world coordinates using the inverse function and the true camera parameters. Multiple estimated values ​​are calculated by projecting the aforementioned multiple reference points onto the world coordinates using the inverse function and the estimated camera parameters. Based on each combination of the multiple true values ​​and each of the multiple estimated values, a network error is calculated that represents the error between the true camera parameters and the estimates of the camera parameters by the deep neural network. The parameters of the deep neural network are learned so that the calculated network error is minimized. Camera parameter calculation device.

12. A method for calculating camera parameters in a computer, The camera captures an image, The acquired image is input into a deep neural network trained by deep learning to estimate the camera parameters of the camera. Output the estimated camera parameters, During the training of the deep neural network, Training images have been acquired. The true camera parameters of the camera that captured the aforementioned training images are obtained. When the training image is input to the deep neural network, camera parameters are estimated, including a distortion parameter representing the lens distortion assuming optical axis symmetry with respect to the angle of incidence to the camera that captured the training image, and a posture parameter representing the camera's orientation. The inverse function of the aforementioned projection function is calculated in closed form, Multiple true values ​​are calculated by projecting multiple reference points on the training image onto world coordinates using the inverse function and the true camera parameters. Multiple estimated values ​​are calculated by projecting the aforementioned multiple reference points onto the world coordinates using the inverse function and the estimated camera parameters. Based on each combination of the multiple true values ​​and each of the multiple estimated values, a network error is calculated that represents the error between the true camera parameters and the estimates of the camera parameters by the deep neural network. The parameters of the deep neural network are learned so that the calculated network error is minimized. Camera parameter calculation method.

13. An image acquisition unit that acquires images captured by a camera, An estimation unit estimates the camera parameters of the camera by inputting the image acquired by the image acquisition unit into a deep neural network trained by deep learning, The computer functions as an output unit that outputs the camera parameters estimated by the estimation unit. During the training of the deep neural network, Training images have been acquired. The true camera parameters of the camera that captured the aforementioned training images are obtained. When the training image is input to the deep neural network, camera parameters are estimated, including a distortion parameter representing the lens distortion assuming optical axis symmetry with respect to the angle of incidence to the camera that captured the training image, and a posture parameter representing the camera's orientation. The inverse function of the aforementioned projection function is calculated in closed form, Multiple true values ​​are calculated by projecting multiple reference points on the training image onto world coordinates using the inverse function and the true camera parameters. Multiple estimated values ​​are calculated by projecting the aforementioned multiple reference points onto the world coordinates using the inverse function and the estimated camera parameters. Based on each combination of the multiple true values ​​and each of the multiple estimated values, a network error is calculated that represents the error between the true camera parameters and the estimates of the camera parameters by the deep neural network. The parameters of the deep neural network are learned so that the calculated network error is minimized. Camera parameter calculation program.

Images