Learning device, learning method, learning program, camera parameter calculation device, camera parameter calculation method, and camera parameter calculation program

Abstract

A learning unit of this learning device: learns a deep neural network by a deep learning using both an image captured by a camera causing the occurrence of a distortion and the acquired real coordinates of a plurality of vanishing points; inputting the image to the deep neural network, thereby estimating the coordinates of the plurality of vanishing points for calculating the tilt angle, pan angle and roll angle of the camera; calculates network errors indicating the errors of the tilt angle, pan angle and roll angle on the basis of both the real coordinates of the plurality of vanishing points and the estimated coordinates of the plurality of vanishing points; and learns parameters of the deep neural network such that the calculated network errors are minimized.

Description

[Technical Field]

[0001] This disclosure relates to a technique for training a deep neural network to calculate camera parameters from images, and a technique for calculating camera parameters from images. [Background technology]

[0002] To calibrate cameras such as sensing cameras, geometry-based methods require associating 3D coordinate values ​​in 3D space with pixel positions in a 2D image. Conventionally, a repeating pattern with a known shape has been photographed, and the correspondence between 3D coordinates and pixel positions in the 2D image has been achieved by detecting intersections or the centers of circles from the obtained images.

[0003] Furthermore, deep learning-based methods have been proposed to perform robust camera calibration using a single input image, either for image brightness or for adjusting the subject. Camera calibration, in this context, refers to the calculation of camera parameters.

[0004] For example, in Non-Patent Document 1, camera parameters are calculated using a geometry-based method that associates 3D coordinate values ​​in 3D space with pixel positions in a 2D image using calibration indices.

[0005] Furthermore, for example, Non-Patent Document 2 discloses a deep learning-based method for performing camera calibration from a single image, based on the Manhattan World Assumption.

[0006] The method described in Non-Patent Document 1 requires the process of photographing a repeating pattern with a known shape, detecting intersections or the center of a circle from the obtained image, and mapping 3D coordinates to pixel positions in the 2D image. Therefore, the camera calibration process is complex and may not be easily performed.

[0007] Furthermore, in the method described in Non-Patent Document 2, the camera's orientation is estimated based on vanishing points, which are the intersection points of multiple lines obtained by line detection. Therefore, it is difficult to apply this method to camera calibration for cameras where the horizon is elliptical, such as fisheye cameras. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] RYTsai, “A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses,” IEEE Journal of Robotics and Automation, Volume 3, Number 4, pages 323-344, August 1987. [Non-Patent Document 2] J.Lee, M.Sung, H.Lee, and J.Kim, "Neural Geometric Parser for Single Image Camera Calibration", European Conference on Computer Vision, Volume 12357, pages 541-557, 2020 [Overview of the project]

[0009] This disclosure was made to solve the above-mentioned problems and aims to provide a technology that can calculate camera parameters with high accuracy from a single distorted image.

[0010] The learning device according to this disclosure comprises: an image acquisition unit that acquires an image captured by a camera that causes distortion; a vanishing point acquisition unit that acquires the coordinates of a plurality of true vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera; a learning unit that learns a deep neural network by deep learning using the image acquired by the image acquisition unit and the coordinates of the plurality of true vanishing points acquired by the vanishing point acquisition unit; and an output unit that outputs the deep neural network learned by the learning unit. The learning unit inputs the image to the deep neural network to estimate the coordinates of a plurality of vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera, calculates a network error indicating the error in the tilt angle, pan angle, and roll angle based on the coordinates of the plurality of true vanishing points and the estimated coordinates of the plurality of vanishing points, and learns the parameters of the deep neural network to minimize the calculated network error.

[0011] According to this disclosure, camera parameters can be calculated with high accuracy from a single distorted image. [Brief explanation of the drawing]

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

[0013] (Knowledge that forms the basis of this disclosure) In recent years, camera-based sensing has been implemented, but camera calibration is necessary for high-precision image recognition. However, in camera calibration of cameras with large lens distortion, such as fisheye cameras, conventional deep learning-based camera calibration has difficulty calculating camera parameters with high accuracy from a single distorted image.

[0014] To address the above challenges, the following technologies are disclosed.

[0015] (1) A learning device according to one aspect of the present disclosure includes: an image acquisition unit that acquires an image taken by a camera that causes distortion; a vanishing point acquisition unit that acquires the coordinates of a plurality of true vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera; a learning unit that learns a deep neural network by deep learning using the image acquired by the image acquisition unit and the coordinates of the plurality of true vanishing points acquired by the vanishing point acquisition unit; and an output unit that outputs the deep neural network learned by the learning unit, wherein the learning unit inputs the image to the deep neural network to estimate the coordinates of a plurality of vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera, calculates a network error indicating the error in the tilt angle, pan angle, and roll angle based on the coordinates of the plurality of true vanishing points and the estimated coordinates of the plurality of vanishing points, and learns the parameters of the deep neural network to minimize the calculated network error.

[0016] In this configuration, a single distorted image is input to a deep neural network trained by deep learning, and the coordinates of multiple vanishing points for calculating the camera's tilt, pan, and roll angles are estimated. Then, from the estimated coordinates of the multiple vanishing points, the camera's tilt, pan, and roll angles, which are part of the camera parameters, can be calculated. Therefore, camera parameters can be calculated with high accuracy from a single distorted image.

[0017] (2) In the learning device described in (1) above, the plurality of vanishing points may include a first vanishing point in the direction of the camera's movement on the image, a second vanishing point in the direction of the camera's zenith, a third vanishing point to the right of the camera, and a fourth vanishing point to the left of the camera.

[0018] With this configuration, if the estimated coordinates of the first, second, third, and fourth vanishing points and the true coordinates of the first, second, third, and fourth vanishing points are obtained, the network error indicating the error in the camera's tilt angle, pan angle, and roll angle can be calculated with high accuracy.

[0019] (3) In the learning device described in (2) above, the learning unit may calculate a first distance between the perpendicular bisector of the line segment connecting the true third vanishing point and the true fourth vanishing point and a straight line parallel to the perpendicular bisector and passing through the estimated first vanishing point; calculate a second distance between the perpendicular bisector and a straight line parallel to the perpendicular bisector and passing through the estimated second vanishing point; calculate a third distance between the true first vanishing point and the estimated first vanishing point in the direction along the perpendicular bisector; calculate a fourth distance between the true second vanishing point and the estimated second vanishing point in the direction along the perpendicular bisector; calculate the angle between the line segment connecting the true third vanishing point and the true fourth vanishing point and the line segment connecting the estimated third vanishing point and the estimated fourth vanishing point; and calculate the network error as the sum of the first distance, second distance, third distance, fourth distance and the angle.

[0020] According to this configuration, the first distance between the perpendicular bisector of the line segment connecting the true third vanishing point and the true fourth vanishing point, and the line parallel to the perpendicular bisector and passing through the estimated first vanishing point, corresponds to the pan angle error. Similarly, the second distance between the perpendicular bisector and the line parallel to the perpendicular bisector and passing through the estimated second vanishing point also corresponds to the pan angle error. Furthermore, the third distance between the true first vanishing point and the estimated first vanishing point in the direction along the perpendicular bisector corresponds to the tilt angle error. Similarly, the fourth distance between the true second vanishing point and the estimated second vanishing point in the direction along the perpendicular bisector also corresponds to the tilt angle error. Finally, the angle between the line segment connecting the true third vanishing point and the true fourth vanishing point, and the line segment connecting the estimated third vanishing point and the estimated fourth vanishing point, corresponds to the roll angle error.

[0021] Therefore, the sum of the first distance, second distance, third distance, fourth distance, and angle is calculated as the network error, and the parameters of the deep neural network are learned to minimize the network error, thereby enabling the high-precision calculation of camera tilt angle, pan angle, and roll angle, which are part of the camera parameters.

[0022] Furthermore, this disclosure can be implemented not only as a learning device having the characteristic configuration described above, but also as a learning method that performs characteristic processing corresponding to the characteristic configuration of the learning device. It can also be implemented as a computer program that causes a computer to execute the characteristic processing included in such a learning method. Therefore, the same effects as the learning device described above can be achieved in the following other embodiments.

[0023] (4) A learning method relating to another aspect of the present disclosure is a learning method in a computer, comprising: acquiring an image taken by a camera that causes distortion; acquiring coordinates of a plurality of true vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera; training a deep neural network by deep learning using the acquired image and the acquired coordinates of the plurality of true vanishing points; outputting the trained deep neural network; in training the deep neural network, inputting the image into the deep neural network to estimate the coordinates of a plurality of vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera; calculating a network error indicating the error in the tilt angle, pan angle, and roll angle based on the coordinates of the plurality of true vanishing points and the estimated coordinates of the plurality of vanishing points; and training the parameters of the deep neural network to minimize the calculated network error.

[0024] (5) A learning program according to another aspect of the present disclosure includes an image acquisition unit that acquires an image taken by a camera that causes distortion, a vanishing point acquisition unit that acquires coordinates of a plurality of true vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera, a learning unit that learns a deep neural network by deep learning using the image acquired by the image acquisition unit and the coordinates of the plurality of true vanishing points acquired by the vanishing point acquisition unit, and a computer that functions as an output unit that outputs the deep neural network learned by the learning unit, wherein the learning unit inputs the image into the deep neural network to estimate the coordinates of a plurality of vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera, calculates a network error indicating the error in the tilt angle, pan angle, and roll angle based on the coordinates of the plurality of true vanishing points and the estimated coordinates of the plurality of vanishing points, and learns the parameters of the deep neural network to minimize the calculated network error.

[0025] (6) A camera parameter calculation device according to another aspect of the present disclosure includes: an image acquisition unit that acquires an image taken by a camera that causes distortion; an estimation unit that estimates the coordinates of a plurality of vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera by inputting the image acquired by the image acquisition unit into a deep neural network trained by deep learning; a calculation unit that calculates the tilt angle, pan angle, and roll angle based on the coordinates of the plurality of vanishing points estimated by the estimation unit; and an output unit that outputs camera parameters including the tilt angle, pan angle, and roll angle calculated by the calculation unit, wherein the deep neural network During training, training images are acquired, and the coordinates of multiple true vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera that captured the training images are acquired. When the training images are input to the deep neural network, the coordinates of multiple vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera that captured the training images are estimated. Based on the coordinates of the multiple true vanishing points and the estimated coordinates of the multiple vanishing points, a network error indicating the error in the tilt angle, pan angle, and roll angle is calculated, and the parameters of the deep neural network are learned to minimize the calculated network error.

[0026] In this configuration, a single distorted image is input to a deep neural network trained by deep learning, and the coordinates of multiple vanishing points for calculating the camera's tilt, pan, and roll angles are estimated. Then, from the estimated coordinates of the multiple vanishing points, the camera's tilt, pan, and roll angles, which are part of the camera parameters, can be calculated. Therefore, camera parameters can be calculated with high accuracy from a single distorted image.

[0027] (7) In the camera parameter calculation device described in (6) above, the plurality of vanishing points may include a first vanishing point in the direction of camera movement on the image, a third vanishing point to the right of the camera, and a fourth vanishing point to the left of the camera.

[0028] With this configuration, if the coordinates of the estimated first, third, and fourth vanishing points are obtained, the tilt angle, pan angle, and roll angle of the camera can be calculated with high accuracy.

[0029] (8) In the camera parameter calculation device described in (7) above, the calculation unit may calculate the roll angle using the coordinates of the first vanishing point and the coordinates of the midpoint of the line segment connecting the third vanishing point and the fourth vanishing point; calculate the tilt angle using the y coordinate of the first vanishing point, the y coordinate of the midpoint of the line segment connecting the third vanishing point and the fourth vanishing point and the inverse function of the projection function of the camera; and calculate the pan angle using the x coordinate of the principal point image coordinates of the camera, the x coordinate of the midpoint of the line segment connecting the third vanishing point and the fourth vanishing point and the inverse function of the projection function.

[0030] In this configuration, the roll angle is calculated using the coordinates of the first vanishing point and the coordinates of the midpoint of the line segment connecting the third and fourth vanishing points. The tilt angle is calculated using the y-coordinate of the first vanishing point, the y-coordinate of the midpoint of the line segment connecting the third and fourth vanishing points, and the inverse function of the camera's projection function. The pan angle is calculated using the x-coordinate of the camera's principal point image coordinates, the x-coordinate of the midpoint of the line segment connecting the third and fourth vanishing points, and the inverse function of the projection function. Therefore, by estimating the coordinates of the first, third, and fourth vanishing points, the camera's tilt angle, pan angle, and roll angle can be calculated.

[0031] Furthermore, this disclosure can be implemented not only as a camera parameter calculation device having the characteristic configuration described above, but also as a camera parameter calculation method that performs characteristic processing corresponding to the characteristic configuration of the camera parameter calculation device. It can also be implemented as a computer program that causes a computer to execute the characteristic processing included in such a camera parameter calculation method. Therefore, the same effects as the above-described camera parameter calculation device can be achieved in the following other embodiments.

[0032] (9) A camera parameter calculation method in another aspect of the present disclosure is a computer camera parameter calculation method comprising: acquiring an image taken by a camera that causes distortion; inputting the acquired image into a deep neural network trained by deep learning to estimate the coordinates of a plurality of vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera; calculating the tilt angle, pan angle, and roll angle based on the estimated coordinates of the plurality of vanishing points; outputting camera parameters including the calculated tilt angle, pan angle, and roll angle; and during the training of the deep neural network, training image The system acquires the coordinates of multiple true vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera that captured the training image. The training image is input to the deep neural network, which estimates the coordinates of multiple vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera that captured the training image. Based on the coordinates of the multiple true vanishing points and the estimated coordinates of the multiple vanishing points, a network error indicating the error in the tilt angle, pan angle, and roll angle is calculated, and the parameters of the deep neural network are learned to minimize the calculated network error.

[0033] (10) A camera parameter calculation program according to another aspect of the present disclosure includes an image acquisition unit that acquires an image taken by a camera that causes distortion; an estimation unit that estimates the coordinates of a plurality of vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera by inputting the image acquired by the image acquisition unit into a deep neural network trained by deep learning; a calculation unit that calculates the tilt angle, pan angle, and roll angle based on the coordinates of the plurality of vanishing points estimated by the estimation unit; and an output unit that outputs camera parameters including the tilt angle, pan angle, and roll angle calculated by the calculation unit, with the computer functioning as the deep neural network. During the training of the deep neural network, training images are acquired, and the coordinates of multiple true vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera that captured the training images are acquired. When the training images are input to the deep neural network, the coordinates of multiple vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera that captured the training images are estimated. Based on the coordinates of the multiple true vanishing points and the estimated coordinates of the multiple vanishing points, a network error indicating the error in the tilt angle, pan angle, and roll angle is calculated, and the parameters of the deep neural network are learned to minimize the calculated network error.

[0034] Furthermore, this disclosure allows computer programs to be distributed via computer-readable non-temporary recording media such as CD-ROMs or communication networks such as the Internet. Therefore, the same effects as the learning device or camera parameter calculation device described above can be achieved in the following other embodiments.

[0035] (11) A computer-readable non-temporary recording medium recording a learning program according to another aspect of the present disclosure includes an image acquisition unit that acquires an image taken by a camera that causes distortion, a vanishing point acquisition unit that acquires coordinates of a plurality of true vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera, a learning unit that learns a deep neural network by deep learning using the image acquired by the image acquisition unit and the coordinates of the plurality of true vanishing points acquired by the vanishing point acquisition unit, and a computer that functions as an output unit that outputs the deep neural network learned by the learning unit, wherein the learning unit inputs the image into the deep neural network to estimate the coordinates of a plurality of vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera, calculates a network error indicating the error in the tilt angle, pan angle, and roll angle based on the coordinates of the plurality of true vanishing points and the estimated coordinates of the plurality of vanishing points, and learns the parameters of the deep neural network to minimize the calculated network error.

[0036] (12) A computer-readable non-temporary recording medium recording a camera parameter calculation program according to another aspect of the present disclosure includes: an image acquisition unit that acquires an image taken by a camera that causes distortion; an estimation unit that estimates the coordinates of a plurality of vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera by inputting the image acquired by the image acquisition unit into a deep neural network trained by deep learning; a calculation unit that calculates the tilt angle, pan angle, and roll angle based on the coordinates of the plurality of vanishing points estimated by the estimation unit; and an output unit that outputs camera parameters including the tilt angle, pan angle, and roll angle calculated by the calculation unit, with a computer as the output unit. During the training of the deep neural network, training images are acquired, and the coordinates of multiple true vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera that captured the training images are acquired. When the training images are input to the deep neural network, the coordinates of multiple vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera that captured the training images are estimated. Based on the coordinates of the multiple true vanishing points and the estimated coordinates of the multiple vanishing points, a network error indicating the error in the tilt angle, pan angle, and roll angle is calculated, and the parameters of the deep neural network are learned to minimize the calculated network error.

[0037] Embodiments of this disclosure will be described below with reference to the attached drawings. Note that the embodiments described below are all specific examples of this disclosure. The numerical values, shapes, components, steps, and order of steps shown in the following embodiments are examples only and are not intended to limit this disclosure. Furthermore, components in the following embodiments that are not described in the independent claim representing the highest-level concept will be described as optional components. Also, in all embodiments, the contents of each can be combined.

[0038] (Embodiment) The embodiments of this disclosure will be described below with reference to the drawings.

[0039] Figure 1 is a block diagram showing an example of the configuration of a camera parameter calculation system in an embodiment of the present disclosure.

[0040] The camera parameter calculation system comprises a camera parameter calculation device 1 and a camera 4.

[0041] In this embodiment, camera 4 is, for example, a fixed camera installed on a vehicle. Camera 4 captures images of the area around the vehicle at a predetermined frame rate and inputs the captured images to the camera parameter calculation device 1 at a predetermined frame rate. Camera 4 is, for example, a fisheye camera (ultra-wide-angle camera) with a field of view of 180° or more. Alternatively, camera 4 may be a wide-angle camera with a field of view of 60° or more.

[0042] The camera parameter calculation device 1 consists of a computer including a processor 2, memory 3, and interface circuitry (not shown in the diagram). The processor 2 is, for example, a central processing unit. The memory 3 is, for example, a non-volatile, rewritable storage device such as flash memory, a hard disk drive, or a solid-state drive. The interface circuitry is, for example, a communication circuit.

[0043] The camera parameter calculation device 1 may be configured as an edge server installed in the vehicle, or as a cloud server. When the camera parameter calculation device 1 is configured as an edge server, the camera 4 and the camera parameter calculation device 1 are connected via a local area network. When the camera parameter calculation device 1 is configured as a cloud server, the camera 4 and the camera parameter calculation device 1 are connected via a wide-area communication network such as the internet. Furthermore, a portion of the camera parameter calculation device 1's configuration may be provided on the edge side, with the remainder provided on the cloud side.

[0044] The processor 2 includes an acquisition unit 21, a vanishing point estimation unit 22, a camera parameter calculation unit 23, and an output unit 24. The acquisition unit 21 to the output unit 24 may be implemented by a central processing unit executing a camera parameter calculation program, or they may be composed of dedicated hardware circuits such as an ASIC (Application Specific Integrated Circuit).

[0045] The acquisition unit 21 acquires images captured by the camera 4 that causes distortion. The acquisition unit 21 stores the acquired images in the frame memory 31.

[0046] The vanishing point estimation unit 22 inputs the image acquired by the acquisition unit 21 into a deep neural network (hereinafter also called DNN) trained by deep learning, thereby estimating the coordinates of multiple vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera 4. The vanishing point estimation unit 22 reads the DNN from the DNN memory unit 32. The vanishing point estimation unit 22 estimates the coordinates of multiple vanishing points on the image from the image read from the frame memory 31 using the DNN trained by deep learning. An example of a DNN is a convolutional neural network including convolutional layers and pooling layers.

[0047] During DNN training, training images are acquired. Next, the coordinates of multiple true vanishing points are obtained to calculate the tilt, pan, and roll angles of the camera that captured the training images. Then, the training images are input to the DNN, and the coordinates of multiple vanishing points for calculating the tilt, pan, and roll angles of the camera that captured the training images are estimated. Next, based on the coordinates of the multiple true vanishing points and the estimated coordinates of the multiple vanishing points, a network error representing the error in the tilt, pan, and roll angles is calculated. Finally, the parameters of the DNN are trained to minimize the calculated network error.

[0048] Note that the true vanishing point is the vanishing point of the correct answer.

[0049] The multiple vanishing points include a first vanishing point in the direction of camera 4's movement, a third vanishing point to the right of camera 4, and a fourth vanishing point to the left of camera 4.

[0050] The camera parameter calculation unit 23 calculates the tilt angle, pan angle, and roll angle based on the coordinates of multiple vanishing points estimated by the vanishing point estimation unit 22. The tilt angle, pan angle, and roll angle represent the attitude of the camera 4 and are part of the camera parameters.

[0051] The camera parameter calculation unit 23 calculates the roll angle using the first vanishing point and the midpoint of the line segment connecting the third and fourth vanishing points. The camera parameter calculation unit 23 calculates the tilt angle using the y-coordinate of the first vanishing point, the y-coordinate of the midpoint of the line segment connecting the third and fourth vanishing points, and the inverse function of the projection function of camera 4. The camera parameter calculation unit 23 calculates the pan angle using the x-coordinate of the principal point image coordinates of camera 4, the x-coordinate of the midpoint of the line segment connecting the third and fourth vanishing points, and the inverse function of the projection function.

[0052] The output unit 24 outputs camera parameters, including the tilt angle, pan angle, and roll angle, which are calculated by the camera parameter calculation unit 23.

[0053] Memory 3 includes frame memory 31 and DNN storage unit 32.

[0054] The frame memory 31 stores images acquired by the acquisition unit 21 from the camera 4. The frame memory 31 stores time-series images acquired by the acquisition unit 21.

[0055] The DNN storage unit 32 pre-stores the DNN to be used by the vanishing point estimation unit 22. The DNN storage unit 32 also stores the DNN generated by the learning device 5, which will be described later. The DNN may be stored in the DNN storage unit 32 when the camera parameter calculation device 1 is manufactured, or it may be received from an external server and stored in the DNN storage unit 32.

[0056] The camera parameter calculation device 1 does not necessarily have to be implemented by a single computer device, but may be implemented by a distributed processing system (not shown) including a terminal device and a server. For example, the acquisition unit 21 and frame memory 31 may be provided in the terminal device, and the DNN storage unit 32, vanishing point estimation unit 22, camera parameter calculation unit 23, and output unit 24 may be provided in the server. In this case, data is exchanged between the components via a communication line connected to the terminal device and the server.

[0057] Figure 2 is a flowchart showing an example of the camera parameter calculation process by the camera parameter calculation device 1 of the embodiment of this disclosure. The operation of the camera parameter calculation device 1 will be described below in accordance with Figure 2. The camera parameter calculation process is performed when the camera 4 is installed, and thereafter is performed periodically, for example, every week or every month.

[0058] First, in step S1, the acquisition unit 21 acquires an image (fisheye image) captured by the camera 4. The acquisition unit 21 stores the acquired image in the frame memory 31.

[0059] Next, in step S2, the vanishing point estimation unit 22 reads an image from the frame memory 31 and inputs the read image into a pre-trained DNN to estimate the coordinates of multiple vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera 4. The DNN training method will be described later.

[0060] Next, in step S3, the camera parameter calculation unit 23 calculates the tilt angle, pan angle, and roll angle of the camera 4 based on the coordinates of the multiple vanishing points estimated by the vanishing point estimation unit 22.

[0061] Next, in step S4, the output unit 24 outputs camera parameters including the tilt angle, pan angle, and roll angle calculated by the camera parameter calculation unit 23.

[0062] Thus, when one distorted image learned by deep learning is input into the DNN, the coordinates of a plurality of vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera 4 are estimated. Then, from the coordinates of the plurality of estimated vanishing points, the tilt angle, pan angle, and roll angle of the camera 4, which are part of the camera parameters, can be calculated. Therefore, the camera parameters can be calculated with high accuracy from one distorted image.

[0063] Subsequently, an example of the camera parameters in the present disclosure will be described below. The conversion formula from the world coordinate system to the image coordinate system is represented by the following mathematical formulas (1) to (4). The camera parameters are the parameters of the projection formula that projects the world coordinates onto the image coordinates. Γ(η) in the mathematical formula (3) is a projection function representing the lens distortion with respect to the incident angle η. Details of the projection function will be described later. Note that η is the incident angle.

[0064] [Number]

[0065] Here, (X, Y, Z) are world coordinate values, and (x, y) are image coordinate values. (C x , C y ) are the principal point image coordinates of the camera, r 11 to r 33 are the components of the 3x3 rotation matrix R representing the rotation with respect to the reference of the world coordinates, (T X , T Y , T Z ) is the translation vector with respect to the reference of the world coordinates, d x and d y are the pixel pitches in the horizontal and vertical directions of the image sensor of the camera. In the mathematical formulas (1) to (4), d x , d y , C x , C y , r 11 to r33, T X , T Y , T ZThese are the camera parameters. Camera parameters consist of external parameters related to the camera's orientation (rotation and translation relative to the world coordinate system) and internal parameters related to focal length and lens distortion.

[0066] Formulas (1) to (4) represent the transformation from (X,Y,Z) to (x,y). When transforming from (x,y) to (X,Y,Z) on the unit sphere, the inverse function or inverse matrix of formulas (1) to (4) is used.

[0067] An example of the γ (projection function) of an optically axially symmetric lens is expressed as a function of the angle of incidence η by the following equations (5) to (9).

[0068] γ = fsin(η)···(5) γ = 2fsin(η / 2)···(6) γ = fη···(7) γ = 2ftan(η / 2)···(8) γ = ftan(η)···(9)

[0069] Equation (5) represents the projection function of orthogonal projection, equation (6) represents the projection function of equisolid angle projection, equation (7) represents the projection function of equidistant projection, equation (8) represents the projection function of stereoscopic projection, and equation (9) represents the projection function of a pinhole camera. f represents the focal length, and η represents the angle of incidence.

[0070] Furthermore, a general camera model, which is an N-th degree polynomial, can be expressed by the following equation (10).

[0071] γ = k1η + k2η 3 +k2η 5 +··· ···(10)

[0072] In the above equation (10), η represents the angle of incidence, and k1 and k2 represent the distortion parameters (distortion coefficients), which are one of the camera parameters.

[0073] To simplify the explanation, the following will describe the case where the projection function is the projection function of the orthogonal projection in equation (5), and four camera parameters are estimated: tilt angle θ, pan angle φ, roll angle ψ, and focal length f. Note that the tilt angle θ, pan angle φ, and roll angle ψ are components r of the rotation matrix in equation (2). 11 ~r 33 This is expressed in terms of angles.

[0074] Figure 3 is a schematic diagram illustrating the world coordinates in the Manhattan World Hypothesis. In Figure 3, vehicle 8 is viewed from above.

[0075] The Manhattan World hypothesis is a world coordinate system in which buildings 81 and roads 82 exist in a grid pattern, the X and Y axes of the XYZ-O space are parallel to the outer walls of the rectangular buildings 81, and the positive direction of the Z axis points towards the sky. In this embodiment, it is assumed that a camera 4 is mounted on a vehicle 8 traveling along a road 82 in this Manhattan World hypothesis. The direction of travel of the vehicle 8, indicated by arrow 83, is the positive direction of the Y axis.

[0076] As shown in Figure 3, the rotation angles of camera 4 in the XYZ-O coordinate system with respect to the optical axis of camera 4 are defined as the tilt angle θ, pan angle φ, and roll angle ψ. The focal length f represents the scale of the image. Therefore, by using a DNN, the focal length f can be directly estimated from a single fisheye image. Accordingly, the method for calculating the rotation angle of camera 4 from a single fisheye image will be described below. Note that the coordinate system and rotation angles will be explained in right-handed terms.

[0077] Figures 4 to 7 show examples of various fisheye images and examples of the horizon line that appears in each fisheye image.

[0078] Figure 4 shows an example of the first fisheye image 41 in this embodiment, Figure 5 shows an example of the second fisheye image 42 in this embodiment, Figure 6 shows an example of the third fisheye image 43 in this embodiment, and Figure 7 shows an example of the fourth fisheye image 44 in this embodiment.

[0079] In a fisheye image, the horizon is an ellipse or an elliptical arc. The horizon is an ellipse only when the fisheye camera's projection method is orthogonal projection; when the projection method is anything other than orthogonal projection, the horizon is close to an ellipse. Hereafter, even when the projection method is anything other than orthogonal projection, the horizon will be treated as an ellipse for explanation. In a fisheye image, the horizon is the position obtained by projecting infinity on the XY plane onto the image. Ellipse E1 shown in Figures 4 to 7 represents the horizon in a fisheye image.

[0080] Furthermore, as shown in Figures 4 to 7, each point on the ellipse E1 is defined. First vanishing point V fro nt This is a coordinate point in the direction of camera 4's movement, corresponding to infinity in the positive Y-axis direction. Second vanishing point V zenith This is the coordinate point at the zenith of camera 4, corresponding to infinity in the positive Z-axis direction. Third vanishing point V right This is a coordinate point located to the right of camera 4, corresponding to infinity in the positive X-axis direction. Note that this is the third vanishing point V. right This point is located to the right of the direction of camera 4's movement and is the intersection of the horizontal line (ellipse E1) and the X-axis. Fourth vanishing point V lef t This is a coordinate point located to the left of camera 4, corresponding to infinity in the negative direction of the X-axis. Note that this is the fourth vanishing point V. left This point is located to the left of the direction of camera 4's movement and is the intersection of the horizontal line (ellipse E1) and the X-axis. Fifth vanishing point V back Point V is a coordinate point located in the opposite direction to the camera 4's direction of travel, and corresponds to infinity in the negative direction of the Y-axis. cross This is the third vanishing point V r ight and the fourth vanishing point V left It is the midpoint of the line segment connecting and the major axis L of the ellipse E1. long and the minor axis L short It is the intersection point with [the other line].

[0081] 1st vanishing point V front , third vanishing point V right , the fourth vanishing point V left and the fifth vanishing point V back It lies on the ellipse E1. First vanishing point V front and the fifth vanishing point Vback The line segment connecting these points represents the minor axis L of the ellipse E1. short Therefore, the third vanishing point is V. right and the fourth vanishing point V left The line segment connecting these points represents the major axis L of the ellipse E1. long Therefore, the second vanishing point is V. zen ith This is the first vanishing point V front and point V cross It lies on a straight line that includes [the specified location].

[0082] The tilt angle of the camera that acquired the fisheye images in Figures 4 and 6 is negative, while the tilt angle of the camera that acquired the fisheye images in Figures 5 and 7 is positive.

[0083] The vanishing point estimation unit 22 determines the first vanishing point V front , third vanishing point V right and the fourth vanishing point V left Estimate the coordinates.

[0084] Next, we will explain the relationship between each point defined above and the camera's rotation angle.

[0085] First, we will explain how to calculate the roll angle ψ, referring to Figure 8.

[0086] Figure 8 is a schematic diagram illustrating the method for calculating the roll angle ψ. When the first fisheye image 41 is rotated according to the roll angle ψ, the ellipse E1 Short axis L short The coordinate system becomes parallel to the y-axis, and the first vanishing point V front Point V cross (V front <V cross The first fisheye image 41 on the left side of Figure 8 shows the state before rotation, and the first fisheye image 41' on the right side of Figure 8 shows the state after rotation. That is, the roll angle ψ is the minor axis L of the ellipse E1. short This is the angle between the plane of the paper and the direction perpendicular to the image's y-axis, and is expressed by the following formula (11).

[0087]

number

[0088] In the above formula (11), e y is the unit vector in the y-axis direction in the image coordinate system. The image coordinate system is a coordinate system where the top left corner of the image is the origin.<a,b> This represents the dot product of two vectors.

[0089] The camera parameter calculation unit 23 calculates the first vanishing point V front The coordinates of and the third vanishing point V ri ght and the fourth vanishing point V left The midpoint of the line segment connecting (point V) cross The roll angle ψ is calculated using the coordinates of ). The camera parameter calculation unit 23 calculates the roll angle ψ based on the above formula (11).

[0090] Next, we will explain how to calculate the tilt angle θ, referring to Figures 9 and 10.

[0091] Figure 9 is a schematic diagram illustrating the method for calculating the tilt angle θ using a first fisheye image 41 rotated according to the roll angle ψ, and Figure 10 is a schematic diagram illustrating the method for calculating the tilt angle θ using a second fisheye image 42 rotated according to the roll angle ψ.

[0092] The roll angle ψ has already been calculated based on equation (11), and the following explanation will use images rotated according to the roll angle ψ, as shown in Figures 9 and 10.

[0093] Depending on the sign of the tilt angle θ, point V cross and the first vanishing point V front The relative magnitudes of the y-coordinates in the image coordinate system change. In Figure 9, the tilt angle θ is negative, and in Figure 10, the tilt angle θ is positive. Focusing on Figure 9, when the tilt angle is -90° (camera 4 is pointing straight down), the horizon line becomes a circle. Conversely, when the tilt angle is 0° (camera 4 is pointing horizontally), the horizon line becomes a straight line in the image, and point V cross and the first vanishing point V fro nt They match.

[0094] Here, assuming that the projection function of the camera in equations (1) to (4) is expressed as r = Ω(η), let the inverse function of Ω be Ω -1 . η is the incident angle, and r is the image height (distance from the image principal point). Assuming that the maximum incident angle of the projection function is 90°, the half-length of the minor axis L short of the ellipse E1 is L short / 2, and L short / 2 = Ω(π / 2) holds. The image height at an incident angle of 90° is half of the minor axis length of the ellipse E1. When this equation is expressed in terms of the inverse function, π / 2 = Ω -1 (L short / 2) holds. For a general incident angle η, η = Ω -1 (|V front,y - V cross,y |) (the incident angle is 0° or more). The incident angle corresponding to the y-component of the image coordinate system after rotation according to the roll angle ψ corresponds to the absolute value of the tilt angle in the world coordinate system. In FIG. 10, when V front,y - V cross,y > 0, since the tilt angle is positive, the tilt angle θ is expressed by the following equation (12).

[0095] θ = sign(V front,y - V cross,y )Ω -1 (|V front,y - V cross,y |) ··· (12)

[0096] In equation (12), sign is the sign function, which returns the sign (1 or -1) of the argument. When the argument is 0, 0 is returned. V front,y is the y-coordinate in the image coordinate system of the first vanishing point V f ront , and V cross,y is the y-coordinate in the image coordinate system of the point V cross . Also, Ω is the projection function of the camera 4 and is known. The projection function is stored in advance in the memory 3.

[0097] The camera parameter calculation unit 23 calculates the y-coordinate of the first vanishing point V front and the third vanishing point Vr ight and the fourth vanishing point V left The midpoint of the line segment connecting (point V cross ), the y coordinate of, and the inverse function Ω of the projection function Ω of camera 4 -1 are used to calculate the tilt angle θ. The camera parameter calculation unit 23 calculates the tilt angle θ based on the above formula (12).

[0098] Next, the method for calculating the pan angle φ will be described while referring to FIG. 11.

[0099] FIG. 11 is a schematic diagram for explaining the method for calculating the pan angle φ.

[0100] Due to the pan angle φ, point V cross is shifted in the horizontal direction (x-axis direction) in the image coordinate system. Pay attention to the x coordinate C of the principal point of the image x and the x coordinate of point V cross . The pan angle φ in the Manhattan world hypothesis is the minimum absolute value angle formed by the X and Y coordinate axes, and -π / 4 ≤ φ ≤ π / 4.

[0101] That is, when a pan angle φ that does not satisfy -π / 4 ≤ φ ≤ π / 4 is calculated, the pan angle φ is selected according to the following procedure. The pan angle φ' that is not in the Manhattan world hypothesis can be expressed as 0 ≤ φ' < 2π. For the pan angle φ' in the range of 0 ≤ φ' ≤ π / 4 that is not in the Manhattan world hypothesis, the pan angle φ' can be used as the pan angle φ, so the pan angle φ' and the pan angle φ in the Manhattan world hypothesis coincide. On the other hand, for the pan angle φ' in the range of π / 4 < φ' < 2π that is not in the Manhattan world hypothesis, in the Manhattan world hypothesis, either π / 2, π, or 3π / 2 is subtracted from the pan angle φ', and a pan angle φ that satisfies -π / 4 ≤ φ ≤ π / 4 is selected. For example, when the pan angle φ' is 11π / 12, the pan angle φ is -π / 12 (= 11π / 12 - π). In this way, by subtracting either π / 2, π, or 3π / 2 from the pan angle φ', there always exists a pan angle φ that satisfies -π / 4 ≤ φ ≤ π / 4.

[0102] Similar to the explanation of the tilt angle θ above, let's assume that the projection function of camera 4 is expressed as r = Ω(η). If δ is the displacement in the x-axis direction from the principal point of the image, then δ = V cross,x -C x Therefore, η = Ω -1 (|δ|) V cross,x Point V cross This is the x-coordinate in the image coordinate system. From Figure 11, when δ > 0, the pan angle φ is positive, so the pan angle φ is expressed by the following formula (13).

[0103] φ = sign(V) cross,x -C x )Ω -1 (|V cross,x -C x |)···(13)

[0104] In formula (13), `sign` is the sign function and returns the sign of the argument (1 or -1). If the argument is 0, 0 is returned. x This is the x-coordinate of the principal image coordinates of camera 4. cross,x Point V cross This is the x-coordinate in the image coordinate system. Furthermore, Ω is the projection function of camera 4 and is known. The projection function is pre-stored in memory 3.

[0105] The camera parameter calculation unit 23 calculates the x-coordinate of the principal point image coordinates of camera 4 and the third vanishing point V r ight and the fourth vanishing point V left The midpoint of the line segment connecting (point V) cross The x-coordinate of ) and the inverse function Ω of the projection function Ω of camera 4. -1 The pan angle φ is calculated using the above formula (13). The camera parameter calculation unit 23 calculates the pan angle φ based on the above formula (13).

[0106] Next, we will explain the relationship between the ellipse and each vanishing point on the ellipse. As shown in Figures 4 to 7, each point is defined, but the point necessary to calculate the pan angle φ, tilt angle θ, and roll angle ψ is point V. cross and the first vanishing point V front These are the coordinate values ​​of the two points. Note that the first vanishing point Vfront , third vanishing point V right , the fourth vanishing point V left , the fifth vanishing point V ba ck and point V cross If the coordinates of 3 out of the 5 points are known, the ellipse E1 can be uniquely determined. Each point is on the major axis L long and the minor axis L short It is selected from and .

[0107] point V cross Since it is not a vanishing point, estimation using DNN is difficult. Therefore, from here on, the first vanishing point V front , third vanishing point V right , the fourth vanishing point V left and the fifth vanishing point V back This section explains the case where the coordinates of three or more of the four points can be estimated. Each point is located along the major axis L. long and the minor axis L short It is selected from and .

[0108] Also, the second vanishing point V zenith The fact that the coordinates of the vanishing point (V) are not essential for estimating the rotation angle is a key difference from perspective projection techniques in non-fisheye images. This is because, unlike a straight horizontal line, an ellipse can represent a plane in three-dimensional space. The degrees of freedom of an ellipse are one dimension higher than those of a straight line, and the second vanishing point V zenith The coordinates are not essential for estimating the rotation angle. First vanishing point V f ront , second vanishing point V zenith and the fifth vanishing point V back Since these three points lie on the same straight line, the second vanishing point V zenith By using this method, it is possible to stabilize the estimation. In the case of the third fisheye image 43 shown in Figure 6, multiple vanishing points exist outside the image, making estimation by DNN difficult, so this kind of stabilization is important. Second vanishing point V zenith This is the point where vertical lines (curves in fisheye images) of buildings and other structures converge, and it is easier to estimate using DNNs than other vanishing points.

[0109] Also, the fifth vanishing point V back This is the vanishing point behind the camera. Therefore, the fifth vanishing point V backThe fifth vanishing point V is either not present in the image or exists on the image circle. The image circle is the boundary between the projected circular area in the image and the unprojected area in the image. In the projection method of a fisheye camera capable of projecting up to an incident angle of 180 degrees, the area behind the camera is projected onto the image circle. Therefore, the fifth vanishing point V back This is difficult to estimate using DNNs.

[0110] In light of the above, from here on, the first vanishing point V front , second vanishing point V zenith , third vanishing point V right and the fourth vanishing point V left This section explains the case where the coordinates of the four points are estimated using a DNN. Note that the coordinates of points other than these four may also be estimated.

[0111] The learning device used to learn the DNN in the vanishing point estimation unit 22 will be described below.

[0112] Figure 12 is a block diagram showing an example of the configuration of the learning device 5 in an embodiment of this disclosure.

[0113] The learning device 5 consists of a computer including a processor 6, memory 7, and interface circuitry (not shown in the diagram). The processor 6 is, for example, a central processing unit. The memory 7 is, for example, a non-volatile, rewritable storage device such as flash memory, a hard disk drive, or a solid-state drive. The interface circuitry is, for example, a communication circuit.

[0114] The learning device 5 may consist of a cloud server or a personal computer.

[0115] The processor 6 includes an image acquisition unit 60, a vanishing point acquisition unit 61, a learning unit 62, and an output unit 63. The image acquisition unit 60 to the output unit 63 may be implemented by a central processing unit executing a learning program, or they may be composed of dedicated hardware circuits such as an ASIC.

[0116] Memory 7 includes a learning image storage unit 71, a vanishing point storage unit 72, and a DNN storage unit 73.

[0117] The learning image storage unit 71 pre-stores multiple learning images captured by a camera that generates distortion. These learning images are used when training the DNN. The camera used to obtain the learning images is the same as camera 4. The learning images are fisheye images, obtained by capturing them in advance with a fisheye camera. Alternatively, the learning images may be generated by performing computer graphics (CG) processing on a panoramic image using the camera parameters of the fisheye camera.

[0118] The vanishing point storage unit 72 pre-stores the coordinates of multiple true vanishing points for calculating the camera's tilt angle, pan angle, and roll angle. These multiple true vanishing points are used when training the DNN. These multiple true vanishing points are multiple vanishing points on the training image. The vanishing point storage unit 72 stores the multiple true vanishing points corresponding to the training image.

[0119] The image acquisition unit 60 acquires training images captured by a camera that causes distortion. The image acquisition unit 60 reads training images from the training image storage unit 71. In this embodiment, the image acquisition unit 60 acquires training images that have been stored in advance from the training image storage unit 71, but this disclosure is not limited to this. The image acquisition unit 60 may acquire training images from an external server. In this case, the image acquisition unit 60 may receive training images from an external server. The image acquisition unit 60 may also acquire training images from a camera connected to the learning device 5.

[0120] The vanishing point acquisition unit 61 acquires the coordinates of multiple true vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera. The vanishing point acquisition unit 61 reads multiple true vanishing points from the vanishing point storage unit 72 and further reads the coordinates associated with each vanishing point. In this embodiment, the vanishing point acquisition unit 61 acquires the coordinates of multiple true vanishing points that are stored in advance from the vanishing point storage unit 72, but this disclosure is not particularly limited to this. The vanishing point acquisition unit 61 may acquire the coordinates of multiple true vanishing points from an external server. In this case, the vanishing point acquisition unit 61 may receive the coordinates of multiple true vanishing points from an external server. Alternatively, the vanishing point acquisition unit 61 may acquire the coordinates of multiple true vanishing points input by an operator.

[0121] The learning unit 62 trains a deep neural network using deep learning with training images acquired by the image acquisition unit 60 and the coordinates of multiple true vanishing points acquired by the vanishing point acquisition unit 61.

[0122] The learning unit 62 inputs training images into the DNN to estimate the coordinates of multiple vanishing points for calculating the camera's tilt angle, pan angle, and roll angle. Based on the coordinates of the multiple true vanishing points and the estimated coordinates of the multiple vanishing points, the learning unit 62 calculates a network error that indicates the error in the tilt angle, pan angle, and roll angle.

[0123] Here, the multiple vanishing points include a first vanishing point in the direction of camera movement, a second vanishing point in the zenith direction of the camera, a third vanishing point to the right of the camera, and a fourth vanishing point to the left of the camera.

[0124] The learning unit 62 calculates a first distance between the perpendicular bisector of the line segment connecting the true third vanishing point and the true fourth vanishing point, and the line parallel to the perpendicular bisector that passes through the estimated first vanishing point. The learning unit 62 also calculates a second distance between the perpendicular bisector and the line parallel to the perpendicular bisector that passes through the estimated second vanishing point. The learning unit 62 also calculates a third distance between the true first vanishing point and the estimated first vanishing point in the direction along the perpendicular bisector. The learning unit 62 also calculates a fourth distance between the true second vanishing point and the estimated second vanishing point in the direction along the perpendicular bisector. The learning unit 62 also calculates the angle between the line segment connecting the true third vanishing point and the true fourth vanishing point, and the line segment connecting the estimated third vanishing point and the estimated fourth vanishing point. The learning unit 62 also calculates the network error as the sum of the calculated first distance, second distance, third distance, fourth distance, and angle.

[0125] The learning unit 62 learns the DNN parameters to minimize the calculated network error.

[0126] The output unit 63 outputs the DNN learned by the learning unit 62. The output unit 63 outputs the DNN to the DNN storage unit 73.

[0127] The DNN storage unit 73 stores the DNN learned by the learning unit 62. In this embodiment, the output unit 63 stores the DNN learned by the learning unit 62 in the DNN storage unit 73, but this disclosure is not limited to this. The output unit 63 may output the DNN learned by the learning unit 62 to an external server. In this case, the output unit 63 may transmit the DNN to an external server.

[0128] Next, the learning process performed by the learning device 5 will be explained with reference to the diagram.

[0129] Figure 13 is a flowchart showing an example of the learning process by the learning device 5 in the embodiment of this disclosure. The operation of the learning device 5 will be described below in accordance with Figure 13.

[0130] First, in step S11, the image acquisition unit 60 acquires training images to be used for training the DNN.

[0131] Next, in step S12, the vanishing point acquisition unit 61 acquires the coordinates of multiple true vanishing points. The multiple true vanishing points are the true first vanishing point V front , true second vanishing point V zenit h , true third vanishing point V right and the true fourth vanishing point V left That is the case.

[0132] Next, in step S13, the learning unit 62 learns a DNN using the training image and the coordinates of multiple true vanishing points (DNN learning process).

[0133] Here, we will explain the DNN training process in step S13 of Figure 13.

[0134] Figure 14 is a flowchart showing an example of the DNN learning process in step S13 of Figure 13. The operation of the learning unit 62 will be described below in accordance with Figure 14.

[0135] First, in step S21, the learning unit 62 inputs the training image into the DNN to estimate the coordinates of multiple vanishing points. The DNN extracts image features from the convolutional layer, etc., and finally outputs the estimated coordinates of multiple vanishing points. The estimated multiple vanishing points are: First vanishing point V' front , second vanishing point V' zenith , third vanishing point V' right and the fourth vanishing point V' left The learning unit 62 is the first vanishing point V'. front Coordinates of the second vanishing point V' zenith Coordinates of the third vanishing point V' right The coordinates and the fourth vanishing point V' left Estimate the coordinates.

[0136] Next, in step S22, the learning unit 62 determines the true third vanishing point V right and the true fourth vanishing point V leftThe perpendicular bisector of the line segment connecting and and the estimated first vanishing point V' which is parallel to the perpendicular bisector. front A straight line L passes through it. front The first distance ΔV between front,φ Calculate.

[0137] Next, in step S23, the learning unit 62 determines the true third vanishing point V right and the true fourth vanishing point V left The perpendicular bisector of the line segment connecting and , and the estimated second vanishing point V' which is parallel to the perpendicular bisector. zenith A straight line L passes through it. zenith The second distance ΔV between zenit h,φ Calculate.

[0138] Next, in step S24, the learning unit 62 determines the true third vanishing point V right and the true fourth vanishing point V left The true first vanishing point V in the direction along the perpendicular bisector of the line segment connecting and front The first vanishing point V' was estimated to be this. front The third distance ΔV between front,θ Calculate.

[0139] Next, in step S25, the learning unit 62 determines the true third vanishing point V right and the true fourth vanishing point V left The true second vanishing point V is in the direction along the perpendicular bisector of the line segment connecting and zenith The second vanishing point V' was estimated to be this. zenith The fourth distance ΔV between zenith ,θ Calculate.

[0140] Next, in step S26, the learning unit 62 determines the true third vanishing point V right and the true fourth vanishing point V left The line segment connecting and the estimated third vanishing point V' right The fourth vanishing point V' was estimated to be this. left Calculate the angle Δψ formed by the line segment connecting point A and point B.

[0141] Next, in step S27, the learning unit 62 calculates the first distance ΔV front,φ, second distance ΔV zenith,φ , third distance ΔV front,θ , fourth distance ΔV zeni th,θ The network error is calculated by adding the angle Δψ to the value obtained by adding the angle Δψ.

[0142] Next, in step S28, the learning unit 62 updates the DNN parameters by backpropagation using the calculated network error. Stochastic gradient descent or the like is used to optimize backpropagation.

[0143] Returning to Figure 13, in step S14, the learning unit 62 determines whether the DNN training is complete. For example, the learning unit 62 determines that the DNN training is complete if the number of DNN parameter updates exceeds a threshold, and determines that the DNN training is not complete if the number of DNN parameter updates is less than or equal to the threshold. The threshold is, for example, 10,000 times.

[0144] The learning unit 62 may determine that DNN training is complete if the network error is less than the threshold, and determine that DNN training is not complete if the network error is greater than or equal to the threshold.

[0145] If it is determined at this point that the DNN training is not complete (NO in step S14), the process returns to step S11. Then, in step S11, the image acquisition unit 60 acquires other training images.

[0146] On the other hand, if it is determined that the DNN training is complete (YES in step S14), in step S15, the output unit 63 outputs the DNN trained by the training unit 62. The output unit 63 stores the DNN in the DNN storage unit 73.

[0147] In this way, by inputting a single distorted image into a DNN trained by deep learning, the coordinates of multiple vanishing points for calculating the camera's tilt, pan, and roll angles are estimated. Then, from the estimated coordinates of these multiple vanishing points, the camera's tilt, pan, and roll angles, which are part of the camera parameters, can be calculated. Therefore, camera parameters can be calculated with high accuracy from a single distorted image.

[0148] Next, we will explain how the learning unit 62 calculates the network error, referring to Figure 15.

[0149] Figure 15 is a schematic diagram illustrating the method for calculating the network error in this embodiment.

[0150] In Figure 15, the first vanishing point V front , second vanishing point V zenith , third vanishing point V right , and the fourth vanishing point V left This is the true value in training image 45. Also, the first vanishing point V' front , second vanishing point V' zenith , third vanishing point V' right , and the fourth vanishing point 'V' left This is an estimated value calculated by the learning unit 62.

[0151] First, let's explain the error in the pan angle φ.

[0152] True first vanishing point V front and the true second vanishing point V zenith This is the true third vanishing point V. right and the true fourth vanishing point V left It lies on the perpendicular bisector of the line segment connecting and . However, due to the error in the DNN estimate, the estimated first vanishing point V' front and the estimated second vanishing point V' zenith This is the true third vanishing point V. right and the true fourth vanishing point V le ft It may not lie on the perpendicular bisector of the line segment connecting the two points.

[0153] The error in the pan angle φ corresponds to the true third vanishing point V. right and the true fourth vanishing point V left The perpendicular bisector of the line segment connecting and and the estimated first vanishing point V' which is parallel to the perpendicular bisector. front A straight line L passes through it. front The first distance ΔV between front,φ It is defined as follows. Also, the error amount of the pan angle φ is the true third vanishing point V right and the true fourth vanishing point V left The perpendicular bisector of the line segment connecting and , and the estimated second vanishing point V' which is parallel to the perpendicular bisector. zenith A straight line L passes through it. zenith The second distance ΔV between zenith,φ It is defined as the first distance ΔV. front,φ and the second distance ΔV zenith,φ This corresponds to the error in the parameterized pan angle φ.

[0154] Next, I will explain the error in the tilt angle θ.

[0155] Similar to the error in the pan angle φ, as shown in Figure 15, the amount of error in the tilt angle θ is the true third vanishing point V right and the true fourth vanishing point V left The true first vanishing point V in the direction along the perpendicular bisector of the line segment connecting and front The first vanishing point V' was estimated to be this. front The third distance ΔV between front,θ It is defined as follows. Also, the amount of error in the tilt angle θ is the true third vanishing point V right and the true fourth vanishing point V left The true second vanishing point V is in the direction along the perpendicular bisector of the line segment connecting and zenith The second vanishing point V' was estimated to be this. zenith The fourth distance ΔV between zenith,θ It is defined as the third distance ΔV. front,θ and the fourth distance ΔV zenith,θ This corresponds to the error in the parameterized tilt angle θ.

[0156] Next, I will explain the error in the roll angle ψ.

[0157] The error in the roll angle ψ is equal to the true third vanishing point V. right and the true fourth vanishing point V left The line segment connecting and the estimated third vanishing point V' right The fourth vanishing point V' was estimated to be this. left It is defined as the angle Δψ formed by the line segment connecting and . The angle Δψ corresponds to the error of the parameterized roll angle ψ.

[0158] The network error (loss) representing the errors in tilt angle, pan angle, and roll angle is expressed by the following formula (14).

[0159] Network error = w1ΔV front,φ +w2ΔV zenith,φ +w3ΔV f ront,θ +w4ΔV zenith,θ +w5Δψ···(14)

[0160] In the above formula (14), w1 to w5 are linear combination coefficients of the errors. For example, w1, w2, w3, and w4 are 0.5, and w5 is 1.

[0161] The learning unit 62 updates the DNN parameters to minimize the calculated network error.

[0162] Furthermore, the learning unit 62 measures the first distance ΔV front,φ The squared value of ΔV 2 front,φ , second distance ΔV zenith,φ The squared value of ΔV 2 zenith,φ , third distance ΔV front ,θ The squared value of ΔV 2 front,θ , fourth distance ΔV zenith,θ The squared value of ΔV 2 ze nith,θ , and the squared value of the angle Δψ Δψ 2 The network error may be calculated by adding the first distance ΔV. Alternatively, the learning unit 62 calculates the first distance ΔV front,φ Huber loss, second distance ΔV zenith,φ Huber loss, third distance ΔV front,θHuber loss, fourth distance ΔV zenith,θ The network error may also be calculated by adding the Huber loss for the given angle Δψ. The Huber loss is an error function that is a squared error for absolute errors less than 0.5 and a first-power error for absolute errors greater than or equal to 0.5.

[0163] By following the procedure described above, the camera's orientation can be calculated from an elliptical horizon, and a DNN can be trained using the network error based on world coordinates. This allows for highly accurate calculation of camera parameters from a single distorted image captured by a fisheye camera.

[0164] (modified version) The camera parameter calculation device and learning device according to one or more embodiments of this disclosure have been described above based on embodiments, but this disclosure is not limited to these embodiments. Without departing from the spirit of this disclosure, various modifications that a person skilled in the art can conceive of may be applied to these embodiments, and forms constructed by combining components from different embodiments may also be included within the scope of one or more embodiments of this disclosure.

[0165] In each of the above embodiments, each component may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0166] Some or all of the functions of the apparatus according to the embodiments of this disclosure are typically implemented as an integrated circuit, or LSI (Large Scale Integration). These may be individually integrated onto a single chip, or some or all of them may be integrated onto a single chip. Furthermore, the integration is not limited to LSIs, but may also be implemented using dedicated circuits or general-purpose processors. An FPGA (Field Programmable Gate Array) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells inside the LSI may also be used.

[0167] Furthermore, some or all of the functions of the apparatus according to the embodiments of this disclosure may be realized by a processor such as a CPU executing a program.

[0168] Furthermore, all figures used above are illustrative examples provided to illustrate this disclosure, and this disclosure is not limited to these illustrative figures.

[0169] Furthermore, the order in which the steps shown in the flowchart above are performed is illustrative for the purpose of specifically illustrating this disclosure, and other orders are acceptable as long as similar effects are achieved. Also, some of the steps above may be performed simultaneously (in parallel) with other steps. [Industrial applicability]

[0170] The technology disclosed herein can calculate camera parameters with high accuracy from a single distorted image, and is therefore useful as a technology for training a deep neural network for calculating camera parameters from an image, and as a technology for calculating camera parameters from an image.

Claims

1. An image acquisition unit that acquires images taken by a camera that causes distortion, A vanishing point acquisition unit that acquires the coordinates of multiple true vanishing points for calculating the tilt angle, pan angle, and roll angle of the aforementioned camera, A learning unit that trains a deep neural network by deep learning using the image acquired by the image acquisition unit and the coordinates of the plurality of true vanishing points acquired by the vanishing point acquisition unit, An output unit that outputs the deep neural network learned by the learning unit, Equipped with, The aforementioned learning unit, By inputting the aforementioned image into the deep neural network, the coordinates of multiple vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera are estimated. Based on the coordinates of the multiple true vanishing points and the estimated coordinates of the multiple vanishing points, a network error indicating the error in the tilt angle, pan angle, and roll angle is calculated. The parameters of the deep neural network are learned to minimize the calculated network error. The plurality of vanishing points include, in the image, a first vanishing point in the direction of the camera's movement, a second vanishing point in the zenith direction of the camera, a third vanishing point to the right of the camera, and a fourth vanishing point to the left of the camera. Learning device.

2. The aforementioned learning unit, The first distance is calculated between the perpendicular bisector of the line segment connecting the true third vanishing point and the true fourth vanishing point, and the straight line parallel to the perpendicular bisector and passing through the estimated first vanishing point. The second distance is calculated between the perpendicular bisector and the straight line that is parallel to the perpendicular bisector and passes through the estimated second vanishing point. A third distance is calculated between the true first vanishing point and the estimated first vanishing point in the direction along the perpendicular bisector. The fourth distance is calculated between the true second vanishing point and the estimated second vanishing point in the direction along the perpendicular bisector. The angle between the line segment connecting the true third vanishing point and the true fourth vanishing point, and the line segment connecting the estimated third vanishing point and the estimated fourth vanishing point is calculated. The sum of the first distance, the second distance, the third distance, the fourth distance, and the angle is calculated as the network error. The learning device according to claim 1.

3. A learning method in computers, By acquiring images taken with a camera that produces distortion, To calculate the tilt angle, pan angle, and roll angle of the aforementioned camera, obtain the coordinates of multiple true vanishing points. A deep neural network is trained using deep learning with the acquired image and the coordinates of the acquired multiple true vanishing points. The trained deep neural network is output, In the training of the deep neural network described above, By inputting the aforementioned image into the deep neural network, the coordinates of multiple vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera are estimated. Based on the coordinates of the multiple true vanishing points and the estimated coordinates of the multiple vanishing points, a network error indicating the error in the tilt angle, pan angle, and roll angle is calculated. The parameters of the deep neural network are learned to minimize the calculated network error. The plurality of vanishing points include, in the image, a first vanishing point in the direction of the camera's movement, a second vanishing point in the zenith direction of the camera, a third vanishing point to the right of the camera, and a fourth vanishing point to the left of the camera. Learning methods.

4. An image acquisition unit that acquires images taken by a camera that causes distortion, A vanishing point acquisition unit that acquires the coordinates of multiple true vanishing points for calculating the tilt angle, pan angle, and roll angle of the aforementioned camera, A learning unit that trains a deep neural network by deep learning using the image acquired by the image acquisition unit and the coordinates of the plurality of true vanishing points acquired by the vanishing point acquisition unit, The computer is made to function as an output unit that outputs the deep neural network learned by the learning unit. The aforementioned learning unit, By inputting the aforementioned image into the deep neural network, the coordinates of multiple vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera are estimated. Based on the coordinates of the multiple true vanishing points and the estimated coordinates of the multiple vanishing points, a network error indicating the error in the tilt angle, pan angle, and roll angle is calculated. The parameters of the deep neural network are learned to minimize the calculated network error. The plurality of vanishing points include, in the image, a first vanishing point in the direction of the camera's movement, a second vanishing point in the zenith direction of the camera, a third vanishing point to the right of the camera, and a fourth vanishing point to the left of the camera. Learning program.

5. An image acquisition unit that acquires images taken by a camera that causes distortion, An estimation unit estimates the coordinates of multiple vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera by inputting the image acquired by the image acquisition unit into a deep neural network trained by deep learning. A calculation unit calculates the tilt angle, the pan angle, and the roll angle based on the coordinates of the plurality of vanishing points estimated by the estimation unit, An output unit that outputs camera parameters including the tilt angle, pan angle, and roll angle calculated by the calculation unit, Equipped with, During the training of the deep neural network, Training images have been acquired. The coordinates of multiple true vanishing points are obtained to calculate the tilt angle, pan angle, and roll angle of the camera that captured the aforementioned training image. When the training image is input to the deep neural network, the coordinates of multiple vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera that captured the training image are estimated. Based on the coordinates of the multiple true vanishing points and the estimated coordinates of the multiple vanishing points, a network error indicating the error in the tilt angle, pan angle, and roll angle is calculated. The parameters of the deep neural network are learned to minimize the calculated network error. The plurality of vanishing points include, in the image, a first vanishing point in the direction of the camera's movement, a third vanishing point to the right of the camera, and a fourth vanishing point to the left of the camera. Camera parameter calculation device.

6. The calculation unit described above, The roll angle is calculated using the coordinates of the first vanishing point and the coordinates of the midpoint of the line segment connecting the third vanishing point and the fourth vanishing point. The tilt angle is calculated using the y-coordinate of the first vanishing point, the y-coordinate of the midpoint of the line segment connecting the third vanishing point and the fourth vanishing point, and the inverse function of the camera's projection function. The pan angle is calculated using the x-coordinate of the principal point image coordinate of the camera, the x-coordinate of the midpoint of the line segment connecting the third vanishing point and the fourth vanishing point, and the inverse function of the projection function. The camera parameter calculation device according to claim 5.

7. A method for calculating camera parameters in a computer, By acquiring images taken with a camera that produces distortion, By inputting the acquired image into a deep neural network trained by deep learning, the coordinates of multiple vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera are estimated. Based on the estimated coordinates of the multiple vanishing points, the tilt angle, pan angle, and roll angle are calculated. The camera parameters, including the calculated tilt angle, pan angle, and roll angle, are output. During the training of the deep neural network, Training images have been acquired. The coordinates of multiple true vanishing points are obtained to calculate the tilt angle, pan angle, and roll angle of the camera that captured the aforementioned training image. When the training image is input to the deep neural network, the coordinates of multiple vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera that captured the training image are estimated. Based on the coordinates of the multiple true vanishing points and the estimated coordinates of the multiple vanishing points, a network error indicating the error in the tilt angle, pan angle, and roll angle is calculated. The parameters of the deep neural network are learned to minimize the calculated network error. The plurality of vanishing points include, in the image, a first vanishing point in the direction of the camera's movement, a third vanishing point to the right of the camera, and a fourth vanishing point to the left of the camera. Camera parameter calculation method.

8. An image acquisition unit that acquires images taken by a camera that causes distortion, An estimation unit estimates the coordinates of multiple vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera by inputting the image acquired by the image acquisition unit into a deep neural network trained by deep learning. A calculation unit calculates the tilt angle, the pan angle, and the roll angle based on the coordinates of the plurality of vanishing points estimated by the estimation unit, The computer functions as an output unit that outputs camera parameters including the tilt angle, pan angle, and roll angle calculated by the calculation unit. During the training of the deep neural network, Training images have been acquired. The coordinates of multiple true vanishing points are obtained to calculate the tilt angle, pan angle, and roll angle of the camera that captured the aforementioned training image. When the training image is input to the deep neural network, the coordinates of multiple vanishing points for calculating the tilt angle, pan angle, and roll angle of the camera that captured the training image are estimated. Based on the coordinates of the multiple true vanishing points and the estimated coordinates of the multiple vanishing points, a network error indicating the error in the tilt angle, pan angle, and roll angle is calculated. The parameters of the deep neural network are learned to minimize the calculated network error. The plurality of vanishing points include, in the image, a first vanishing point in the direction of the camera's movement, a third vanishing point to the right of the camera, and a fourth vanishing point to the left of the camera. Camera parameter calculation program.

Images