Information processing device, information processing method, and information processing program

Abstract

An information processing device acquires an image captured by a camera, and estimates a vanishing point by inputting the image into a learning completion model generated by performing machine learning using a heat map representing a true value of the vanishing point as training data on a learning model used in bone estimation.

Description

Technical Field

[0001] This disclosure relates to techniques for detecting the posture of a camera. Background Technology

[0002] In recent years, a technique for estimating multiple vanishing points based on the Manhattan world hypothesis using distorted images has been known (e.g., non-patent documents 1 and 2). This technique involves the following steps: acquiring a distorted image; detecting multiple candidate arcs in the acquired image based on the Manhattan world hypothesis, representing the vertical, horizontal, and longitudinal directions; searching for the optimal combination among the detected arcs; and estimating multiple vanishing points based on the search results.

[0003] However, the aforementioned prior art has the following drawback: in locations where it is difficult to detect arcs, the vanishing point cannot be accurately estimated. Therefore, the aforementioned prior art suffers from the problem of not being able to accurately determine the camera's pose.

[0004] Prior art literature

[0005] Non-patent literature

[0006] Non-patent literature 1: Y. Lochman, O. Dobosevych, R. Hryniv, and J. Pritts. Minimalsolvers for single-view lens-distorted camera autocalibration. In Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV), pages2886-2895, 2021.

[0007] Non-patent literature 2: J. Pritts, Z. Kukelova, V. Larsson, and O. Chum. Radially-distorted conjugate translations. In Proceedings of IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1993-2001, 2018. Summary of the Invention

[0008] The purpose of this disclosure is to provide a technique that can accurately estimate the vanishing point.

[0009] An information processing apparatus according to one aspect of this disclosure includes: an acquisition unit for acquiring an image captured by a camera; and an estimation unit for estimating a vanishing point by inputting the image into a learned model, the learned model being generated by performing machine learning on a learning model used in skeleton estimation, using a heatmap representing the truth value of the vanishing point as training data.

[0010] Based on this structure, the vanishing point can be accurately estimated. Attached Figure Description

[0011] Figure 1 This is a diagram illustrating an example of the structure of the information processing apparatus in Embodiment 1 of this disclosure.

[0012] Figure 2 This is a flowchart illustrating an example of the process in Implementation 1.

[0013] Figure 3 This is a diagram illustrating an example of a world coordinate system and a camera coordinate system.

[0014] Figure 4A This is a flowchart illustrating an example of the processing in Variation 1 of Embodiment 1.

[0015] Figure 4B This is a flowchart illustrating an example of the processing in Variation 2 of Embodiment 1.

[0016] Figure 5 This is a diagram illustrating an example of the structure of the information processing device in Embodiment 2.

[0017] Figure 6 This is a flowchart illustrating an example of the processing of the information processing device in Embodiment 2.

[0018] Figure 7 This is a diagram showing an outline of the process in Implementation Method 2.

[0019] Figure 8 This is a diagram illustrating an example of the structure of the information processing device in Embodiment 3.

[0020] Figure 9 This is a flowchart illustrating an example of the processing of the information processing device in Embodiment 3.

[0021] Figure 10 It is a diagram showing the vanishing point projected onto the unit sphere and the auxiliary diagonal points of the first pattern.

[0022] Figure 11 It is Figure 10 The table summarizes the configurations of the vanishing points and auxiliary diagonal points shown.

[0023] Figure 12 This is a diagram illustrating an example of the structure of the information processing device in Embodiment 4.

[0024] Figure 13 This is a flowchart illustrating an example of the processing of the information processing device in Embodiment 4.

[0025] Figure 14 This is a diagram showing the auxiliary diagonal points of the second pattern.

[0026] Figure 15 This is a diagram showing the auxiliary diagonal points of the third pattern. Detailed Implementation

[0027] (A method for realizing this disclosure)

[0028] In the autonomous driving control of mobile devices such as cars and drones, the attitude of the mobile device is treated as rotation relative to the road. Therefore, such mobile devices are equipped with odometry or gyroscope sensors to estimate the attitude. Standardly, mobile devices are equipped with cameras for external sensing. If the attitude of the mobile device can be estimated solely by the camera, then odometry or gyroscope sensors are unnecessary and are therefore preferred.

[0029] In the aforementioned non-patent documents 1 and 2, based on the Manhattan world hypothesis and an image, multiple arcs are detected as candidates in the vertical, horizontal, and longitudinal directions, and multiple vanishing points are estimated based on the detected arcs. However, in non-patent documents 1 and 2, the detection of multiple arcs is a prerequisite for accurately estimating the vanishing points. Therefore, in patent documents 1 and 2, when dealing with urban areas where multiple trees are arranged along roads, the unclear outlines of buildings lead to the problem of not being able to detect multiple arcs. Consequently, accurate vanishing points cannot be estimated, and the camera's pose cannot be accurately determined.

[0030] Therefore, the inventors studied a learning model for estimating vanishing points without using circular arcs. It was then recognized that vanishing point estimation has a high affinity with skeletal point estimation, and that a learning model used for skeletal point estimation can also be used for vanishing point estimation.

[0031] This disclosure was made to solve such a problem, and its purpose is to provide a technique that can accurately estimate the vanishing point.

[0032] (1) An information processing apparatus according to one aspect of the present disclosure includes: an acquisition unit for acquiring an image captured by a camera; and an estimation unit for estimating a vanishing point by inputting the image into a learned model, the learned model being generated by performing machine learning on a learning model used in skeleton estimation, using a heatmap representing the truth value of the vanishing point as training data.

[0033] According to this structure, a fully learned model is generated by performing machine learning on the learning model used in skeleton point estimation, using heatmaps representing the ground truth of the vanishing point as training data. This fully learned model is then used to estimate the vanishing point. Therefore, by utilizing the learning model used for skeleton point estimation, a fully learned model capable of estimating the vanishing point with high accuracy is obtained. As a result, the vanishing point can be estimated with high precision.

[0034] (2) In the information processing apparatus described in (1) above, the estimation unit may further estimate an auxiliary diagonal point based on the image. The auxiliary diagonal point is any point among a group of eight or more points that are positioned to maintain symmetry with a regular octahedron inscribed in the unit sphere when projected onto a unit sphere in the world coordinate system.

[0035] According to this structure, since auxiliary diagonal points are presumed in addition to the vanishing point, the camera's pose can be uniquely determined using the auxiliary diagonal points even if the vanishing point required to uniquely determine the camera's pose is not obtained.

[0036] (3) In the learning completed model described in (1) or (2) above, machine learning can also be performed using an error function that evaluates the error between the estimated heatmap output from the learning model and the true heatmap representing the true value of the vanishing point. The error function uses the vanishing points included in the estimated heatmap and the vanishing points not included in the estimated heatmap for the evaluation of the error.

[0037] In the error function of machine learning used for skeleton point estimation, skeleton points not mapped to the image are not used for error evaluation. In contrast, in vanishing point estimation, since the image is captured by a camera, the vanishing point is essentially included in the image. Therefore, the vanishing point should be considered in the error evaluation regardless of whether it is included in the estimated heatmap. In this structure, because the vanishing point is used for error evaluation regardless of whether it is included in the estimated heatmap, a learned model with high accuracy in vanishing point estimation is obtained.

[0038] (4) In any of the information processing apparatuses described in (1) to (3) above, the learned model may also perform machine learning using an error function that evaluates the error between the estimated heatmap output from the learned model and the truth heatmap representing the vanishing point and the truth value of the auxiliary diagonal point.

[0039] Based on this structure, a fully learned model with high accuracy in estimating the vanishing point and auxiliary diagonal points is obtained.

[0040] (5) In any of the information processing apparatuses described in (1) to (4) above, the information processing apparatus may further include: a projection unit that projects the estimated vanishing point onto a unit sphere disposed in the world coordinate system based on the internal parameters of the camera; and a calculation unit that calculates a rotation angle representing the posture of the camera based on the error between the projected vanishing point and a reference vanishing point pre-projected onto the unit sphere.

[0041] Based on this structure, the rotation angle that accurately represents the camera's posture can be obtained.

[0042] (6) Another aspect of the information processing method in this disclosure is a computer information processing method in which an image captured by a camera is acquired and a vanishing point is estimated by inputting the image into a learned model, the learned model being generated by having a learning model used in skeleton estimation perform machine learning with a heatmap representing the truth value of the vanishing point as training data.

[0043] Based on this structure, an information processing method capable of accurately estimating the vanishing point can be provided.

[0044] (7) The information processing program in another aspect of this disclosure enables the computer to function as an information processing device as described in any one of (1) to (5) above.

[0045] Based on this structure, an information processing program can be provided to accurately estimate the vanishing point.

[0046] This disclosure can also be implemented as an information update program that causes a computer to execute the characteristic structures included in such an information update method, or as an information update system that operates according to such an information update program. Furthermore, such a computer program can of course be distributed via a computer-readable non-transitory recording medium such as a CD-ROM or a communication network such as the Internet.

[0047] Furthermore, the embodiments described below are all specific examples of this disclosure. The numerical values, shapes, structural elements, steps, and order of steps shown in the following embodiments are examples and are not intended to limit this disclosure. In addition, structural elements in the following embodiments that are not described in the independent technical solutions representing the highest-level concept are described as arbitrary structural elements. Furthermore, various contents can be combined in all embodiments.

[0048] (Implementation Method 1)

[0049] Figure 1This diagram illustrates an example of the structure of the information processing apparatus 1 according to Embodiment 1 of this disclosure. The information processing apparatus 1 is composed of a computer including a communication interface. The information processing apparatus 1 can be composed of a cloud server or an edge-side computer. The information processing apparatus 1 includes a processor 10 and a memory 20. An example of the processor 10 is a central processing unit (CPU). The processor 10 includes an acquisition unit 11 and a setting unit 12. The acquisition unit 11 and the setting unit 12 are implemented, for example, by executing an information processing program by the processor 10. The acquisition unit 11 and the setting unit 12 can be installed in one computer or distributed across multiple computers. The processor 10 and the memory 20 can be installed in one computer or distributed across multiple computers.

[0050] The acquisition unit 11 acquires information representing the coordinate axes of the world coordinate system 21 from the memory 20. The world coordinate system 21 is a three-dimensional coordinate system based on the Manhattan world hypothesis. The acquisition unit 11 acquires the frontal orientation of the camera 2 by acquiring information representing the coordinate axes of the camera coordinate system 22 from the memory 20. The camera coordinate system 22 is the coordinate system possessed by the camera mounted on the moving body. In the camera coordinate system 22, the frontal orientation of the camera is predetermined. In this embodiment, the frontal orientation of the camera 2 is regarded as the frontal orientation of the moving body 3. Furthermore, the moving body is not necessarily limited to a narrow sense of moving body such as a car, but may also be a device worn by a person, such as smart glasses (eyeglass-type electronic display devices).

[0051] The setting unit 12 sets the first axis, one of the two axes in the world coordinate system 21 that defines the ground, as the reference axis for the pan / tilt angle of the camera 2. The first axis includes a first direction extending to one side from the origin of the world coordinate system 21 and a second direction extending to the other side from the origin. Furthermore, in this disclosure, the term "ground" refers to the reference surface used to construct the image obtained by the camera, which includes not only roads but also floor surfaces existing inside and outside buildings.

[0052] The setting unit 12 calculates a first angle, which is the angle formed by the first direction and the front direction of the camera 2, and a second angle, which is the angle formed by the second direction and the front direction of the camera 2. Furthermore, the setting unit 12 sets the direction of the first axis toward the side with the smaller angle between the first angle and the second angle as the forward direction.

[0053] Figure 3 This is a diagram illustrating an example of world coordinate system 21 and camera coordinate system 22. Figure 3In this system, world coordinate system 21 has three mutually orthogonal axes: Xm, Ym, and Zm. World coordinate system 21 is a right-handed system. It is based on the Manhattan world hypothesis, in which the world is considered to consist of a grid of roads 35 and 36. Two of the three axes of world coordinate system 21 are parallel to roads 35 and 36, and the remaining axis is the height direction orthogonal to the ground. Figure 3 In this example, the Xm axis is parallel to road 36, the Zm axis is parallel to road 35, and the Ym axis defines the height direction. Furthermore, in the Manhattan world hypothesis, buildings 31-34 are considered to be composed of cuboids. The downward direction of the Ym axis is the positive direction.

[0054] In this embodiment, it is assumed that the Xm-Zm plane is a surface that defines the ground.

[0055] Camera coordinate system 22 is the coordinate system of camera 2 mounted on moving body 3. Camera coordinate system 22 is a three-dimensional coordinate system composed of three mutually orthogonal axes: Xc, Yc, and Zc. Camera coordinate system 22 is a right-handed system. The Zc axis defines the frontal direction of camera 2. Since the frontal direction of camera 2 is the same as the frontal direction of moving body 3, the Zc axis refers to the frontal direction of moving body 3. For ease of understanding, the roll angle and pitch angle will be set to 0 degrees in the following description; however, the present invention is not limited to 0 degrees, and can be implemented with arbitrary roll and pitch angles. For example, when the roll angle is 180 degrees and the pitch angle is 0 degrees, the downward direction of the Yc axis (described later) becomes the negative direction (due to the 180-degree rotation of the roll angle, the camera coordinate system is reversed vertically). The Yc axis defines the height direction orthogonal to the ground. The Xc axis defines the lateral direction of camera 2 and moving body 3. The Xc-Zc plane is parallel to the Xm-Zm plane. In this embodiment, it is assumed that the configuration of the camera coordinate system 22 in the world coordinate system 21 is known. The downward direction of the Yc axis is the positive direction.

[0056] Having calculated the pan angle φ of camera 2, either the Zm axis or the Xm axis can be set as the reference axis for the pan angle φ. Furthermore, when defining the pan angle φ, it can be specified which direction of the reference axis is forward and which is backward. Furthermore, a direction on the ground orthogonal to the forward / backward direction can be set as the lateral direction, and it can be specified which direction of the lateral direction is right and which is left.

[0057] In conventional techniques, no specific reference axis is set for the roll angle φ. The reference angle for the roll angle φ is randomly determined from the Zm axis and Xm axis each time the roll angle is calculated. Therefore, conventional techniques suffer from fourfold rotational symmetry indeterminacy, and the roll angle can only be specified within the range of -45° to 45°.

[0058] Therefore, in this embodiment, the setting unit 12 sets the first axis, which serves as one of the Xm and Zm axes defining the ground, as the reference axis for the pan / tilt angle of the camera 2 in the world coordinate system 21. Here, the Zm axis, which is parallel to the predetermined road direction K1, is set as the reference axis. This eliminates the fourfold rotational symmetry indeterminacy.

[0059] Furthermore, the setting unit 12 calculates the first angle α formed by the positive direction (first direction) of the Zm axis and the Zc axis. The setting unit 12 calculates the second angle β formed by the negative direction (second direction) of the Zm axis and the Zc axis. The setting unit 12 sets the direction of the Zm axis toward the side with the smaller angle between the first angle α and the second angle β as the forward direction. Here, since the first angle α is smaller than the second angle β, the positive direction of the Zm axis is set as the forward direction. The setting unit 12 takes the forward direction as the front, sets the positive direction of the Xm axis, which is the right direction, as the right direction, and sets the negative direction of the X axis, which is the left direction, as the left direction. In addition, four directions—front, back, left, and right—are defined.

[0060] Figure 2 This is a flowchart illustrating an example of the process in Embodiment 1. First, in step S1, the acquisition unit 11 acquires information representing the coordinate axes of the world coordinate system 21 from the memory 20. Next, in step S2, the Ym axis, which is orthogonal to the Xm-Zm plane corresponding to the ground, is set as the height direction among the three axes constituting the world coordinate system 21. Next, in step S3, the setting unit 12 sets the Zm axis, which is parallel to the road direction K1, as the reference axis for the pan / tilt angle φ. Next, in step S4, the acquisition unit 11 acquires information representing the coordinate axes of the camera coordinate system 22 from the memory 20.

[0061] Next, in step S5, the setting unit 12 calculates... Figure 3 The first angle α and the second angle β are shown. Next, in step S6, the setting unit 12 determines whether the first angle α is less than the second angle β. If the first angle α is less than the second angle β (yes in step S6), the setting unit 12 sets the first angle α side as the forward direction in the Zm axis (step S7). Figure 3 In the example, the positive direction of the Zm axis is set as the forward direction. On the other hand, if the first angle α is greater than or equal to the second angle β (not in step S6), the setting unit 12 sets the second angle β side as the forward direction in the Zm axis (step S9). Figure 3 In this example, the negative direction of the Zm axis is set as the rear direction. Next, in step S8, the setting unit 12 sets the left and right directions for the Xm axis. Figure 3 In the example, the positive direction of the Xm axis is set to the right, and the negative direction of the Xm axis is set to the left.

[0062] Thus, according to this embodiment, in the three-dimensional world coordinate system 21 based on the Manhattan world hypothesis, the Zm axis, one of the Xm and Zm axes defining the ground, is set as the reference axis for the pan angle φ of the camera 2. Therefore, when estimating the pan angle based on the image, the pan angle φ can be represented with the Zm axis as the reference, accurately indicating the direction in which the camera 2 is facing. Therefore, even in locations such as intersections with fourfold rotational symmetry indeterminacy, the pan angle can be accurately represented. If the pan angle can be accurately represented in this way, it is possible to accurately determine the direction from which the image obtained by the camera was captured.

[0063] (Modification 1 of Implementation Method 1)

[0064] When the forward direction is set as the reference direction for the roll angle, it becomes difficult to represent the roll angle as being in a range of -90° to 90° when the travel direction of the moving body 3 is towards the rearward direction. Therefore, in a variation of embodiment 1, the reference direction for the roll angle is set as the rearward direction in such a scenario.

[0065] The following is for reference Figure 1 A variation of Embodiment 1 will be described. When the reference direction of the pan angle is set to the forward direction, and when the first direction information indicating that the image captured by the camera 2 is an image captured from the rear side is obtained, the setting unit 12 sets the rear direction, which is opposite to the forward direction, as the reference direction of the pan angle.

[0066] Figure 4A This is a flowchart illustrating an example of the processing in Variation 1 of Embodiment 1. Figure 4A The flowchart shown is executed, for example, when the moving body 3 is traveling on the road and the camera 2 is capturing images. Furthermore, it is assumed that during execution... Figure 4A Before the flowchart shown, through Figure 2 The flowchart shown defines the forward, backward, left, and right directions for the Xm and Zm axes.

[0067] First, in step S21, the setting unit 12 determines whether the forward direction is set as the reference direction for the pan angle. If the forward direction is not set as the reference direction for the pan angle (no in step S21), the process ends. On the other hand, if the forward direction is set as the reference direction for the pan angle (yes in step S21), the setting unit 12 determines whether first direction information has been acquired (step S22). The first direction information is information accompanying the image captured by the camera 2, for example, set in the camera 2 during shooting. The first direction information may also be information input by the user through the camera 2. If the first direction information is acquired (yes in step S22), the process proceeds to step S23; if the first direction information is not acquired (no in step S22), the process ends. In this case, the reference direction for the pan angle remains the forward direction. Next, in step S23, the setting unit 12 sets the rearward direction as the reference direction for the pan angle.

[0068] Thus, in Variation 1 of Embodiment 1, since the reference direction for the pan angle is set to the rear direction based on whether or not directional information is acquired, the pan angle can be represented within the range of -90° to 90°, making the pan angle easy to handle. Furthermore, in Variation 1 of Embodiment 1, after setting the rear direction as the reference direction for the pan angle, if directional information indicating that the image captured by the camera 2 was obtained from a forward-facing image is acquired, the setting unit 12 can simply return the reference direction for the pan angle to the forward direction.

[0069] (Modification 2 of Implementation Method 1)

[0070] When the right or left direction is set as the reference direction for the roll angle, and the travel direction of the moving body 3 is opposite to the reference direction, the roll angle cannot be represented within the range of -90° to 90°, making it difficult to handle. Therefore, in a variation 2 of embodiment 1, in such a scenario, the reference direction for the roll angle is set to the opposite direction.

[0071] When either the right or left direction is set as the reference direction for the pan angle, and a second direction information indicating that the image captured by the camera 2 is an image captured from the opposite side of the reference direction is obtained, the setting unit 12 sets the reference direction for the pan angle to the opposite direction.

[0072] Figure 4B This is a flowchart illustrating an example of the processing in Variation 2 of Embodiment 1. Figure 4B The flowchart shown is executed, for example, when the moving body 3 is traveling on the road and the camera 2 is capturing images. Furthermore, it is assumed that during execution... Figure 4B Before the flowchart shown, through Figure 2The flowchart shown defines the forward, backward, left, and right directions for the Xm and Zm axes. In this flowchart, it is assumed that the default reference direction is set to the right.

[0073] First, in step S31, the setting unit 12 determines whether the right direction is set as the reference direction for the pan angle. If the right direction is not set as the reference direction for the pan angle (no in step S31), the process ends. On the other hand, if the right direction is set as the reference direction for the pan angle (yes in step S31), the setting unit 12 determines whether second direction information has been acquired (step S32). Here, it determines whether second information indicating that the image captured by the camera 2 was captured from the left side has been acquired. If the second direction information has been acquired (yes in step S32), the process proceeds to step S33; if the second direction information has not been acquired (no in step S32), the process ends. In this case, the reference direction for the pan angle remains the right direction. Next, in step S33, the setting unit 12 sets the left direction as the reference direction for the pan angle.

[0074] Thus, in Variation 2 of Embodiment 1, since the reference direction of the pan angle is set to the opposite direction of the default reference direction based on whether or not direction information is acquired, the pan angle can be represented in the range of -90° to 90°, making the pan angle easy to handle. Furthermore, in Variation 2 of Embodiment 1, after setting the left direction as the reference direction of the pan angle, if direction information indicating that the image captured by the camera 2 was obtained from a right-facing image is acquired, the setting unit 12 can simply return the reference direction of the pan angle to the right direction.

[0075] (Modification 3 of Implementation Method 1)

[0076] In implementation 1, the world coordinate system 21 and the camera coordinate system are right-handed systems, but they can also be left-handed systems.

[0077] (Implementation Method 2)

[0078] Implementation Method 2 calculates the rotation angle representing the posture of the camera 2 based on the image captured by the camera 2. Figure 5 This is a diagram illustrating an example of the structure of the information processing apparatus 1A in Embodiment 2. Furthermore, in Embodiment 2, it is assumed that the processing shown in Embodiment 1 is executed beforehand, and the orientation of the coordinate axes of the world coordinate system 21 is set. The processor 110 and memory 120 of the information processing apparatus 1A shown in Embodiment 2 may also have the blocks present in the processor 10 and memory 20 of the information processing apparatus 1 shown in Embodiment 1. These situations are also the same in Embodiments 3 and 4 described later. In Embodiment 2, the same reference numerals are given to the same structural elements as in Embodiment 1, and descriptions are omitted.

[0079] Information processing device 1A includes a processor 110 and a memory 120. Due to the hardware structure of information processing device 1A and... Figure 1 The information processing device 1 shown is identical, therefore detailed description is omitted. Information processing device 1A is connected to camera 2. Camera 2 is communicatively connected to information processing device 1A via a given communication path. In the case where information processing device 1A is a cloud server, an example of the communication path is the Internet. In the case where information processing device 1A is an edge-side device, an example of the communication path is a wireless LAN or a wired LAN. In the case where information processing device 1A is installed on a mobile device 3, an example of the communication path is a vehicle-mounted network.

[0080] Camera 2 is mounted on the moving body 3. Camera 2, for example, captures images of the area around the moving body 3 at a given frame rate, and sends the captured images to information processing device 1A at the same frame rate. However, this is just one example; camera 2 can also capture images of the area around the moving body 3 based on shooting instructions from a user or information processing device 1A, and send the resulting images to information processing device 1A. An example of an image is a fisheye image. Other examples of images include panoramic images or rectangular normal images. Images can also be still images.

[0081] The processor 110 includes an image acquisition unit 111 (an example of an acquisition unit), a vanishing point estimation unit 112 (an example of an estimation unit), an internal parameter estimation unit 113 (an example of an estimation unit), a projection unit 114, an arithmetic unit 115, and an output unit 116. The image acquisition unit 111 to the output unit 116 can be installed in one computer or distributed across multiple computers.

[0082] Image acquisition unit 111 acquires images from camera 2. Vanishing point estimation unit 112 estimates multiple vanishing points by inputting the images acquired by image acquisition unit 111 into a first learning model. The first learning model is a machine learning model that has been pre-performed to estimate multiple vanishing points based on images. Based on the input images, the first learning model outputs heatmaps of multiple pixels, each represented by the likelihood of a vanishing point. Vanishing point estimation unit 112 outputs heatmaps sorted with the same number of vanishing points as a predetermined estimated object, and establishes a correspondence between the order and labels representing the types of vanishing points, such that the vanishing point estimated from the first heatmap of this order is called the right-hand vanishing point, and the vanishing point estimated from the second heatmap is called the left-hand vanishing point, thereby enabling the acquisition of labels representing the types of vanishing points.

[0083] The coordinates of the vanishing point in each heatmap are the coordinates of the pixel that becomes the pixel with the highest likelihood. However, the vanishing point doesn't necessarily have to be the pixel with the highest likelihood; for example, it could be the pixel that becomes the pixel with the highest likelihood after applying a Gaussian filter to the heatmap. For instance, when focusing on 9 pixels in a 3×3 grid, if the likelihood of the central pixel is 0 and the remaining 8 pixels each have a maximum likelihood of 0.9, the central pixel, which doesn't have the highest likelihood, can be estimated as the vanishing point. This mitigates the effects of errors included in the heatmap, improving the accuracy of the vanishing point estimation. Alternatively, the vanishing point can also be a pixel estimated with sub-pixel precision from the vicinity of the highest likelihood.

[0084] The first learning model is generated by performing machine learning that uses a heatmap representing the true values ​​of the vanishing points as training data. The vanishing point estimation unit 112 estimates multiple vanishing points based on the heatmap output by the first learning model.

[0085] In this embodiment, there are six types of vanishing points, numbered 1 to 6. The first vanishing point is the vanishing point in the direction directly in front of camera 2. The second vanishing point is the vanishing point in the direction opposite to the direct view of camera 2. The third vanishing point is the vanishing point in the direction of the zenith of camera 2. The fourth vanishing point is the vanishing point in the direction opposite to the zenith of camera 2. The fifth vanishing point is the vanishing point to the right in the horizontal direction of camera 2. The sixth vanishing point is the vanishing point to the left in the horizontal direction of camera 2.

[0086] The intrinsic parameter estimation unit 113 estimates the intrinsic parameters of the camera 2 by inputting an image into a second learning model. The second learning model is a machine learning model that has been pre-processed for estimating the intrinsic parameters. The intrinsic parameters include the focal length of the camera 2 and the distortion coefficient of the camera 2. Techniques for estimating the focal length and distortion coefficient from an image are disclosed, for example, in the following document D1.

[0087] Document D1: N. Wakai, Y. Ishii, S. Sato, and T. Yamashita. Rethinkinggenericcamera models for deep single image camera calibration to recover rotation and fisheye distortion. In Proceedings of European Conference on ComputerVision (ECCV), volume 13678, pages 679-698, 2022.

[0088] Therefore, the internal parameter estimation unit 113 can simply use the technique in document D1 to estimate the internal parameters.

[0089] The projection unit 114 projects multiple vanishing points estimated by the vanishing point estimation unit 112 onto a unit sphere arranged in the world coordinate system 21 based on the internal parameters estimated by the internal parameter estimation unit 113.

[0090] For the sake of later explanation, the three-dimensional rotation involved in camera calibration will be described. In reference D1, the external parameters are represented by a rotation matrix. The three-dimensional coordinates after rotational translation using the rotation matrix are uniquely determined, but this rotational translation has multiple representations besides the rotation matrix. Pan, pitch, and roll are examples of rotational representations, obtained by decomposing the rotation matrix into three rotational components. Furthermore, since the decomposition of the rotation matrix is ​​not unique, it is decomposed under constraints. For example, the group of pan / pitch / roll angles whose sum of the absolute values ​​of each angle is the smallest can be selected. In addition, the rotation matrix can be represented by a Rodrigues vector. In this case, the vector is represented by the rotation axis, and the vector length is represented by the rotation amount. Like the Rodrigues vector, the rotation matrix can also be represented by a quaternion, which represents the rotation by the rotation axis and the rotation amount. Furthermore, the conversion between the Rodrigues vector and the quaternion can be a one-to-one conversion, and its calculation method is disclosed in reference D2 below.

[0091] Document D2: D. Mortari, F. Markley, and P. Singla. Optimal linear attitude estimator. Journal of Guidance, Control, and Dynamics (JGCD), 3: 1619-1627, 2007.

[0092] Therefore, regarding three-dimensional rotations, they can be converted to each other, so any of the above rotation representations can be used depending on the content being processed.

[0093] The camera model shown in equations (1) and (2) establishes a correspondence between a point p in the world coordinate system and a pixel u in the image coordinate system. However, point p is a point on a unit sphere located at the origin of the world coordinate system.

[0094] [Mathematical Expression 1]

[0095]

[0096] u represents the two-dimensional coordinate data of the image coordinate system. R is the rotation matrix representing the rotation between camera coordinate system 22 and world coordinate system 21, and t is the translation vector representing the translation between camera coordinate system 22 and world coordinate system 21. Furthermore, in the camera parameters estimated from a single image processed according to this disclosure, since the amount of movement of the translation vector is arbitrary, the translation vector is set as a zero vector. (cu, cv) are the principal points of the image. (du, dv) are the pixel spacing of the image sensor of camera 2, which is known. γ represents distortion. Distortion γ is represented by equation (2). In equation (2), η represents the incident angle, and k1 represents the distortion coefficient. The rotation matrix R and the translation vector t are examples of the external parameters of camera 2.

[0097] Projection unit 114 projects the vanishing point, which is estimated from the image using the camera model, onto the unit sphere. Furthermore, equation (1) represents the forward projection from world coordinates to image coordinates, and the back projection from the image coordinates to the world coordinates is calculated as a positive real solution obtained by solving a cubic equation related to the incident angle η. To obtain the world coordinates corresponding to the camera's line-of-sight vector, the rotation matrix R in equation (1) is set to the identity matrix, and the translation vector t is set to the zero vector to calculate the back projection. In the back projection, the dimension increases from two-dimensional image coordinates to three-dimensional world coordinates, but by selecting the world coordinates of the unit sphere, the back projection point can be uniquely calculated. The vanishing point that coincides with the principal point of the image is a singularity that cannot be projected onto the unit sphere. Therefore, when the vanishing point coincides with the principal point of the image, projection unit 114 only needs to add a tiny amount to the vanishing point coordinates. An example of a tiny amount is 0.0000001. Furthermore, according to equation (1), the projection from the image onto the unit sphere is a back projection, but in the following description, it is only represented as a projection.

[0098] Return to reference Figure 5 The computation unit 115 calculates the rotation angle representing the pose of camera 2 based on the error between multiple vanishing points projected onto the unit sphere and multiple reference vanishing points pre-projected onto the unit sphere. The rotation angle represents the rotation of camera 2 relative to the world coordinate system 21. The rotation angle includes pan, pitch, and roll angles. (See reference...) Figure 3 The pan angle φ represents the rotation of camera 2 around the pan axis (Yc axis). The pitch angle represents the rotation of camera 2 around the pitch axis (Xc axis). The roll angle represents the rotation of camera 2 around the roll axis (Zc axis). The reference vanishing point refers to the vanishing point when the camera coordinate system 22 is not rotated relative to the world coordinate system 21. The reference vanishing point will be described later in embodiment 3. The error between the vanishing point and the reference vanishing point refers to the angle between the projected vanishing point and the reference vanishing point at the origin of the world coordinate system 21.

[0099] Specifically, the computation unit 115 searches for the rotation angle of multiple vanishing points that minimizes the error between the multiple vanishing points of the projection and the multiple reference vanishing points, and calculates the minimum rotation angle as the rotation angle representing the pose of the camera 2. The problem of minimizing such error is known as the absolute orientation problem, and the solution to this problem is disclosed in the following document D3, which calculates the quaternion that minimizes the error.

[0100] Document D3: Z. Wang and Jepson. A new closed-form solution for absolute orientation. In Proceedings of IEEE Conference on Computer Vision and PatternRecognition (CVPR), pages 129-134, 1994.

[0101] Additionally, as a solution to the absolute orientation problem with lower computational cost, there is the solution in reference D2. In reference D2, the Rodrigues vector that minimizes the error is calculated. The method for calculating the rotation angle using the solution in reference D2 will be explained later; however, the solution disclosed in reference D3 can also be used to calculate the rotation angle. In this case, the quaternion is calculated directly.

[0102] The arithmetic unit 115 calculates the error between multiple vanishing points and multiple reference vanishing points corresponding to each vanishing point. The arithmetic unit 115 then calculates the Rodriguez vector that minimizes this error. The Rodriguez vector is defined by the principal axes of rotation that minimize the error and the amount of rotation that minimizes the error. The arithmetic unit 115 calculates quaternions based on the Rodriguez vector. The arithmetic unit 115 then calculates the pan, tilt, and roll angles based on the quaternions. From this, the rotation angle of camera 2 is calculated.

[0103] The output unit 116 outputs the rotation angle of the camera 2 calculated by the calculation unit 115. Here, the output unit 116 can output the rotation angle to the memory 120, to an external device, or to the camera 2. Furthermore, pan, tilt, roll, and the aforementioned Rodriguez vector and quaternion can be converted. Therefore, the output unit 116 can also output a Rodriguez vector or quaternion instead of the aforementioned rotation angle. In this way, the output unit 116 only needs to output the rotation angle that the application expects to achieve the desired performance. This reduces unnecessary calculations.

[0104] The memory 120 stores, for example, the world coordinate system 21 and the camera coordinate system 22 that were initially set in Implementation 1.

[0105] Figure 6 This is a flowchart illustrating an example of the processing of the information processing apparatus 1A in Embodiment 2. First, in step S101, the acquisition unit 11 acquires an image from the camera 2. Next, in step S102, the vanishing point estimation unit 112 estimates the vanishing point by inputting the image into a first learning model. The first learning model outputs six heatmaps corresponding to the first to sixth vanishing points, respectively. The vanishing point estimation unit 112 detects peak values ​​based on the six heatmaps output from the first learning model. If the detected peak value is above a threshold, it is determined that a vanishing point has been estimated. For example, if the peak value of the heatmap corresponding to the first vanishing point is above the threshold, it is determined that the first vanishing point has been estimated. For example, if the peak value of the heatmap corresponding to the second vanishing point is below the threshold, it is determined that the second vanishing point has not been estimated. Furthermore, as described above, the vanishing point estimation unit 112 can detect peak values ​​using methods such as applying a Gaussian filter, or it can detect peak values ​​with sub-pixel precision.

[0106] Next, in step S103, the intrinsic parameter estimation unit 113 inputs the image into the second learning model, thereby estimating the intrinsic parameters. Here, the focal length f and the distortion coefficient k1 are estimated.

[0107] Next, in step S104, the projection unit 114 projects the vanishing point estimated in step S102 onto the unit sphere by applying the internal parameters estimated in step S103 to the camera model.

[0108] Next, in step S105, the calculation unit 115 calculates the error between the vanishing point projected onto the unit sphere and the reference vanishing point pre-projected onto the unit sphere corresponding to the vanishing point. For example, if the first vanishing point and the third vanishing point are estimated, the error between the first vanishing point and the first reference vanishing point, which serves as the reference vanishing point for the first vanishing point, and the error between the third vanishing point and the third reference vanishing point, which serves as the reference vanishing point for the third vanishing point, are calculated.

[0109] Next, in step S106, the calculation unit 115 searches for the rotation angle of the vanishing point that minimizes the error calculated in step S105. The details of this process are as described above.

[0110] Next, in step S107, the calculation unit 115 calculates the rotation angle of the camera 2 as the rotation angle that minimizes the error obtained in step S106.

[0111] Next, in step S108, the output unit 116 outputs the calculated rotation angle of the camera 2. Thus, the pan, tilt, and roll angles of the camera 2 are obtained.

[0112] Figure 7 This diagram illustrates an outline of the processing in Embodiment 2. Images captured by camera 2 are input to a first learning model and a second learning model. The first learning model outputs a heatmap. The second learning model outputs internal parameters. The vanishing point estimation unit 112 estimates the vanishing points based on the heatmap. The projection unit 114 projects the vanishing points onto a unit sphere positioned at the origin of the world coordinate system 21 using the internal parameters. In this example, since three vanishing points P1 to P3 are estimated, they are projected onto the unit sphere. The calculation unit 115 calculates the error between the reference vanishing point corresponding to each of vanishing points P1 to P3 and the vanishing points P1 to P3. Based on the calculated errors, the calculation unit 115 calculates the pan angle φ, the pitch angle θ, and the roll angle ψ.

[0113] Thus, according to Embodiment 2, multiple vanishing points are estimated by inputting an image into the first learning model, instead of estimating multiple vanishing points based on an arc. Therefore, even in locations where the outline of a building is unclear, the vanishing points can be accurately estimated. Furthermore, the estimated multiple vanishing points are projected onto a unit sphere based on internal parameters estimated using the image, and a rotation angle representing the camera's posture is estimated based on the error between the projected vanishing points and a reference vanishing point. Therefore, the camera's posture can be accurately estimated.

[0114] (A variation of Implementation Method 2)

[0115] Camera 2 can also be mounted on the moving body 3 such that its optical axis is oriented in a direction intersecting the frontal direction. This makes it easier to represent multiple vanishing points in the image, and facilitates the estimation of these vanishing points. For example, camera 2 can also be mounted on the moving body 3 such that its angle downwards relative to the frontal direction is a given angle (e.g., 30°, 45°).

[0116] (Implementation Method 3)

[0117] In addition to the vanishing point, implementation method 3 also takes into account auxiliary diagonal points to estimate the rotation angle of camera 2. Figure 8 This is a diagram showing an example of the structure of the information processing apparatus 1B in Embodiment 3. Furthermore, in Embodiment 3, the same reference numerals are used for structural elements identical to those in Embodiments 1 and 2, and descriptions are omitted. In the processor 110 of the information processing apparatus 1B, the structures of the vanishing point estimation unit 112B (an example of an estimation unit), the projection unit 114B, and the arithmetic unit 115B are particularly different.

[0118] In addition to the vanishing point, the vanishing point estimation unit 112B further estimates auxiliary diagonal points. Figure 10 This is a diagram showing the vanishing point and auxiliary diagonal points projected onto the unit sphere 1000. The unit sphere 1000 is configured with its center located at the origin of world coordinate system 21.

[0119] There are six vanishing points: PF, PB, PT, PM, PR, and PL. When projected onto a unit sphere 1000 using the camera model described above, these six vanishing points are located at the six vertices of a regular octahedron (illustration omitted) inscribed within the unit sphere 1000. The six vertices of this octahedron are located on any of the Xm, Ym, or Zm axes. That is, vanishing points PT and PM are located on the Ym axis, vanishing points PF and PB are located on the Zm axis, and vanishing points PR and PL are located on the Xm axis.

[0120] Vanishing point PF is the vanishing point in the direction directly in front of camera 2, and is the first vanishing point mentioned above. Vanishing point PB is the vanishing point in the opposite direction from the direct view of camera 2, and is the second vanishing point mentioned above. Vanishing point PT is the vanishing point in the direction of the zenith of camera 2, and is the third vanishing point mentioned above. Vanishing point PM is the vanishing point in the opposite direction from the zenith of camera 2, and is the fourth vanishing point mentioned above. Vanishing point PR is the vanishing point to the right of camera 2, and is the fifth vanishing point mentioned above. Vanishing point PL is the vanishing point to the left of camera 2, and is the sixth vanishing point mentioned above.

[0121] As mentioned above, the vanishing point represents the positive and negative infinity directions of each axis in a three-dimensional orthogonal coordinate system. Furthermore, on a unit sphere, when treating the vanishing point as a three-dimensional coordinate, at least six vanishing points are used to form a regular octahedron based on their positional relationships. Such a regular octahedron is a regular polygon with high symmetry. Based on the properties of this highly symmetric regular octahedron (also known as the regular octahedral symmetry group), the original auxiliary diagonal points described later were realized.

[0122] There are eight auxiliary diagonal points: FRT, FLT, BLT, BRT, FRB, FLB, BLB, and BRB. These eight auxiliary diagonal points are positioned to maintain the symmetry with respect to a regular octahedron inscribed within a unit sphere of 1000. Figure 10 The diagram illustrates an arrangement of eight auxiliary diagonal points that exhibit high spatial uniformity. These eight auxiliary diagonal points are located at the eight vertices of a cube 1001, which is inscribed within a unit sphere 1000 and, for example, has its upper surface orthogonal to the Ym axis. These eight auxiliary diagonal points possess the symmetry of the octahedral group and are spatially uniformly arranged. Figure 10 The configuration pattern of the 8 auxiliary diagonal points shown is called the first pattern.

[0123] Auxiliary diagonal point FRT is the auxiliary diagonal point in the forward-right-up direction; auxiliary diagonal point FLT is the auxiliary diagonal point in the forward-left-up direction; auxiliary diagonal point BLT is the auxiliary diagonal point in the backward-left-up direction; auxiliary diagonal point BRT is the auxiliary diagonal point in the backward-right-up direction; auxiliary diagonal point FRB is the auxiliary diagonal point in the forward-right-down direction; auxiliary diagonal point FLB is the auxiliary diagonal point in the forward-left-down direction; auxiliary diagonal point BLB is the auxiliary diagonal point in the backward-left-down direction; and auxiliary diagonal point BRB is the auxiliary diagonal point in the backward-right-down direction.

[0124] In addition, there are other cases where the auxiliary diagonal points are configured to maintain the symmetry of the regular octahedron. Figure 14 This is a diagram showing the second pattern of the configuration of the auxiliary diagonal points. Figure 15 This is a diagram showing the third pattern illustrating the configuration of the auxiliary diagonal points. The regular octahedron groups have six axes of symmetry, C2 (…). Figure 15 ), 4 axes of symmetry C3 ( Figure 10 ), 3 axes of symmetry C4 ( Figure 14 Furthermore, Cn is the Schoenflies notation, representing the axis of symmetry in a 360° / n rotational symmetry. To maintain the symmetry of the regular octahedron, the auxiliary diagonal points need to be configured on or symmetric about any of the axes of symmetry C2, C3, or C4. Using a large number of auxiliary diagonal points increases the constraints in estimating the camera's extrinsic parameters; however, with a large number of auxiliary diagonal points, optimization in the learning of deep neural networks becomes difficult. Therefore, practically speaking, a configuration with a small number of auxiliary diagonal points and high spatial uniformity is suitable. The auxiliary diagonal points in Pattern 1 are suitable for estimating camera parameters. Pattern 1 is a pattern with auxiliary diagonal points configured on axis of symmetry C3. Pattern 3 is a pattern with auxiliary diagonal points configured on axis of symmetry C2. The third pattern is a configuration consisting of a total of 12 points: the four midpoints of each of the four sides of the top surface of cube 1001, the four midpoints of each of the four sides of the bottom surface of cube 1001, and the four midpoints of each of the four sides of cube 1001 parallel to the Ym axis. The top surface of cube 1001 is the face facing the vanishing point PT, and the bottom surface of cube 1001 is the face facing the vanishing point PM.

[0125] The second pattern is a pattern with 8 auxiliary diagonal points arranged on the midline C5 of the 3 axes of symmetry C4. That is, the second pattern includes 8 auxiliary diagonal points where the 4 midlines C5 intersect the unit sphere 1000.

[0126] Among the configuration patterns of eight points maintaining the symmetry of a regular octahedron, the configuration pattern with the greatest spatial uniformity is the one where the minimum angle between any two of the eight auxiliary diagonal points and the origin is the largest. This minimum angle is 54.7° for the first pattern. Comparing the 54.7° of the first pattern with the 45° of the second pattern, the first pattern is larger than the second. Furthermore, the results of studying a third pattern including auxiliary diagonal points arranged on the axis of symmetry C2 show that, with eight auxiliary diagonal points, the minimum angle in the first pattern is the largest. Therefore, in Figure 10 In the example, the auxiliary diagonal points of pattern 1 are used. However, this is just an example; pattern 2 or pattern 3 could also be used. Alternatively, 16 auxiliary diagonal points including patterns 1 and 2 could be used, or a pattern including pattern 3 in addition to patterns 1 and 2 could be used. That is, a pattern including at least one of patterns 1 through 3 could be used.

[0127] Figure 11 It is Figure 10 The table summarizes the configurations of the vanishing points and auxiliary diagonal points. The labels indicate the types of vanishing points and auxiliary diagonal points. The directions are vectors representing the directions of the vanishing points and auxiliary diagonal points from the origin. The image coordinates are the coordinates of the vanishing points and auxiliary diagonal points in the panoramic image that serves as the projection source. Furthermore, the panoramic image is set to an equirectangular projection. W represents the width in the panoramic image, and H represents the height. The Xm, Ym, and Zm vectors shown in the direction column are the unit vectors of the Xm, Ym, and Zm axes, respectively.

[0128] For example, the vanishing point PF is located at (W / 2, H / 2) in the panoramic image, and its direction in world coordinate system 21 is represented by the Zm vector. The vanishing point PB is located at (0, H / 2) in the panoramic image, and its direction in world coordinate system 21 is represented by the -Zm vector. The vanishing point PL is located at (W / 4, H / 2) in the panoramic image, and its direction in world coordinate system 21 is represented by the -Xm vector. The vanishing point PR is located at (3W / 4, H / 2) in the panoramic image, and its direction in world coordinate system 21 is represented by the Xm vector. The vanishing point PT is located at (0, 0) in the panoramic image, and its direction in world coordinate system 21 is represented by the -Ym vector. The vanishing point PM is located at (0, H) in the panoramic image, and its direction in world coordinate system 21 is represented by the Ym vector.

[0129] For example, auxiliary diagonal point FLT is located at (3W / 8, H / 4) in the panoramic image, and its direction in world coordinate system 21 is represented by (Zm vector - Xm vector - Ym vector) / √3. Auxiliary diagonal point FRT is located at (5W / 8, H / 4) in the panoramic image, and its direction in world coordinate system 21 is represented by (Zm vector + Xm vector - Ym vector) / √3. Auxiliary diagonal point FLB is located at (3W / 8, 3H / 4) in the panoramic image, and its direction in world coordinate system 21 is represented by (Zm vector - Xm vector + Ym vector) / √3. Auxiliary diagonal point FRB is located at (5W / 8, 3H / 4) in the panoramic image, and its direction in world coordinate system 21 is represented by (Zm vector + Xm vector + Ym vector) / √3. Auxiliary diagonal point BLT is located at (W / 8, H / 4) in the panoramic image, and its direction in world coordinate system 21 is represented by (-Zm vector - Xm vector - Ym vector) / √3. The auxiliary diagonal point BRT is located at (7W / 8, H / 4) in the panoramic image, and its direction in world coordinate system 21 is represented by (-Zm vector + Xm vector - Ym vector) / √3. The auxiliary diagonal point BLB is located at (W / 8, 3H / 4) in the panoramic image, and its direction in world coordinate system 21 is represented by (-Zm vector - Xm vector + Ym vector) / √3. The auxiliary diagonal point BRB is located at (7W / 8, 3H / 4) in the panoramic image, and its direction in world coordinate system 21 is represented by (-Zm vector + Xm vector + Ym vector) / √3.

[0130] so, Figure 10 The six vanishing points and eight auxiliary diagonal points shown are respectively positioned in world coordinate system 21 at... Figure 11 direction shown.

[0131] also, Figure 10 , Figure 11 The vanishing points and auxiliary diagonals shown represent the reference vanishing point and reference auxiliary diagonals. The reference vanishing point is the ideal vanishing point projected onto the unit sphere 1000 when the camera coordinate system 22 is not rotated relative to the world coordinate system 21. The reference auxiliary diagonals are the ideal auxiliary diagonals projected onto the unit sphere 1000 when the camera coordinate system 22 is not rotated relative to the world coordinate system. Therefore, when the camera coordinate system 22 is rotated relative to the world coordinate system, the six vanishing points and eight auxiliary diagonals are projected to positions offset relative to the six reference vanishing points and eight reference auxiliary diagonals, respectively.

[0132] Return to reference Figure 8The vanishing point estimation unit 112B estimates the vanishing point and auxiliary diagonal points by inputting an image into a first learning model. The first learning model is a machine learning model that has been pre-processed for estimating the vanishing point and auxiliary diagonal points. Based on the input image, the first learning model outputs a heatmap of multiple pixels, each represented by the likelihood of the vanishing point. Furthermore, based on the input image, the first learning model outputs a heatmap of multiple pixels, each represented by the likelihood of the auxiliary diagonal points. The first learning model is generated through machine learning by using heatmaps representing the ground truth values ​​of the vanishing point and auxiliary diagonal points as training data. The vanishing point estimation unit 112B estimates the vanishing point based on the heatmap of the vanishing point output by the first learning model, and also estimates the vanishing point based on the heatmap of the auxiliary diagonal points output by the first learning model.

[0133] The projection unit 114B projects the vanishing point estimated by the vanishing point estimation unit 112B using the camera model shown in equations (1) and (2) onto the unit sphere 1000 configured in the world coordinate system 21.

[0134] The calculation unit 115B calculates the rotation angle representing the posture of the camera 2 based on the error between the vanishing point projected by the projection unit 114B and the reference vanishing point pre-projected onto the unit sphere 1000, and the error between the auxiliary diagonal point projected by the projection unit 114B and the reference auxiliary diagonal point pre-projected onto the unit sphere 1000. Furthermore, the error between the auxiliary diagonal point and the reference auxiliary diagonal point refers to the angle formed by the auxiliary diagonal point and the reference diagonal point at the origin of the world coordinate system 21.

[0135] Figure 9 This is a flowchart illustrating an example of the processing of the information processing device 1B in Embodiment 3. The processing of step S201 is... Figure 6 The same as S101. Next, in step S202, the vanishing point estimation unit 112B estimates the vanishing point and auxiliary diagonal points by inputting the image into the first learning model. The output of the first learning model is the same as... Figure 10 The six vanishing points shown correspond to six heatmaps, and the output is consistent with... Figure 10 The diagram shows eight heatmaps corresponding to the eight auxiliary diagonal points. The vanishing point estimation unit 112B detects peak values ​​in each of the six heatmaps corresponding to the vanishing point output from the first learning model. If a detected peak value is above a threshold, the vanishing point is determined to be estimated. Furthermore, if the peak values ​​of the eight heatmaps corresponding to the auxiliary diagonal points output from the first learning model are above the threshold, the vanishing point estimation unit 112B determines that an auxiliary diagonal point has been estimated. For example, if the peak value of the heatmap corresponding to the auxiliary diagonal point FRT is above the threshold, the auxiliary diagonal point FRT is determined to be estimated; if the peak value of the heatmap corresponding to the auxiliary diagonal point BLB is below the threshold, the auxiliary diagonal point BLB is determined not to be estimated.

[0136] The processing of step S203 and Figure 6 The same applies to step S103. Next, in step S204, the projection unit 114B applies the internal parameters estimated in step S203 to the camera model, and projects the vanishing point and auxiliary diagonal point estimated in step S202 onto the unit sphere 1000.

[0137] Next, in step S205, the calculation unit 115B calculates the error between the vanishing point projected onto the unit sphere 1000 and the corresponding reference vanishing point projected onto the unit sphere, and calculates the error between the auxiliary diagonal point projected onto the unit sphere 1000 and the corresponding reference auxiliary diagonal point projected onto the unit sphere 1000. For example, if the vanishing point PF and the auxiliary diagonal point FRT are estimated, the error between the vanishing point PF and the reference vanishing point of the vanishing point PF, and the error between the auxiliary diagonal point FRT and the reference vanishing point of the auxiliary diagonal point FRT are calculated.

[0138] Next, in step S206, the calculation unit 115B searches for the vanishing point and the rotation angle of the auxiliary diagonal point that minimize the error calculated in step S205. The details of this process are the same as in Embodiment 1, except that an auxiliary diagonal point is added in addition to the vanishing point; therefore, a detailed description is omitted. The processes in steps S207 and S208 are the same as in steps S107 and S108.

[0139] Thus, according to Embodiment 3, in addition to the vanishing point, auxiliary diagonal points are also estimated. Here, the auxiliary diagonal points consist of eight or more points that can maintain the symmetry of the six vanishing points corresponding to the vertices of the regular octahedron projected onto the unit sphere 1000. Therefore, the projected auxiliary diagonal points are evenly distributed in space and, like the vanishing point, become points with high geometric constraint. Thus, this structure can provide information that uniquely determines the camera's pose even if the vanishing point estimated from the image is insufficient.

[0140] (Implementation Method 4)

[0141] Implementation method 4 uses a learning model for skeleton estimation to generate a fully learned model. Figure 12 This is a diagram illustrating an example of the structure of the information processing apparatus 1C in Embodiment 4. Furthermore, in Embodiment 4, the same reference numerals are used for structural elements identical to those in Embodiments 1-3, and descriptions are omitted. In addition to the structure of Embodiment 3, the processor 110 of the information processing apparatus 1C further includes a learning unit 117. The completed learning model corresponds to the first learning model shown in Embodiments 2 and 3. The learning models before and during learning are referred to as the pre-learning model.

[0142] Learning unit 117 performs machine learning on the learning data using the pre-learning model, thereby generating a fully learned model. The learning data includes a learning image and a ground truth heatmap representing the ground truth values ​​of vanishing points and auxiliary diagonal points included in the learning image. The ground truth heatmap has... Figure 10 The six vanishing points shown correspond to six truth heatmaps, and the sum of the six truth heatmaps and the sum of the six truth heatmaps. Figure 10 The eight auxiliary diagonal points shown correspond to eight ground truth heatmaps.

[0143] Learning Unit 117 uses an error function to evaluate the difference between the estimated heatmap and the true heatmap output by inputting the learning image into the pre-learning model, enabling the pre-learning model to perform machine learning. The learning model output and... Figure 10 The six heatmaps corresponding to the six vanishing points are shown. The learning model output is... Figure 10 The eight heatmaps corresponding to the eight auxiliary diagonal points are shown.

[0144] The error function uses the vanishing points included in the estimated heatmap and those not included in the estimated heatmap for error evaluation. Furthermore, the error function also uses the vanishing points of auxiliary diagonal points included in the estimated heatmap and those not included in the estimated heatmap for error evaluation.

[0145] As a pre-learning model, HRNet, which is widely used for skeletal point estimation and is disclosed in document D4, can be adopted. In the machine learning of HRNet in skeletal point estimation, the pre-learning model outputs an estimation heatmap based on the number of skeletal points that are to be estimated. If the peak value of the estimation heatmap is above a threshold, it is determined that a skeletal point exists. The estimation heatmap sets each pixel to a value of 0 to 1, for example, and the threshold value is 0.8, for example. Furthermore, the error function of HRNet in skeletal point estimation calculates the squared error for each pixel of the ground truth heatmap and the estimation heatmap, and sums them to obtain the error evaluation value. At this time, in the machine learning of HRNet in skeletal point estimation, only the estimation heatmap with skeletal points exists has a squared error calculated, and the estimation heatmap without skeletal points does not have a squared error calculated. The error function in HRNet in skeletal point estimation is represented by equation (3).

[0146] Document D4: K. Sun, B. Xiao, D. Liu, and J. Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of IEEE / CVFConference on Computer Vision and Pattern Recognition (CVPR), pages 5686-5696, 2019.

[0147] [Mathematical Expression 2]

[0148]

[0149] Jpose is the evaluation value of the error function. N is the number of skeletal points identified as existing in the presumed heatmap, and i is the index of the skeletal point. M is the number of pixels in the presumed heatmap, and j is the index of the pixel. sj is the pixel value of the presumed heatmap, and sj' is the pixel value of the ground truth heatmap. ai is a Boolean value indicating whether the presumed heatmap includes skeletal points, which is "1" if included and "0" if not included.

[0150] In the estimation of human skeletal points, there are cases where the human is reflected in the image and cases where the human is not reflected. It is desirable to learn the reflected cases first, and Boolean value ai is used in equation (3).

[0151] However, the above error function is inappropriate when estimating the vanishing point in an image using HRNet. This is because, since there are cameras capturing images in all images, the vanishing point and auxiliary diagonal points should be considered in all images. However, since vanishing points and auxiliary diagonal points sometimes exist outside the image's range, they may not necessarily exist within the image. Therefore, in this embodiment, machine learning is performed on the pre-learning model using an error function that does not use the Boolean value ai. The error function in embodiment 4 is expressed as equation (4).

[0152] Jvp is the evaluation value, N is the number of vanishing points and auxiliary diagonal points identified in the estimated heatmap, and i is the index of the vanishing point and auxiliary diagonal point. M, j, sj, and sj' are the same as in equation (3). As shown in equation (4), the Boolean value ai is omitted in this embodiment. Therefore, regardless of whether the estimated heatmap includes vanishing points and auxiliary diagonal points, the vanishing points and auxiliary diagonal points are used for error evaluation, thus obtaining a learned model with high estimation accuracy of vanishing points and auxiliary diagonal points.

[0153] The learning unit 117 corrects the parameters of the model before learning so that the error shown in equation (4) is minimized, and generates the model after learning.

[0154] The learning data is generated as follows. First, the learning unit 117 acquires a panoramic image captured by a calibrated camera. The panoramic image captured by the calibrated camera can be converted into any image, such as a fisheye image or a distortion-free image. Next, the learning unit 117 determines the camera model. An example of a camera model is a fisheye camera model with equidistant projection. Next, the learning unit 117 determines camera parameters related to pan, tilt, roll, focal length, and lens distortion using random numbers. Next, the learning unit 117 sets these camera parameters to true values ​​and generates a learning image captured by the camera model with these camera parameters based on the panoramic image. This process uses general image processing techniques, such as OpenCV's remap function. Next, the learning unit 117 obtains the following based on the image coordinates of the panoramic image: Figure 11 The vanishing point and auxiliary diagonal points are labeled. Next, the learning unit 117 generates a binary image with the pixel value of the vanishing point in the learning image set to 1 and all other pixel values ​​set to 0. Next, the learning unit 117 applies a two-dimensional Gaussian filter to the binary image, thereby generating a ground truth heatmap image. The standard deviation of the Gaussian filter is, for example, 2 pixels. The peak value after the filter application becomes smaller than 1. Therefore, the learning unit 117 sets the pixel values ​​of all pixels in the ground truth heatmap to a constant multiple, making the peak value 1. Learning data is generated through the above process. The generated learning data is stored in the memory 120.

[0155] Figure 13 This is a flowchart illustrating an example of the processing of the information processing apparatus 1C in Embodiment 4. First, in step S301, the learning unit 117 retrieves learning data from the memory 120. Next, in step S302, the learning unit 117 inputs the learning images included in the learning data into the pre-learning model, thereby generating an estimated heatmap. For example, generating a heatmap with... Figure 10 The diagram shows a total of 14 heatmaps: 6 estimated heatmaps corresponding to the 6 vanishing points and 8 estimated heatmaps corresponding to the 8 auxiliary diagonal points. However, this is just an example; it's not necessary to use all 6 vanishing points and 8 auxiliary diagonal points for learning. For example, at least 1 vanishing point and at least 1 auxiliary diagonal point can be used for learning.

[0156] Next, in step S303, the learning unit 117 uses equation (4) to calculate the evaluation value of the error between the estimated heatmap and the true heatmap.

[0157] Next, in step S304, the parameters of the model before learning are corrected to reduce the evaluation value. This correction can be achieved, for example, by applying backpropagation.

[0158] Next, the learning unit 117 determines whether the machine learning has reached the termination condition. If the termination condition has been met (yes in step S305), the process ends. A fully learned model is then generated. The generated fully learned model is stored in the memory 120. On the other hand, if the termination condition has not been met (no in step S305), the process returns to step S301, and the processing after step S301 is repeated. The termination condition may be, for example, that the number of learning iterations has reached a given number.

[0159] Thus, according to Embodiment 4, in the pre-learning model used for skeleton estimation, a learned model is generated by performing machine learning with heatmaps representing the ground truth values ​​of the vanishing point and auxiliary diagonals as training data, and the vanishing point is estimated using this learned model. Therefore, a learned model is obtained that can accurately estimate the vanishing point and auxiliary diagonals using the learned model for skeleton point estimation. As a result, the vanishing point and auxiliary diagonals can be estimated with high accuracy.

[0160] (Modification 1 of Implementation Method 4)

[0161] The learned model is composed of HRNet, but this disclosure is not limited to this. Any learning model can be used to infer key points such as skeleton points.

[0162] (Modification 2 of Implementation Method 4)

[0163] A learned model can also be a learned model that has undergone machine learning so that only the vanishing point is estimated.

[0164] (Factors of the effects of this disclosure)

[0165] This invention demonstrates that, compared to previous camera calibration methods, this disclosure offers high precision, high robustness, and low computational cost.

[0166] Non-patent document 1 and non-patent document 2 disclose geometric methods for estimating the pan / tilt / roll angles.

[0167] In conventional camera calibration methods, the information available for camera calibration is limited to the vanishing point of a geometry-based circular arc detector. However, this disclosure utilizes a deep neural network to leverage auxiliary diagonal points that conventional geometry-based circular arc detectors cannot extract. Unlike vanishing points, auxiliary diagonal points cannot be detected using geometry-based methods, but because they possess geometric significance such as diagonal direction, they can be detected using a deep neural network. Therefore, the proposed method, which utilizes more vanishing points and auxiliary diagonal points, achieves higher accuracy in camera calibration than previous methods.

[0168] Furthermore, conventional camera calibration methods require iterative optimization using random numbers to estimate the vanishing point from multiple arcs, resulting in high computational costs. In contrast, this disclosure can estimate pan, tilt, and roll angles without repeated calculations, thus reducing computational costs compared to conventional camera calibration methods.

[0169] Furthermore, the method of sensing vanishing points and auxiliary diagonal points based on heatmaps is more robust than the previous learning-based method disclosed in document D5 below, which does not use heatmaps.

[0170] Document D5: M. Lopez-Antequera, R. Mari, P. Gargallo, Y. Kuang, J. Gonzalez-Jimenez, and G. Haro. Deep single image camera calibration with radialdistortion. In Proceedings of IEEE / CVF Conference on Computer Vision andPattern Recognition (CVPR), pages 11809-11817, 2019.

[0171] Previous methods based on regression without heatmaps to estimate rotation angles focus on learning large areas of the sky and roads in the image, making it difficult to distinguish between overcast skies and gray roads. While the sky and roads occupy a large area in the image, it is difficult to extract geometric features from them. On the other hand, the method disclosed in this paper, which estimates vanishing points and auxiliary diagonal points based on heatmaps, provides geometric feature points with strong constraints in camera calibration and achieves high robustness.

[0172] Industrial availability

[0173] This disclosure can be utilized in the field of estimating the posture of a moving body.

Claims

1. An information processing device, comprising: The acquisition unit acquires images captured by the camera; and The estimation unit estimates the vanishing point by inputting the image into the learned model. The learned model is generated by performing machine learning on the learning model used in skeleton inference, using heatmaps representing the truth values ​​of the vanishing points as training data.

2. The information processing apparatus according to claim 1, wherein, The estimation unit further estimates auxiliary diagonal points based on the image. The auxiliary diagonal point is any point among a group of eight or more points positioned to maintain symmetry with a regular octahedron inscribed in the unit sphere when projected onto a unit sphere in the world coordinate system.

3. The information processing apparatus according to claim 1 or 2, wherein, The learned model is then used for machine learning using an error function that evaluates the error between the inferred heatmap output from the learned model and the truth heatmap representing the true value of the vanishing point. The error function uses the vanishing points included in the estimated heatmap and the vanishing points not included in the estimated heatmap to evaluate the error.

4. The information processing apparatus according to claim 3, wherein, The learned model is then used for machine learning using an error function that evaluates the error between the inferred heatmap output by the learned model and the ground truth heatmap representing the vanishing points and the ground truth values ​​of the auxiliary diagonal points. In the error function, The vanishing points included in the estimated heatmap and the vanishing points not included in the estimated heatmap are used to evaluate the error. The auxiliary diagonal points included in the estimated heatmap and those not included in the estimated heatmap are used to evaluate the error.

5. The information processing apparatus according to claim 1 or 2, wherein, The information processing device also includes: The projection unit projects the estimated vanishing point onto a unit sphere configured in the world coordinate system based on the internal parameters of the camera. and The computation unit calculates a rotation angle representing the camera's posture based on the error between the projected vanishing point and a reference vanishing point pre-projected onto the unit sphere.

6. An information processing method, specifically an information processing method in a computer. In the information processing method, Acquire images captured by the camera. The vanishing point is estimated by inputting the image into the learned model. The learned model is generated by performing machine learning on the learning model used in skeleton inference, using heatmaps representing the truth values ​​of the vanishing points as training data.

7. An information processing program that enables a computer to function as the information processing apparatus of claim 1 or 2.

Images