How to update map information

Abstract

This map information update method comprises: a map information acquisition step (S20) for acquiring map information; an observation data acquisition step (S30) for acquiring landmark observation positions indicating the positions of one or more landmarks in a captured image; an additional step (S40) for generating additional map information by adding information related to the landmark observation position to the map information and updating the map information acquired in the map information acquisition step (S20) to the additional map information; a prediction step (S50) for calculating, by using a prediction inference engine including a trained prediction neural network, prediction map information, on the basis of the map information updated in the additional step (S40), and updating the map information to prediction map information; and an update step (S60) for calculating updated map information for the map information updated in the prediction step (S50) by using a gradient method, and updating the map information to the updated map information.

Description

[Technical Field]

[0001] The present disclosure relates to a map information updating method, a landmark generating method, and a feature point distribution adjusting method. [Background technology]

[0002] A conventional technique known as VSLAM (Visual Simultaneous Localization and Mapping) is to capture an image using a camera and simultaneously estimate the position of the camera and the positions of surrounding landmarks from information contained in the captured image.

[0003] The core processing in VSLAM technology is to calculate the reprojection error, which is the error between the position of a photographed landmark in the photographed image and the reprojected position, which is the position in the photographed image calculated from the assumed camera pose (i.e., the camera position and orientation) and the assumed landmark position, and to calculate the camera pose and landmark position that make the reprojection error zero (in reality, the error will be sufficiently small). Information that combines the camera pose and landmark position is called map information.

[0004] Searching for map information that reduces the reprojection error to zero is called bundle adjustment, and is a process generally classified as an optimization problem of the nonlinear least squares method. For this reason, bundle adjustment requires a process of minutely correcting the map information so that the reprojection error decreases, and repeating this correction until the value of the reprojection error converges (see, for example, Patent Document 1, Non-Patent Document 1, and Non-Patent Document 2). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Special Publication No. 2017-503290 [Non-patent literature]

[0006] [Non-Patent Document 1] Yuki Iwamoto, Yasuyuki Sugaya, Kenichi Kanaya, Implementation and Evaluation of Bundle Adjustment for 3D Reconstruction, IPSJ Technical Report, 2011-CVIM-175-19, pp.1-8, 2011 [Non-patent document 2] Takayuki Okatani, Bundle Adjustment, Information Processing Society of Japan Technical Report, 2009-CVIM167-37, pp.1-16, 2009 Summary of the Invention [Problem to be solved by the invention]

[0007] In VSLAM, every time a new image (i.e., a keyframe) is added, the reprojection error is calculated and the map information is updated through bundle adjustment. This process is essential to maintain the accuracy of the map information.

[0008] Gradient-based algorithms are commonly used as algorithms for achieving convergence during bundle adjustment. One well-known example of a gradient-based algorithm is an algorithm that combines the steepest descent method and the Gauss-Newton method. This algorithm uses the steepest descent method to perform correction until the reprojection error approaches a minimum value, after which it switches to the Gauss-Newton method. This type of algorithm requires computationally intensive processing for each correction iteration, including the generation of an approximate Hessian matrix and the calculation of the correction amount by solving simultaneous equations. The inclusion of such computationally intensive processing poses a problem when using VSLAM technology.

[0009] The present disclosure has been made to solve such problems, and aims to provide a map information updating method and the like that can reduce the amount of calculations. [Means for solving the problem]

[0010] In order to achieve the above object, a map information updating method according to one embodiment of the present disclosure includes: a map information acquisition step of acquiring map information including estimated positions of a camera and one or more landmarks in a first coordinate system; an observation data acquisition step of acquiring landmark observation positions indicating the positions of the one or more landmarks in a second coordinate system in an image captured by the camera; an addition step of generating additional map information by adding information related to the landmark observation positions to the map information acquired in the map information acquisition step, and updating the map information acquired in the map information acquisition step to the additional map information; a prediction step of calculating predicted map information based on the map information updated in the addition step using a predictive inference engine including a trained predictive neural network, and updating the map information updated in the addition step to the predicted map information; and an update step of calculating updated map information using a gradient method on the map information updated in the prediction step, and updating the map information updated in the prediction step to the updated map information.

[0011] In order to achieve the above-mentioned object, a map information updating method according to one embodiment of the present disclosure includes: a map information acquisition step of acquiring map information including estimated positions of a camera and one or more landmarks in a first coordinate system; an observation data acquisition step of acquiring landmark observation positions indicating the positions of the one or more landmarks in a second coordinate system in an image captured by the camera; an addition step of generating additional map information by adding information related to the landmark observation positions to the map information acquired in the map information acquisition step and updating the map information acquired in the map information acquisition step to the additional map information; an inference step of calculating inferred map information by updating the map information updated in the addition step using an update inference engine including a trained update neural network and updating the map information updated in the addition step to the inferred map information; and an update step including a step of calculating updated map information using a gradient method on the map information updated in the inference step and updating the map information updated in the inference step to the updated map information.

[0012] In order to achieve the above object, a map information updating method according to one embodiment of the present disclosure includes a map information acquisition step of acquiring map information including estimated positions of a camera and one or more landmarks in a first coordinate system; an observation data acquisition step of acquiring landmark observation positions indicating positions of the one or more landmarks in a second coordinate system in an image captured by the camera; an addition step of generating additional map information by adding information related to the landmark observation positions to the map information acquired in the map information acquisition step, and updating the map information acquired in the map information acquisition step to the additional map information; an updating step of calculating updated map information using a gradient method on the map information updated in the adding step and updating the map information updated in the adding step to the updated map information; and a number determination step of determining an upper limit to the number of iterations of the updating step based on the amount of change estimated in the change estimation step, wherein the reprojection error is calculated by a reprojection error function that calculates the error between the landmark observation position and a reprojection position on the captured image that corresponds to the landmark observation position and is calculated based on the map information.

[0013] In order to achieve the above object, a landmark generation method according to one embodiment of the present disclosure is a landmark generation method that generates landmarks by performing triangulation based on a first captured image and a second captured image taken by a camera, and includes: a feature point extraction step of extracting a first feature point included in the first captured image and a second feature point included in the second captured image that is a target for matching with the first feature point; a nearby feature point extraction step of extracting a third feature point included in the first captured image that is a feature point close to the first feature point, and a fourth feature point included in the second captured image that is a target for matching with the third feature point; an error probability prediction step of predicting an error probability of matching between the first feature point and the second feature point based on information of the first feature point, the second feature point, the third feature point, and the fourth feature point; and a determination step of determining whether to generate a landmark based on the first feature point and the second feature point based on the error probability.

[0014] In order to achieve the above object, a feature point distribution adjustment method according to one embodiment of the present disclosure is a feature point distribution adjustment method for adjusting the distribution of feature points corresponding to one or more landmarks included in a captured image captured by a camera, and includes: a first extraction step of extracting a plurality of first feature points corresponding to the one or more landmarks included in the captured image; a second extraction step of extracting a feature point group constituted by a plurality of second feature points from the plurality of first feature points; and a feature point adjustment step of performing at least one of deleting one or more second feature points included in the feature point group based on the distribution of the plurality of second feature points in the captured image and adding a third feature point based on the plurality of second feature points. [Effects of the Invention]

[0015] The present disclosure can provide a map information update method that can reduce the amount of calculation. [Brief explanation of the drawings]

[0016] [Figure 1]FIG. 1 is a flowchart showing the flow of the map information updating method according to the first embodiment. [Figure 2] FIG. 2 is a schematic graph showing the relationship between map information and reprojection error in bundle adjustment. [Figure 3] FIG. 3 is a schematic graph for explaining an overview of prediction map information according to the first embodiment. [Figure 4] FIG. 4 is a flowchart showing the flow of the map information updating method according to the second embodiment. [Figure 5] FIG. 5 is a flowchart showing the flow of the map information updating method according to the third embodiment. [Figure 6] FIG. 6 is a schematic diagram for explaining the landmark generation method. [Figure 7] FIG. 7 is a flowchart showing the flow of the landmark generating method according to the fourth embodiment. [Figure 8] FIG. 8 is a schematic diagram for explaining the third and fourth features according to the fourth embodiment. [Figure 9] FIG. 9 is a flowchart showing the flow of the feature point distribution adjustment method according to the fifth embodiment. [Figure 10] FIG. 10 is a schematic diagram showing a first example of a plurality of first feature points according to the fifth embodiment. [Figure 11] FIG. 11 is a schematic diagram showing a second example of a plurality of first feature points according to the fifth embodiment. [Figure 12] FIG. 12 is a schematic diagram showing a first example of a feature point group configured with a plurality of second feature points according to the fifth embodiment. [Figure 13] FIG. 13 is a schematic diagram showing a second example of a feature point group configured with a plurality of second feature points according to the fifth embodiment. [Figure 14] FIG. 14 is a schematic diagram showing an example of adding a third feature point according to the fifth embodiment. [Figure 15] FIG. 15 is a schematic diagram showing an example of deleting the second feature point according to the fifth embodiment. [Figure 16]FIG. 16 is a diagram showing an example of the hardware configuration of a computer for executing the methods according to the embodiments by software. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, materials, specifications, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept of the present disclosure will be described as optional components. Furthermore, each drawing is not necessarily an exact illustration. In each drawing, substantially identical components are assigned the same reference numerals, and duplicated descriptions may be omitted or simplified.

[0018] (Embodiment 1) A description will be given of a map information updating method according to embodiment 1. The map information updating method according to this embodiment is a method used in, for example, VSLAM technology, which takes an image using a camera and simultaneously estimates the position of the camera and the positions of surrounding landmarks from information contained in the captured image.

[0019] In VSLAM technology, first, a three-dimensional coordinate system is assumed as a first coordinate system, and a camera is placed in the first coordinate system. The first coordinate system is a coordinate system fixed with respect to the space in which the camera and other devices are placed, and is also referred to as a world coordinate system. Landmarks are generated from images captured by the camera. Here, landmarks are three-dimensional points created in the first coordinate system. For example, landmarks are generated by triangulation based on corresponding feature points included in each of two captured images. In a map information update method according to this embodiment, map information including estimated positions of the camera and one or more landmarks in the first coordinate system is updated. Specifically, the map information according to this embodiment is updated when information on an image captured by a camera is added to the map information.

[0020] The map information update method according to this embodiment will be described below with reference to Fig. 1. Fig. 1 is a flowchart showing the flow of the map information update method according to this embodiment.

[0021] As shown in Fig. 1, in the map information updating method according to this embodiment, first, map information is acquired (S20). The map information includes at least estimated positions of the camera and one or more landmarks in a first coordinate system. Note that the map information may include information other than the estimated positions of the camera and one or more landmarks in the first coordinate system. For example, the map information may include information about the orientation of the camera in the first coordinate system.

[0022] Next, landmark observation positions are acquired (S30), which indicate the positions of one or more landmarks in a second coordinate system in the captured image captured by the camera. The second coordinate system is a coordinate system fixed with respect to the captured image and is also called a keyframe coordinate system. The landmark observation positions are the positions in the second coordinate system of feature points corresponding to the positions of one or more landmarks.

[0023] Next, additional map information is generated by adding information about landmark observation positions to the map information acquired in step S20, and the map information is updated to the additional map information (S40). Information about landmark observation positions is, for example, estimated positions of landmarks generated based on feature points included in the captured image. Information about landmark observation positions may also be information that is simply estimated from the camera position, etc.

[0024] After step S40, a predictive inference engine including a trained predictive neural network is used to calculate predicted map information based on the map information updated in step S40, and the map information updated in step S40 is updated to the predicted map information (S50). Here, a method for calculating the predicted map information will be described. When updating the map information, bundle adjustment is generally performed. That is, map information that will result in a reprojection error of zero is searched for. In this embodiment, the reprojection error is calculated using a reprojection error function, which is a function that calculates the error between a landmark observation position and a reprojection position on a captured image that corresponds to the landmark observation position and is calculated based on the map information. Note that the reprojection error may include a sum of one or both of an error calculated using the reprojection error function for multiple landmarks included in the map information and an error calculated using the reprojection error function for each of the multiple captured images.

[0025] Here, an overview of bundle adjustment will be explained using Fig. 2. Fig. 2 is a schematic graph showing the relationship between map information and reprojection error in bundle adjustment. The horizontal axis of Fig. 2 shows a quantity that is a schematic representation of the map information as a single variable, and the vertical axis shows the reprojection error relative to the map information.

[0026] As shown in Figure 2, bundle adjustment corrects the map information before bundle adjustment to map information that minimizes the reprojection error. For example, when performing bundle adjustment using a gradient-based algorithm, such as an algorithm that combines the steepest descent method and the Gauss-Newton method, map information that minimizes the reprojection error is searched for by repeatedly correcting the map information before bundle adjustment with small correction amounts and calculating the reprojection error after correction. Note that, below, the map information that minimizes the reprojection error is also referred to as the map information solution. Here, in algorithms using gradient methods, each correction iteration involves computationally intensive processes, such as generating an approximate Hessian matrix and calculating the correction amount by solving simultaneous equations using the nonlinear least squares method. In particular, when the error between the map information before bundle adjustment and the map information solution is large, multiple iterations are required.

[0027] In contrast to such conventional techniques, in this embodiment, the amount of calculation is reduced by calculating prediction map information using a predictive inference engine. An outline of a calculation method for prediction map information according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a schematic graph for explaining the outline of prediction map information according to this embodiment.

[0028] As shown in FIG. 3, in this embodiment, a predictive inference engine is used to calculate, as predicted map information, map information in which the reprojection error is close to the minimum value based on map information. This makes it possible to obtain map information in which the reprojection error is close to the minimum value without repeating calculations using an algorithm that uses a gradient method. The predictive neural network included in such a predictive inference engine is a trained neural network that uses training map information as input and has trained using updated training map information as training data. The training map information is not particularly limited as long as it is information similar to the additional map information used in the map information updating method according to this embodiment.

[0029] The updated map information for learning is generated based on the map information for learning and is map information that reduces the reprojection error calculated using a reprojection error function. Here, the reprojection error function is a function that calculates the error between a landmark observation position and a reprojection position on the captured image that corresponds to the landmark observation position and is calculated based on the map information. Specifically, the reprojection error function can be a well-known function such as that described in Non-Patent Document 1.

[0030] The updated map information for learning can be obtained, for example, by actually performing bundle adjustment on the learning landmark observation positions and the learning map information using a gradient method. Note that the map information that reduces the reprojection error may be, for example, map information that minimizes the reprojection error. The map information that minimizes the reprojection error is not limited to map information that strictly minimizes the reprojection error, but also includes map information that generally minimizes the reprojection error. For example, map information in which the difference between the reprojection error of the map information and the minimum value of the reprojection error is 5% or less of the minimum value is also included in the map information that minimizes the reprojection error.

[0031] By performing the above learning, the predictive neural network according to this embodiment learns the shape of the error function that indicates the relationship between map information and reprojection error. Learning the predictive neural network is a process equivalent to fitting to the error function. By learning the shape of the error function, the predictive neural network becomes able to predict map information that minimizes the reprojection error. Note that while information such as the camera position contained in the map information changes depending on the map information, the error function learned by the predictive neural network does not change. Furthermore, the predicted map information does not have to be map information that minimizes the reprojection error.

[0032] Note that a step may be added to prevent the predicted map information calculated using the predictive inference engine from moving away from the solution of the map information (i.e., the difference between the predicted map information and the solution of the map information becomes larger than the difference between the additional map information and the solution of the map information). For example, an inference engine may be prepared in advance to predict a correction direction for the additional map information that will bring it closer to the solution of the map information, and the inference engine may determine whether the predicted map information is closer to the solution of the map information than the additional map information.

[0033] In this embodiment, at least a portion of the calculations, such as the solution calculations of simultaneous equations in the gradient method of the prior art, can be replaced with inference using a predictive inference engine. Therefore, in this embodiment, the amount of calculation can be reduced compared to the gradient method, and the parallelism of calculations can be increased. Therefore, in this embodiment, effects such as faster map information updates and lower power consumption can be achieved. Furthermore, inference using a predictive inference engine can also reduce calculation accuracy. Therefore, it is also possible to simplify the hardware configuration of a computer or the like for executing the map information update method.

[0034] 1, updated map information is calculated using the gradient method for the map information updated in step S50, and the map information updated in step S50 is updated to the updated map information (S60). In other words, as in the conventional technology, the map information is updated using an algorithm that combines the steepest descent method and the Gauss-Newton method, for example, to bring the map information closer to the solution of the map information.

[0035] Next, the re-projection error for the map information is calculated (S70). Specifically, the re-projection error for the map information is calculated using the re-projection error function described above.

[0036] Subsequently, based on the reprojection error calculated using the reprojection error function for the map information updated in step S60, it is determined whether the update of the map information updated in step S60 converges. Based on the result of the determination, it is decided whether to return to the prediction step or the update step, or to end the update of the map information updated in step S60 (S80). For example, when the change amount ΔE of the reprojection error from the previous determination (at the first determination, it is the change amount from the reprojection error for the predicted map information) is smaller than a predetermined convergence threshold Sc (ΔE < Sc in S80), it is determined that the solution of the map information has been obtained, and the update of the map information is ended. Also, when the reprojection error E is larger than the upper limit value Su (E > Su in S80), it is determined that the predicted map information is inappropriate, and the process returns to step S50 to calculate the predicted map information again. Further, when the change amount ΔE of the reprojection error from the previous determination is greater than or equal to the convergence threshold Sc and the reprojection error E is less than or equal to the upper limit value Su (ΔE ≥ Sc, E ≤ Su in S80), the process returns to step S60, and the update of the map information using the gradient method is performed again.

[0037] By the map information update method as described above, the amount of calculation required for map information update can be reduced compared to the case of using the conventional technology.

[0038] (Embodiment 2) The map information update method according to Embodiment 2 will be described. The map information update method according to this embodiment is mainly different from the map information update method according to Embodiment 1 in that at least a part of the step of calculating the correction amount of the map information using the gradient method is replaced with the step of obtaining the correction amount by inference. Hereinafter, the map information update method according to this embodiment will be described mainly focusing on the differences from the map information update method according to Embodiment 1 with reference to FIG. 4. FIG. 4 is a flowchart showing the flow of the map information update method according to this embodiment.

[0039] As shown in FIG. 4, the map information update method according to this embodiment is the same as the map information update method according to Embodiment 1 up to step S40.

[0040] Following step S40, the map information updated in step S40 is updated (S150). Specifically, an updating inference engine including a trained updating neural network is used to calculate inferred map information by updating the map information updated in step S40, and the map information updated in step S40 is updated to inferred map information (S152). The updating neural network receives, for example, a set of coefficients of simultaneous equations using a nonlinear least-squares method to calculate correction amounts for the map information updated in step S40 as input, and learns the solutions of the simultaneous equations as training data. That is, instead of solving simultaneous equations using a nonlinear least-squares method to calculate correction amounts for the map information as in the conventional technology, an updating inference engine receives the set of coefficients of the simultaneous equations as input and infers the correction amounts for the map information to calculate correction amounts for the map information, and calculates inferred map information from the correction amounts. This allows at least a portion of the calculations, such as the solution calculation of simultaneous equations using a nonlinear least-squares method as in the conventional technology, to be replaced by inference using the updating inference engine. Therefore, in this embodiment, the amount of calculation can be reduced and the degree of parallelism of calculations can be increased compared to the gradient method. Therefore, in this embodiment, effects such as faster map information update and lower power consumption can be obtained. Furthermore, inference using an update inference engine can also reduce calculation accuracy. Therefore, it is possible to simplify the hardware configuration of a computer or the like for executing the map information update method.

[0041] The updating neural network according to this embodiment learns the slope of the error function that indicates the relationship between map information and reprojection error. The learning of the updating neural network corresponds to pattern matching to the local slope of the error function. In other words, the updating neural network can calculate an amount equivalent to the correction amount of the map information. Information such as the camera position contained in the map information changes depending on the map information, but the error function learned by the updating neural network does not change. Furthermore, the inferred map information does not have to match the map information obtained by solving simultaneous equations using the nonlinear least squares method.

[0042] It is also possible to add a step of preventing the inference map information calculated using the update inference engine from moving away from the solution of the map information (i.e., preventing the difference between the inference map information and the solution of the map information from becoming larger than the difference between the additional map information and the solution of the map information). For example, it is possible to predict a correction direction for the additional map information that will bring it closer to the solution of the map information, and determine whether the inference map information is closer to the solution of the map information than the additional map information.

[0043] Although the above example shows a case where the coefficients of a system of simultaneous equations are input to the updating neural network and the solutions of the system of equations are used as training data, the training method is not limited to this. For example, map information may be input to the updating neural network, and updated map information calculated using a gradient method or the like may be used as training data.

[0044] Next, the re-projection error for the map information updated in step S152 is calculated using the update inference engine (S154). Specifically, the re-projection error for the map information updated in step S152 is calculated using the re-projection error function described above.

[0045] Next, it is determined whether an iteration end condition for ending the iteration of step S152 has been met (S156). The iteration end condition may be, for example, that the number of updates of the map information using the update inference engine reaches a predetermined number. Alternatively, the iteration end condition may be that the reprojection error converges to a predetermined degree.

[0046] If it is determined in step S156 that the iteration end condition is not satisfied (No in S156), the process returns to step S152 again.

[0047] On the other hand, if it is determined in step S156 that the iteration termination condition is met (Yes in S156), as in step S60 in embodiment 1, updated map information is calculated using the gradient method on the map information updated in step S152, and the map information updated in step S152 is updated to the updated map information (S160).

[0048] Next, the re-projection error for the map information updated in step S160 is calculated using the gradient method (S162). Specifically, the re-projection error for the map information is calculated using the re-projection error function described above.

[0049] Next, the convergence of the map information update is determined based on the reprojection error calculated using a reprojection error function for the map information updated in step S160, and based on the result of the determination, it is determined (S170) whether to return to the update step (S150) or terminate the update of the map information updated in step S160. For example, in step S170, it is determined whether the change amount ΔE in the reprojection error since the previous determination (at the time of the first determination, the change amount from the reprojection error for the predicted map information) is smaller than a predetermined convergence threshold Sc. If the change amount ΔE is smaller than the predetermined convergence threshold Sc (Yes in S170), it is determined that a solution for the map information has been found, and the update of the map information is terminated. On the other hand, if the change amount ΔE is equal to or greater than the convergence threshold Sc (No in S170), the process returns to the update step S150 and updates the map information again.

[0050] By using the map information updating method described above, the amount of calculation required to update the map information can be reduced compared to when using the conventional technology.

[0051] The map information update methods according to this embodiment and embodiment 1 may be combined. For example, instead of steps S60 and S70 according to embodiment 1, update step S150 according to this embodiment may be used. That is, in the map information update method according to embodiment 1, after step S50, an updating inference engine including a trained updating neural network may be used to calculate inference map information by updating the map information updated in step S50, and the map information updated in step S50 may be updated to inference map information. This further reduces the amount of calculation in the map information update method.

[0052] (Embodiment 3) A map information updating method according to the third embodiment will be described. The map information updating method according to this embodiment determines an upper limit on the number of iterations for updating map information based on the amount of change in reprojection error due to bundle adjustment. The map information updating method according to this embodiment will be described below with reference to FIG. 5, focusing on the differences from the map information updating method according to the first embodiment. FIG. 5 is a flowchart showing the flow of the map information updating method according to this embodiment.

[0053] As shown in FIG. 5, the map information update method according to this embodiment is the same as the map information update method according to the first embodiment up to step S40.

[0054] Following step S40, the amount of change in reprojection error due to bundle adjustment for the map information updated in step S40 is estimated (S250). The amount of change in reprojection error due to bundle adjustment for map information is the amount of change obtained by comparing the reprojection error for map information before bundle adjustment with the reprojection error for map information after bundle adjustment when bundle adjustment is performed on the map information. In this embodiment, this amount of change is obtained by estimation. Specifically, the amount of change is calculated using an inference engine including a trained neural network. The neural network receives training map information as input and learns the difference between the reprojection error for the training map information and the reprojection error for adjusted map information obtained by performing bundle adjustment on the training map information as training data. Note that the training data may be calculated from the difference between each element included in the training map information and each element included in the adjusted map information.

[0055] The amount of change in the re-projection error estimated in this way corresponds to the number of iterations of the map information update process that must be performed until a solution for the map information is obtained. In other words, as the amount of change in the re-projection error decreases, the number of iterations of the map information update process that must be performed until a solution for the map information is obtained decreases.

[0056] Following step S250, an upper limit on the number of iterations of the map information update step S280 (described later) is determined based on the amount of change in the reprojection error estimated in step S250 (S260). As described above, the amount of change in the reprojection error corresponds to the number of iterations of the map information update process that must be performed until a solution for the map information is obtained. Therefore, by determining an upper limit on the number of iterations of the update step S280 based on the amount of change in the reprojection error, it is possible to ensure that the update step S280 is performed the number of times necessary to obtain a solution for the map information, while preventing the update step S280 from being performed more than necessary. In this way, in this embodiment, the amount of calculation required for iterating the update step S280 can be reduced.

[0057] The upper limit of the number of iterations determined here is appropriately determined according to the amount of change in the reprojection error. Specifically, in step S260, the upper limit of the number of iterations is reduced as the amount of change in the reprojection error decreases. Also, in step S260, if the amount of change in the reprojection error is smaller than a predetermined threshold, the upper limit of the number of iterations may be set to 0. In fact, when adding information about a captured image to map information, if the camera position, etc., is estimated relatively accurately, it is not uncommon for the reprojection error to be minimized without updating the map information. Therefore, by determining the upper limit of the number of iterations in this manner, the amount of calculation required to update the map information can be reduced.

[0058] Next, it is determined whether or not to terminate the iteration of the map information update step S280 (described later) (S270). In this embodiment, the iteration is terminated when the upper limit of the number of iterations determined in step S260 is exceeded, or when it is determined that the reprojection error calculated in the update step S280 (described later) has converged. As a method for determining convergence of the reprojection error, for example, a method can be used in which convergence is determined based on whether the amount of change in the reprojection error before and after executing the update step S280 is equal to or less than a predetermined threshold.

[0059] If it is determined in step S270 that the iteration should be ended (Yes in S270), the updating of the map information is ended. On the other hand, if it is determined in step S270 that the iteration should be continued (No in S270), updated map information is calculated using the gradient method on the map information updated in step S40, as in step S60 according to the first embodiment, and the map information updated in step S40 is updated to the updated map information (S280).

[0060] Next, the re-projection error for the updated map information is calculated using the gradient method (S290). Specifically, the re-projection error for the map information is calculated using the re-projection error function described above.

[0061] Next, the process returns to step S270, where it is determined whether or not to end the repetition of the map information update step S280. From this point on, the same process is repeated until it is determined in step S270 that the repetition should be ended. In other words, until it is determined in step S270 that the repetition should be ended, the process of calculating updated map information using the gradient method on the map information updated in the previous step S280, and updating the map information updated in the previous step S280 to the updated map information is repeated.

[0062] The map information updating method described above can reduce the amount of calculation required to repeat the updating step S280.

[0063] (Fourth embodiment) A description will be given of a landmark generation method according to embodiment 4. The landmark generation method according to this embodiment is a method that can be used in the map information update methods according to the above embodiments.

[0064] In VSLAM technology, landmarks are generated based on images captured by a camera. The landmark generation method will be explained below with reference to FIG. 6. FIG. 6 is a schematic diagram for explaining the landmark generation method.

[0065] As shown in FIG. 6, in the landmark generation method, a landmark Lm is generated by triangulation based on a first captured image F1 and a second captured image F2 captured by a camera C0. Specifically, a first feature point P1 included in the first captured image F1 is extracted, and a second feature point P2 included in the second captured image F2 and corresponding to the first feature point P1 is extracted. Here, the orientation of the camera C0 when the second captured image F2 was captured is different from the orientation of the camera C0 when the first captured image F1 was captured. Furthermore, the second feature point P2 corresponding to the first feature point P1 refers to a captured point on the same object as the point on the object indicated by the first feature point P1. Assume that the first feature point P1 included in the first captured image F1 and the second feature point P2 included in the second captured image F2 indicate a point on the same object, which is referred to as matching. Based on the position of the matched first feature point P1 on the first captured image F1, the position of the matched second feature point P2 on the second captured image F2, and the orientation of the camera when capturing each captured image, the positions of the points on the subject indicated by the first feature point P1 and the second feature point P2 can be measured in the first coordinate system. The points on the subject whose positions in the first coordinate system have been identified in this way are designated as landmarks Lm.

[0066] If the first feature point P1 and the second feature point P2 are matched without error and the accurate orientation of the camera when the first captured image F1 and the second captured image F2 are obtained, the landmark Lm can be generated accurately. However, if two feature points that do not correspond to each other are matched, an incorrect landmark will be generated. A plurality of landmarks generated incorrectly in this way will become noise in the map information. A plurality of landmarks generated incorrectly in this way will become so-called point cloud noise. There are many factors that cause such point cloud noise. For example, point cloud noise can be caused by the captured image containing many repetitive patterns.

[0067] With map information that includes landmarks that become point cloud noise, there may be cases where reprojection errors cannot be reduced. The landmark generation method according to this embodiment is a landmark generation method that can reduce such point cloud noise. The landmark generation method according to this embodiment will be described below with reference to FIG. 7. FIG. 7 is a flowchart showing the flow of the landmark generation method according to this embodiment.

[0068] The landmark generating method according to this embodiment is a method for generating a landmark Lm by performing triangulation based on a first photographed image F1 and a second photographed image F2 captured by a camera.

[0069] 7, in the landmark generating method according to this embodiment, first, a first feature point P1 contained in a first captured image F1 and a second feature point P2 contained in a second captured image F2 and to be matched with the first feature point P1 are extracted (S320). Here, the second feature point P2 to be matched with the first feature point P1 is a point that is objectively determined, for example, by using image processing technology, to be the same point on the subject in the second captured image F2 as the first feature point P1.

[0070] Next, a third feature point that is included in the first captured image F1 and is close to the first feature point P1, and a fourth feature point that is included in the second captured image and is a target for matching with the third feature point are extracted (S330). The third feature point and the fourth feature point will now be described with reference to FIG. 8. FIG. 8 is a schematic diagram for explaining the third feature point P3 and the fourth feature point P4 according to this embodiment. In the example shown in FIG. 8, in the first captured image F1 captured by the camera C0, a third feature point P3 and a fifth feature point P5 may be extracted in addition to the first feature point P1. Furthermore, in the second captured image F2 captured by the camera C0, a fourth feature point P4 and a sixth feature point P6 may be extracted in addition to the second feature point P2. Here, the fourth feature point P4 is a target for matching with the third feature point P3, and the sixth feature point P6 is a target for matching with the fifth feature point P5. In this embodiment, a third feature point P3, which is included in the first captured image F1 and is close to the first feature point P1, and a fourth feature point, which is included in the second captured image F2 and is a matching target for the third feature point P3, are extracted. Note that being close to the first feature point may be defined as the distance from the first feature point P1 being equal to or less than ½ the maximum dimension of the first captured image F1, equal to or less than ⅓ the maximum dimension of the first captured image F1, equal to or less than ⅕ the maximum dimension of the first captured image F1, or equal to or less than ⅙ the maximum dimension of the first captured image F1.

[0071] 7, the probability of an error in matching between the first feature point P1 and the second feature point P2 is predicted based on information on the first feature point P1, the second feature point P2, the third feature point P3, and the fourth feature point P4 (S340). Such an error probability can be calculated using, for example, an inference engine including a trained neural network.

[0072] The neural network training method used here will be described. For example, from a first training image and a second training image, a first training feature point included in the first training image and a second training feature point included in the second training image that are to be matched with the first training feature point are extracted. Also, a third training feature point included in the first training image and that is close to the first training feature point, and a fourth training feature point included in the second training image that is to be matched with the third training feature point are extracted. Next, information on the first training feature point, the second training feature point, the third training feature point, and the fourth training feature point are input, and the result of determining whether a landmark generated based on the first training feature point and the second feature point is noise is used as training data to train the neural network. Here, whether a landmark is noise can be determined based on, for example, information other than the captured image. For example, by using an image captured by a Light Detection and Ranging (LIDAR) of a subject similar to the captured image, it is possible to determine whether the landmark is noise.

[0073] By performing the above-described learning process on various captured images, a neural network capable of predicting the probability of a matching error can be realized. Here, the reason why the third feature point P3, which is close to the first feature point P1, is used to predict the probability of an error will be explained. For example, as described above, the inclusion of many repetitive patterns in the captured image can be a cause of matching errors. When the captured image contains many repetitive patterns, it is believed that feature points located near the first feature point contain information related to a matching error. For this reason, in this embodiment, information on the third feature point P3 and the fourth feature point P4 is used to predict the probability of a matching error.

[0074] 7, whether or not to generate landmarks based on the first feature point P1 and the second feature point P2 is determined based on the error probability (S350). Specifically, if the error probability predicted in step S340 is greater than a predetermined reference value (Yes in S350), it is determined that there is a high probability that the landmarks will become noise, and the process returns to step S320. On the other hand, if the error probability predicted in step S340 is equal to or less than the predetermined reference value (No in S350), it is determined that there is a low probability that the landmarks will become noise, and landmarks are generated based on the first feature point P1 and the second feature point P2 (S360).

[0075] Thereafter, the same process is repeated for all pairs of feature points of interest.

[0076] As described above, in the landmark generation method according to this embodiment, if the probability of an error in matching between the first feature point P1 and the second feature point P2 is high, no landmarks are generated, thereby reducing the number of landmarks that become noise. This makes it possible to generate highly accurate map information.

[0077] In the above, landmarks are not generated when the matching error probability is high, but the landmark generation method according to this embodiment is not limited to this. For example, landmarks may be generated even when the matching error probability is high, and likelihood information may be assigned to the landmarks according to the matching error probability. By having likelihood information for each landmark in this way, it is possible to determine whether each landmark is noise by also referring to the likelihood of other landmarks. This makes it possible to more accurately determine whether a landmark is noise.

[0078] (Embodiment 5) A feature point distribution adjustment method according to embodiment 5 will be described. The feature point distribution adjustment method according to this embodiment is a method for adjusting the distribution of feature points corresponding to one or more landmarks included in an image captured by a camera, and can be used in each of the above embodiments.

[0079] Feature points extracted from a captured image are determined according to the shape, pattern, etc. of the subject, and therefore are not necessarily distributed evenly within the captured image. For example, if feature points are too densely packed within a captured image, redundant information is included, resulting in more calculations for the feature points than necessary. Furthermore, memory capacity is unnecessarily occupied, which may prevent the storage of other important information. The presence of many similar feature points may also cause matching errors. Furthermore, a lack of feature points in a region of interest makes it difficult to obtain information about the region of interest. The feature point distribution adjustment method according to this embodiment is a method that can equalize the distribution of feature points. The feature point distribution adjustment method according to this embodiment will be described below with reference to FIG. 9. FIG. 9 is a flowchart showing the flow of the feature point distribution adjustment method according to this embodiment.

[0080] As shown in Fig. 9, first, a plurality of first feature points corresponding to one or more landmarks included in the captured image are extracted (S420). Here, the plurality of first feature points will be described with reference to Fig. 10 and Fig. 11. Fig. 10 and Fig. 11 are schematic diagrams respectively showing first and second examples of a plurality of first feature points according to this embodiment.

[0081] In the first example shown in Fig. 10, a plurality of first feature points P11 included in a photographed image F10 are shown. Fig. 10 also shows a region of interest R10 in the photographed image F10.

[0082] A second example shown in Fig. 11 shows a plurality of first feature points P21 included in a captured image F20. In Fig. 11, an image of a rectangular parallelepiped subject is shown in the captured image F20.

[0083] Next, as shown in Fig. 9, a feature point group made up of a plurality of second feature points is extracted from the plurality of first feature points (S430). Here, a feature point group made up of a plurality of second feature points will be described with reference to Fig. 12 and Fig. 13. Fig. 12 and Fig. 13 are schematic diagrams respectively showing first and second examples of feature point groups made up of a plurality of second feature points according to this embodiment.

[0084] The first example shown in Fig. 12 shows a feature point group consisting of four second feature points P12 extracted from a plurality of first feature points P11 included in the captured image F10 shown in Fig. 10. These four second feature points P12 are extracted based on the condition that, for example, a line segment connecting two of the extracted four second feature points P12 intersects with a line segment connecting the other two, and the point of intersection is a feature point-deficient region R11 in the region of interest where first feature points are lacking.

[0085] The second example shown in Fig. 13 shows four second feature points P22 extracted from a plurality of first feature points P21 included in the captured image F20 shown in Fig. 11. Fig. 13 shows four second feature points P22 densely arranged near the edge portion of a rectangular parallelepiped shown in the captured image F20. Furthermore, in extracting the second feature points, not only the positions of the second feature points but also the similarity of the feature amounts of the feature points may be taken into consideration. In other words, first feature points that are similar not only in position but also in feature amount may be extracted as the second feature points.

[0086] The extraction of the above-described feature point group can be easily performed by, for example, specifying in advance an image block including a region of interest in a captured image, its neighboring image blocks, and the position of interest. The extraction of the feature point group can also be performed using, for example, an inference engine including a trained neural network. For example, the extraction of the four second feature points P12 shown in FIG. 12 can also be performed using an inference engine including a trained neural network. For example, in training the neural network, a plurality of first feature points are input. Furthermore, when a line segment connecting two of the extracted four second feature points intersects with a line segment connecting the other two, the intersection point is determined to be a feature point-deficient region in the region of interest where first feature points are lacking. Article The four second feature points that satisfy the condition are used as training data. and By training a neural network based on this data, an inference engine that extracts second feature points can be realized. Alternatively, four second feature points may be extracted so that the intersection point is closest to the center of gravity of the region of interest.

[0087] Whether or not the feature points are on the same object surface can be determined based on, for example, information on the estimated position in the first coordinate system of the landmark corresponding to the feature point.

[0088] Next, as shown in Fig. 9, at least one of deleting one or more second feature points included in the feature point group (i.e., the plurality of second feature points) and adding a third feature point based on the plurality of second feature points is performed (S440) based on the distribution of the plurality of second feature points in the captured image. Here, a processing example of this step will be described with reference to Figs. 14 and 15. Fig. 14 is a schematic diagram showing an example of adding a third feature point P13 according to this embodiment. Fig. 15 is a schematic diagram showing an example of deleting a second feature point P22 according to this embodiment.

[0089] 14, there is a feature-point-deficient region on the left side of the region of interest R10 where second feature points P12 are lacking, so a third feature point P13 is added based on four second feature points P12. In the example shown in FIG. 14, the third feature point P13 is added at the intersection of a line segment connecting two of the four second feature points P12 and a line segment connecting the other two. The feature amount of the third feature point P13 may be determined based on the feature amounts of the four second feature points P12. For example, the feature amount of the third feature point P13 may be an average of the feature amounts of the four second feature points P12, or a weighted average taking into account the distance between the third feature point P13 and the four second feature points P12.

[0090] The feature-point-lacking region may be identified, for example, by defining a region of interest R10 in the captured image F10 and based on the distribution density of the second feature points P12 in the region of interest R10. Alternatively, a region in which the distance to the closest second feature point P12 among points in the region of interest R10 is greater than a predetermined reference value may be identified as a feature-point-lacking region.

[0091] In the example shown in FIG. 15, three of the four densely arranged second feature points P22 shown in FIG. 13 have been deleted. In FIG. 15, the three deleted second feature points are indicated by dashed lines. In this manner, unnecessary second feature points can be deleted. Note that, even in different captured images, matching of the remaining second feature points can be maintained by extracting second feature points using a similar method. Note that the remaining second feature point is also referred to as a representative feature point. For example, the representative feature point may be determined to be the second feature point located closest to the center of gravity of the area in which the multiple second feature points are located. Alternatively, a new representative feature point may be generated at the center of gravity, and all the multiple second feature points may be deleted. In this case, the feature amount of the representative feature point may be, for example, a weighted average of the feature amounts of the multiple second feature points depending on the distance from the center of gravity.

[0092] As described above, the feature point distribution adjustment method according to this embodiment makes it possible to appropriately adjust the distribution of feature points included in a captured image, thereby solving at least some of the problems caused by the distribution of feature points described above.

[0093] It should be noted that the feature point distribution adjustment method according to this embodiment is not limited to the configuration described above. For example, when information related to the likelihood of an extracted feature point group is obtained, the processing content may be changed according to the likelihood. For example, when the likelihood is low, it is not necessary to add a third feature point based on the feature point group. Furthermore, the likelihood of a feature point group may be defined as the likelihood of a third feature point added based on the feature point group.

[0094] (Hardware configuration) A hardware configuration for executing the methods according to the above-described embodiments will be described with reference to Fig. 16. Fig. 16 is a diagram showing an example of the hardware configuration of a computer 1000 for executing the methods according to the above-described embodiments by software. That is, the computer 1000 can realize a map information updating device that executes the map information updating methods according to the first to third embodiments, a landmark generating device that executes the landmark generating method according to the fourth embodiment, and a feature point distribution adjusting device that executes the feature point distribution adjusting method according to the fifth embodiment.

[0095] 16, the computer 1000 includes an input device 1001, an output device 1002, a CPU 1003, an internal storage 1004, a RAM 1005, a reading device 1007, a transmitting / receiving device 1008, and a bus 1009. The input device 1001, the output device 1002, the CPU 1003, the internal storage 1004, the RAM 1005, the reading device 1007, and the transmitting / receiving device 1008 are connected by the bus 1009.

[0096] The input device 1001 is a user interface device such as a keyboard, mouse, input buttons, touchpad, touch panel display, etc., and accepts user operations. Note that the input device 1001 may be configured to accept voice operations and remote operations using a remote control or the like in addition to accepting user touch operations.

[0097] The output device 1002 is a device that outputs a signal from the computer 1000, and may be a device that serves as a user interface such as a signal output terminal, a display, or a speaker.

[0098] The internal storage 1004 is a flash memory, etc. Furthermore, the internal storage 1004 may store in advance programs for executing the steps of the methods according to the first to fifth embodiments.

[0099] The RAM 1005 is a random access memory, and is used to store data calculated when a program or application is executed.

[0100] The reading device 1007 reads information from a recording medium such as a USB (Universal Serial Bus) memory. The reading device 1007 reads the above-mentioned programs and applications from a recording medium on which the programs and applications are recorded, and stores the programs and applications in the built-in storage 1004.

[0101] The transmitting / receiving device 1008 is a communication circuit for performing wireless or wired communication. The transmitting / receiving device 1008 communicates with, for example, a server device connected to a network, downloads the above-mentioned programs and applications from the server device, and stores them in the built-in storage 1004.

[0102] The CPU 1003 is a central processing unit that copies programs, applications, etc. stored in the internal storage 1004 to the RAM 1005, and sequentially reads and executes instructions contained in the copied programs, applications, etc. from the RAM 1005.

[0103] (Variations, etc.) While the methods according to the present disclosure have been described above based on the embodiments, the present disclosure is not limited to these embodiments. As long as they do not deviate from the gist of the present disclosure, various modifications conceivable by those skilled in the art to the embodiments and other forms constructed by combining some of the components of the embodiments are also included within the scope of the present disclosure.

[0104] The following embodiments may also be included within the scope of one or more aspects of the present disclosure.

[0105] The methods according to the present disclosure may be realized as a computer program that is executed by a computer, or as a digital signal that is a computer program. Furthermore, the present disclosure may be realized as a non-transitory computer-readable recording medium, such as a CD-ROM, on which the computer program is recorded.

[0106] The present disclosure may also be a computer system including a microprocessor and a memory, the memory storing the computer program, and the microprocessor operating in accordance with the computer program.

[0107] Furthermore, the program or the digital signal may be recorded on the recording medium and transferred, or the program or the digital signal may be transferred via a network or the like, so that the program or the digital signal may be implemented by another independent computer system.

[0108] The above embodiments may be combined with each other. [Industrial Applicability]

[0109] The present disclosure can be used, for example, in VSLAM technology. [Explanation of symbols]

[0110] 1000 computers 1001 Input Device 1002 Output device 1003 CPU 1004 Internal Storage 1005 RAM 1007 Reading device 1008 Transmitting and receiving device 1009 Bus C0 Camera F1 First image Images taken at F10 and F20 F2 Second image Lm Landmark P1, P11, P21 First feature point P2, P12, P22 Second feature point P3, P13 Third feature point P4 Fourth feature P5 The fifth feature P6 The sixth feature R10 Area of ​​Interest R11 Feature point missing area

Claims

1. a map information acquisition step of acquiring map information including estimated positions of the camera and one or more landmarks in a first coordinate system; an observation data acquisition step of acquiring landmark observation positions indicating positions of the one or more landmarks in a second coordinate system in an image captured by the camera; an adding step of generating additional map information by adding information related to the landmark observation position to the map information acquired in the map information acquiring step, and updating the map information acquired in the map information acquiring step with the additional map information; a prediction step of calculating predicted map information based on the map information updated in the adding step using a predictive inference engine including a trained predictive neural network, and updating the map information updated in the adding step to the predicted map information; an updating step of calculating updated map information using a gradient method on the map information updated in the prediction step, and updating the map information updated in the prediction step to the updated map information, the predictive neural network is a neural network that has been trained to receive the map information updated in the adding step and to calculate the predicted map information having a smaller reprojection error than the map information, the predicted map information is map information including estimated positions of the camera and the one or more landmarks in the first coordinate system, which has been further updated to reduce the reprojection error compared to the map information updated in the additional step; The reprojection error is an error between the landmark observation position and a reprojection position on the captured image corresponding to the landmark observation position, the reprojection position being calculated based on map information. How to update map information.

2. The method further includes a determination step of determining convergence of the update of the map information updated in the update step based on the reprojection error calculated using a reprojection error function for the map information updated in the update step, and determining whether to return to the prediction step or the update step, or to terminate the update of the map information updated in the update step, based on the result of the determination, The reprojection error function is a function for calculating the reprojection error. The map information updating method according to claim 1 .

3. the prediction neural network receives map information for learning as input and learns updated map information for learning as training data, The updated map information for learning is generated based on the map information for learning, and reduces the re-projection error calculated using a re-projection error function; The reprojection error function is a function for calculating the reprojection error.

3. The map information updating method according to claim 1 or 2.

4. The updating step further includes an inference step of calculating inference map information by updating the map information updated in the prediction step using an updating inference engine including a trained updating neural network, and updating the map information updated in the prediction step to the inference map information. The map information updating method according to any one of claims 1 to 3.

Images