How to update map information

Abstract

This map information update method includes: a step for acquiring one or more projection relationships; a step for acquiring reprojection error information for each of the one or more projection relationships; a step for calculating, for each of one or more landmarks, a first total value based on every reprojection error information associated with the landmark; a step for calculating, for each of one or more key frames, a second total value based on every reprojection error information associated with the key frame; a step for inferring, for each of the one or more landmarks, a position information updated value, which is an updated value of position information about the landmark, from the first total value, and updating the position information about the landmark using the position information updated value; and a step for inferring, for each of the one or more key frames, a pose information updated value, which is an updated value of pose information about the key frame, from the second total value, and updating the pose information about the key frame using the pose information updated value.

Description

[Technical Field]

[0001] The present disclosure relates to a map information update method. [Background technology]

[0002] A conventional technique known as VSLAM (Visual Simultaneous Localization and Mapping) is to capture images using a camera and simultaneously estimate the position of the camera and the positions of surrounding landmarks from information contained in the captured images, or keyframes.

[0003] The core processing in VSLAM technology is to calculate the reprojection error, which is the error between the position of a photographed landmark in a keyframe and the reprojected position, which is the position in the keyframe calculated from the assumed posture information of the keyframe (i.e., the position and orientation of the camera) and the assumed landmark position, and to calculate the posture information and landmark position of the keyframe where the reprojection error is 0 (in reality, the error is sufficiently small). The information that combines the posture information of the keyframe and the landmark position is called map information.

[0004] Searching for map information that reduces the reprojection error to zero is called bundle adjustment, and is a process generally classified as an optimization problem of the nonlinear least squares method. For this reason, bundle adjustment requires a process of minutely correcting the map information so that the reprojection error decreases, and repeating this correction until the value of the reprojection error converges (see, for example, Patent Document 1, Non-Patent Document 1, and Non-Patent Document 2). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Special Publication No. 2017-503290 [Non-patent literature]

[0006] [Non-Patent Document 1] Yuki Iwamoto, Yasuyuki Sugaya, Kenichi Kanaya, Implementation and Evaluation of Bundle Adjustment for 3D Reconstruction, IPSJ Technical Report, 2011-CVIM-175-19, pp.1-8, 2011 [Non-patent document 2] Takayuki Okatani, Bundle Adjustment, Information Processing Society of Japan Technical Report, 2009-CVIM167-37, pp.1-16, 2009 Summary of the Invention [Problem to be solved by the invention]

[0007] In VSLAM, every time a new keyframe is added, a new reprojection error is calculated and the map information is updated through bundle adjustment, which is essential for maintaining the accuracy of the map information.

[0008] Gradient-based algorithms are commonly used as algorithms for achieving convergence, which is necessary during bundle adjustment. One well-known example of a gradient-based algorithm is an algorithm that combines the steepest descent method and the Gauss-Newton method. This algorithm uses the steepest descent method to perform correction until the reprojection error approaches a minimum value, after which it switches to the Gauss-Newton method. This type of algorithm requires computationally intensive processing for each correction iteration, including generating a Hessian matrix and calculating the correction amount by solving simultaneous equations. The inclusion of such computationally intensive processing poses a problem when using VSLAM technology.

[0009] The present disclosure has been made to solve such problems, and aims to provide a map information updating method that can reduce the amount of calculations. [Means for solving the problem]

[0010] In order to achieve the above object, a map information updating method according to one embodiment of the present disclosure includes the steps of: acquiring one or more projection relationships; acquiring reprojection error information for each of the one or more projection relationships; calculating, for each of one or more landmarks, a first sum based on all of the reprojection error information associated with the landmark; calculating, for each of one or more key frames, a second sum based on all of the reprojection error information associated with the key frame; inferring, for each of the one or more landmarks, a position information update value that is an update value of position information of the landmark from the first sum and updating position information of the landmark using the position information update value; and inferring, for each of the one or more key frames, an orientation information update value that is an update value of orientation information of the key frame from the second sum and updating orientation information of the key frame using the orientation information update value. In order to achieve the above object, otherA map information updating method according to one embodiment is a map information updating method for updating map information including one or more position information associated with one or more landmarks and one or more posture information associated with one or more key frames, wherein each of the one or more key frames is a captured image captured so as to include at least one of the one or more landmarks, and each of the one or more posture information includes position and orientation information; a projection relationship acquisition step for acquiring one or more projection relationships, wherein each of the one or more projection relationships is a relationship between each of the one or more landmarks, each of the one or more key frames, and projection coordinate information corresponding to coordinates on the key frames of a projection point when each of the one or more landmarks is projected onto each of the one or more key frames; and a projection relationship acquisition step for acquiring, for each of the one or more projection relationships, position information associated with one of the one or more landmarks constituting the projection relationship among the one or more position information, and projection coordinate information associated with one of the one or more key frames constituting the projection relationship among the one or more posture information. a reprojection error information acquisition step of acquiring reprojection error information based on the posture information acquired and the projection coordinate information constituting the projection relationship, and associating the reprojection error information with the projection relationship; a landmark-origin error aggregation step of generating, for each of the one or more landmarks, a first reprojection error information group in which all of the reprojection error information associated with the landmark is aggregated, and calculating a first total value based on all of the reprojection error information included in the first reprojection error information group; a keyframe-origin error aggregation step of generating, for each of the one or more keyframes, a second reprojection error information group in which all of the reprojection error information associated with the keyframe is aggregated, and calculating a second total value based on all of the reprojection error information included in the second reprojection error information group; a position information update step of inferring, for each of the one or more landmarks, a position information update value that is an update value of position information of the landmark from the first total value, and updating position information of the landmark using the position information update value; and, for each of the one or more keyframes,and a posture information updating step of estimating a posture information update value, which is an update value of posture information of the key frame among the one or more posture information, from the second sum value, and updating the posture information of the key frame using the posture information update value. [Effects of the Invention]

[0011] The present disclosure provides a map information update method that can reduce the amount of calculations. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a flowchart showing the flow of the map information updating method according to the first embodiment. [Figure 2] FIG. 2 is a schematic graph showing the relationship between map information and reprojection error in bundle adjustment. [Figure 3] FIG. 3 is a schematic graph for explaining an overview of prediction map information according to the first embodiment. [Figure 4] FIG. 4 is a flowchart showing a method for calculating prediction map information according to the first embodiment. [Figure 5] FIG. 5 is a schematic diagram illustrating a projection relationship between key frames and landmarks. [Figure 6] FIG. 6 is a schematic diagram illustrating the first reprojection error information group according to the first embodiment. [Figure 7] FIG. 7 is a schematic diagram for explaining the second reprojection error information group according to the first embodiment. [Figure 8] FIG. 8 is a block diagram showing a functional configuration of the map information updating device according to the first embodiment. [Figure 9] FIG. 9 is a diagram showing an example of the hardware configuration of a computer for executing the methods according to the embodiments by software. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, shapes, materials, specifications, components, component placement and connection configurations, steps, and step order shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concept of the present disclosure will be described as optional components. Furthermore, each drawing is not necessarily an exact illustration. In each drawing, substantially identical components are assigned the same reference numerals, and duplicated descriptions may be omitted or simplified.

[0014] (Embodiment 1) A map information update method according to the first embodiment will be described.

[0015] [1-1. Overview of how to update map information] First, an overview of the map information update method will be described. The map information update method is a method used in VSLAM technology, which takes an image using a camera and simultaneously estimates the position of the camera and the positions of surrounding landmarks from information contained in key frames, which are captured images. The map information update method according to this embodiment updates map information that includes one or more pieces of position information each associated with one or more landmarks, and one or more pieces of orientation information each associated with one or more key frames. Each of the one or more key frames is a captured image that has been captured so as to include at least one of the one or more landmarks. Each of the one or more pieces of orientation information includes information on the position and orientation of the camera used for capturing the image.

[0016] In VSLAM technology, a three-dimensional coordinate system is first assumed as a first coordinate system, and a camera is placed in the first coordinate system. The first coordinate system is a coordinate system fixed with respect to the space in which the camera and other devices are placed, and is also referred to as a world coordinate system. Landmarks are generated from key frames captured by the camera. Here, landmarks are three-dimensional points created in the first coordinate system. For example, landmarks are generated by triangulation based on corresponding feature points included in each of two key frames. In a map information update method according to this embodiment, map information including estimated positions of the camera and one or more landmarks in the first coordinate system is updated. Specifically, the map information according to this embodiment is updated when information on key frames captured by the camera is added to the map information and / or when bundle adjustment is performed.

[0017] The map information update method according to this embodiment will be described below with reference to Fig. 1. Fig. 1 is a flowchart showing the flow of the map information update method according to this embodiment.

[0018] 1, in the map information update method according to the present embodiment, first, map information is acquired (S20). The map information includes at least position information of one or more landmarks and posture information of one or more key frames.

[0019] Next, projection coordinate information indicating the positions of one or more landmarks in a second coordinate system in a key frame captured by the camera is acquired (S30). The second coordinate system is a coordinate system fixed with respect to the captured image and is also referred to as a key frame coordinate system. The projection coordinate information is the positions in the second coordinate system of feature points corresponding to the positions of one or more landmarks. In other words, the projection coordinate information is information corresponding to the coordinates on one or more key frames of the projection points when each of one or more landmarks is projected onto the key frame.

[0020] Next, additional map information is generated by adding projected coordinate information to the map information acquired in step S20 (S40). Information related to the projected coordinate information may be added to the additional map information. For example, the information related to the projected coordinate information may be an estimated position of a landmark generated based on feature points included in the key frame. The information related to the projected coordinate information may be information that is simply estimated from the position of the camera, etc.

[0021] After step S40, predicted map information is calculated based on the additional map information generated in step S40, and the map information updated in step S40 is updated to the predicted map information (S50). Here, a method for calculating predicted map information will be described. When updating the map information, bundle adjustment is generally performed. That is, map information that will result in a reprojection error of zero is searched for. In this embodiment, the reprojection error is information acquired based on position information, orientation information, and projection coordinate information. More specifically, the reprojection error is calculated using a reprojection error function, which is a function for calculating the error between the projection coordinate information and a reprojection position on the captured image corresponding to the projection coordinate information at the projection coordinates, which is calculated based on the map information. Note that the reprojection error may include a sum of one or both of an error calculated using the reprojection error function for one or more landmarks included in the map information and an error calculated using the reprojection error function for each of one or more key frames.

[0022] Here, an overview of bundle adjustment will be explained using Fig. 2. Fig. 2 is a schematic graph showing the relationship between map information and reprojection error in bundle adjustment. The horizontal axis of Fig. 2 shows a quantity that is a schematic representation of the map information as a single variable, and the vertical axis shows the reprojection error relative to the map information.

[0023] As shown in Figure 2, bundle adjustment corrects the map information before bundle adjustment to map information that minimizes the reprojection error. For example, when performing bundle adjustment using a gradient-based algorithm, such as an algorithm that combines the steepest descent method and the Gauss-Newton method, map information that minimizes the reprojection error is searched for by repeatedly correcting the map information before bundle adjustment with small correction amounts and calculating the reprojection error after correction. Note that, below, the map information that minimizes the reprojection error is also referred to as the map information solution. Here, in spatial geometric algorithms using gradient methods, each correction iteration involves computationally intensive processes, such as generating a Hessian matrix and calculating the correction amount by solving simultaneous equations using the nonlinear least squares method. In particular, when the error between the map information before bundle adjustment and the map information solution is large, multiple iterations are required.

[0024] In contrast to such conventional techniques, this embodiment reduces the amount of calculation by using an inference engine for at least a part of the calculation of prediction map information. An outline of a calculation method of prediction map information according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a schematic graph for explaining the outline of prediction map information according to this embodiment.

[0025] As shown in FIG. 3, in this embodiment, an inference engine is used to calculate, as predicted map information, map information in which the reprojection error is close to the minimum value based on the map information. This makes it possible to obtain map information in which the reprojection error is close to the minimum value without repeating calculations using an algorithm that uses a gradient method. The predictive neural network included in such an inference engine is a trained neural network that uses training map information as input and training updated map information as training data. In this training, a loss function based on the difference between the training updated map information and the map information output from the neural network is considered, and training is carried out so as to reduce this loss function to zero. The training map information is not particularly limited as long as it is information similar to the additional map information used in the map information updating method according to this embodiment.

[0026] The updated map information for learning is generated based on the map information for learning and is map information that reduces the reprojection error calculated using the reprojection error function. For this reason, the loss function in learning may be based on this reprojection error function. Since the reprojection error function is calculated directly from the (learning) map information, the updated map information for learning is not required as training data in this case. Here, the reprojection error function is a function that calculates the error between the projection coordinate information and the reprojection position on the captured image corresponding to the projection coordinate information, which is calculated based on the map information. Specifically, a well-known function such as that described in Non-Patent Document 1 can be used as the reprojection error function.

[0027] The updated map information for learning can be obtained, for example, by actually performing bundle adjustment on the learning projection coordinate information and the learning map information using a gradient method. Note that the map information that reduces the reprojection error may be, for example, map information that minimizes the reprojection error. The map information that minimizes the reprojection error is not limited to map information that strictly minimizes the reprojection error, but also includes map information that generally minimizes the reprojection error. For example, map information in which the difference between the reprojection error of the map information and the minimum value of the reprojection error is 5% or less of the minimum value is also included in the map information that minimizes the reprojection error.

[0028] By performing the above learning, the predictive neural network according to this embodiment learns the shape of the error function that indicates the relationship between map information and reprojection error. Learning the predictive neural network is a process equivalent to fitting to the error function. By learning the shape of the error function, the predictive neural network becomes able to predict map information that minimizes the reprojection error. Note that while information such as the camera position contained in the map information changes depending on the map information, the error function learned by the predictive neural network does not change. Furthermore, the predicted map information does not have to be map information that minimizes the reprojection error.

[0029] In addition, the map information updating method according to this embodiment may include a step of preventing the predicted map information calculated using the inference engine from moving away from the solution of the map information (i.e., preventing the difference between the predicted map information and the solution of the map information from becoming larger than the difference between the additional map information and the solution of the map information). For example, an inference engine may be prepared in advance to predict a correction direction for the additional map information that will bring it closer to the solution of the map information, and the inference engine may determine whether the predicted map information is closer to the solution of the map information than the additional map information.

[0030] In this embodiment, at least a portion of the calculations, such as the solution calculations of simultaneous equations in the gradient method of the prior art, can be replaced with inference using an inference engine. Therefore, in this embodiment, the amount of calculation can be reduced compared to the gradient method, and the parallelism of calculations can be increased. Therefore, in this embodiment, effects such as faster map information updates and lower power consumption can be achieved. Furthermore, inference using an inference engine can also reduce calculation accuracy. Therefore, it is possible to simplify the hardware configuration of a computer or the like for executing the map information update method. Details of the method for calculating predicted map information using an inference engine will be described later.

[0031] 1, updated map information is calculated spatially geometrically for the map information updated in step S50, and the map information updated in step S50 is updated to the updated map information (S60). That is, as in the prior art, the map information is updated using an algorithm that combines the steepest descent method and the Gauss-Newton method, for example, to bring the map information closer to the solution of the map information.

[0032] Next, the re-projection error for the map information is calculated (S70). Specifically, the re-projection error for the map information is calculated using the re-projection error function described above.

[0033] Subsequently, based on the reprojection error calculated using the reprojection error function for the map information updated in step S60, it is determined whether the update of the map information updated in step S60 converges. Based on the result of the determination, it is decided whether to return to the prediction step or the update step, or to end the update of the map information updated in step S60 (S80). For example, when the change amount ΔE of the reprojection error from the previous determination (at the first determination, it is the change amount from the reprojection error for the predicted map information) is smaller than a predetermined convergence threshold Sc (ΔE < Sc in S80), it is determined that the solution of the map information has been obtained, and the update of the map information is ended. Also, when the reprojection error E is larger than the upper limit value Su (E > Su in S80), it is determined that the predicted map information is inappropriate, and the process returns to step S50 to calculate the predicted map information again. Also, when the change amount ΔE of the reprojection error from the previous determination is greater than or equal to the convergence threshold Sc and the reprojection error E is less than or equal to the upper limit value Su (ΔE ≥ Sc, E ≤ Su in S80), the process returns to step S60, and the update of the map information using the gradient method is performed again.

[0034] With the map information update method as described above, the amount of computation required for map information update can be reduced compared to the case of using the prior art.

[0035] [1-2. Method for calculating predicted map information] The method for calculating the predicted map information in the map information update method according to the present embodiment will be described with reference to FIG. 4. FIG. 4 is a flowchart showing the method for calculating the predicted map information according to the present embodiment.

[0036] As shown in FIG. 4, first, a projection relationship is acquired (projection relationship acquisition step S510). Here, the projection relationship will be described with reference to FIG. 5. FIG. 5 is a schematic diagram illustrating the projection relationship related to key frames and landmarks. The projection relationship is the relationship between one landmark, one key frame, and projection coordinate information corresponding to the coordinates on the key frame of the projection point when the landmark is projected onto the key frame. FIG. 5 shows two landmarks LM1 and LM2 and two key frames KF1 and KF2. FIG. 5 also shows projection coordinate information C11 and C12 obtained by projecting the landmark LM1 onto the key frames KF1 and KF2, respectively, and projection coordinate information C21 and C22 obtained by projecting the landmark LM2 onto the key frames KF1 and KF2, respectively.

[0037] In this step, a projection relationship between each of one or more landmarks and each of one or more key frames is obtained. In the example shown in FIG. 5, the projection relationships include a correspondence relationship between landmark LM1, key frame KF1, and projection coordinate information C11; a correspondence relationship between landmark LM1, key frame KF2, and projection coordinate information C12; a correspondence relationship between landmark LM2, key frame KF1, and projection coordinate information C21; and a correspondence relationship between landmark LM2, key frame KF2, and projection coordinate information C22. The projection coordinate information refers to the coordinates of the actual projection point of the landmark projected onto the key frame, not coordinates calculated from map information.

[0038] Next, as shown in FIG. 4, reprojection error information is acquired for each of one or more projection relationships, and the reprojection error information is associated with the projection relationship (reprojection error information acquisition step S520). The reprojection error information includes one of one or more pieces of position information, one of one or more pieces of orientation information, and a feature value indicating the result of inferring the reprojection error from the projection coordinate information corresponding to one projection relationship. Here, the reprojection error is an error calculated spatially and geometrically from the position information, the orientation information, and the projection coordinate information. In this embodiment, instead of calculating the reprojection error, a feature value indicating the result of inferring the reprojection error is obtained. The feature value included in the reprojection error information includes, for example, a weight vector and an error vector corresponding to the projection relationship corresponding to the landmark and the keyframe. In other words, the feature value includes a term that is commonly included in blocks including diagonal elements of the Hessian matrix used to calculate the update value of the map information.

[0039] Here, we will explain this feature in detail using the Hessian matrix. The equation for calculating the update value of map information using the conventional Gauss-Newton method is expressed as the following equation (1) using the Hessian matrix H and the update value Δξ of the parameter ξ.

[0040]

number

[0041] Here, the matrix and vector elements of equation (1) are written as equation (2) below.

[0042]

number

[0043] In the Hessian matrix of equation (2), blocks other than the diagonal blocks are omitted.

[0044] Here, if α is an integer between 1 and N, the blocks related to the landmarks of the Hessian matrix and the elements of the gradient vector on the right side of equation (1) related to the blocks are expressed by the following equations (3a) and (3b).

[0045]

number

[0046]

number

[0047] In formulas (3a) and (3b), I αj is either 0 or 1 and indicates the visibility of the α-th landmark from the j-th camera pose. j is the rotation matrix that indicates the orientation of the j-th camera. αj is a 3x2 matrix calculated from the α-th landmark and the j-th camera pose. αj is the difference between the reprojected coordinates and the projection coordinate information (the coordinates of the actual projection point), and is a two-dimensional coordinate vector.

[0048] Furthermore, the diagonal blocks of the Hessian matrix and the elements related to the diagonal blocks of the gradient vector are expressed by the following equations (4a) to (5c), where κ is an integer between 1 and M.

[0049]

number

[0050]

number

[0051]

number

[0052]

number

[0053]

number

[0054]

number

[0055] where x i is the location information of the i-th landmark. κ is the translation vector of the pose of the κ-th camera and is the camera position.

[0056] When the Hessian matrix is ​​expressed as above, the feature is u ακ , R κ W ακ , (x α -t κ ) where u ακ represents the reprojected coordinates. κ W ακ represents the weighting coefficient. (x α -t κ ) indicates the relative position between the landmark and the key frame. Such features are obtained for each correspondence.

[0057] Next, the landmark-origin errors are aggregated (landmark-origin error aggregation step S530). That is, for each of one or more landmarks, a first reprojection error information group is generated in which all reprojection error information associated with the landmark is aggregated, and a first sum is calculated based on all the reprojection error information included in the first reprojection error information group. Here, the first reprojection error information group will be described with reference to FIG. 6. FIG. 6 is a schematic diagram for explaining the first reprojection error information group according to this embodiment. FIG. 6 shows an example configuration in which a landmark LM1 is projected only onto three key frames KF1, KF2, and KF3. As shown in FIG. 6, a case will be described in which a certain landmark LM1 is projected only onto three key frames KF1, KF2, and KF3. In this case, the reprojection error information is calculated from the position information of the landmark LM1, the orientation information of the key frame KF1, and the projection coordinate information of the landmark LM1 onto the key frame KF1. This calculation of reprojection error information is also performed for the other key frames KF2 and KF3, and the reprojection errors for these three are aggregated. The reprojection errors for each of the other landmarks are also aggregated.

[0058] Specifically, the feature quantities obtained for each of the above correspondences are summed for each element. By aggregating these errors, the landmark-related block shown in formula (3a) and the element related to the block of the gradient vector shown in formula (3b) are obtained.

[0059] Next, as shown in FIG. 4, the key frame-origin errors are aggregated (key frame-origin error aggregation step S540). That is, for each of one or more key frames, a second reprojection error information group is generated in which all reprojection error information associated with the key frame is aggregated, and a second sum is calculated based on all the reprojection error information included in the second reprojection error information group. Here, the second reprojection error information group will be described with reference to FIG. 7. FIG. 7 is a schematic diagram for explaining the second reprojection error information group according to this embodiment. FIG. 7 shows an example configuration in which only three landmarks LM1, LM2, and LM3 are projected onto a key frame KF1. As shown in FIG. 7, a case will be described in which only landmarks LM1, LM2, and LM3 are projected onto a certain key frame KF1. In this case, the reprojection error information is calculated from the posture information of the key frame KF1, the position information of the landmark LM1, and the projection coordinate information of the landmark LM1 onto the key frame KF1. This calculation of reprojection error information is also performed for the other landmarks LM2 and LM3, and the reprojection errors of these three are aggregated. The reprojection errors of the other key frames are also aggregated.

[0060] Specifically, the feature quantities obtained for each of the above correspondences are summed for each element. By aggregating these errors, the diagonal blocks of the Hessian matrix shown in the above formulas (4a) to (4c) and (5a) to (5c) and the elements of the gradient vector shown in the above formula (3b) related to the blocks are obtained.

[0061] Next, as shown in Fig. 4, the location information is updated (location information update step S550). In this step, for each of one or more landmarks, a location information update value, which is an update value of the location information of the landmark, is inferred from the first sum calculated in landmark origin error aggregation step S530, and the location information of the landmark is updated using the location information update value.

[0062] Next, the posture information is updated (posture information updating step S560). In this step, for each of one or more key frames, a posture information update value, which is an update value of the posture information of the key frame, is inferred from the second sum calculated in key frame origin error aggregation step S540, and the posture information of the key frame is updated using the posture information update value.

[0063] As described above, it is possible to obtain position information update values ​​for one or more landmarks and posture information update values ​​for one or more key frames. Based on these position information update values ​​and posture information update values, it is possible to obtain predicted map information.

[0064] In this embodiment, as described above, the position information update value and the posture information update value are found by inference, so the amount of calculation can be significantly reduced compared to when they are calculated spatially geometrically. Furthermore, in this embodiment, by separating the inference of the position information from the inference of the posture information, the degree of freedom of the solution in the inference can be reduced, so that more accurate inference is possible.

[0065] Furthermore, in this embodiment, by separating the inference of position information from the inference of posture information, and further separating the inference of posture information into orientation and position, the degree of freedom of the solution of each inference engine can be reduced, thereby enabling more reliable learning. The learning method of each inference engine will be described later.

[0066] Furthermore, in this embodiment, the reprojection error information is also calculated by inference, which further reduces the amount of calculation.

[0067] [1-3. Map information update device] An example of a map information updating device that can realize the predictive map information calculation method of the map information updating method according to this embodiment will be described with reference to Fig. 8. Fig. 8 is a block diagram showing the functional configuration of a map information updating device 10 according to this embodiment. As shown in Fig. 8, the map information updating device 10 is a device that receives input information including position information, orientation information, and a projection relationship, and outputs output information including position information update values ​​and orientation information update values.

[0068] As shown in FIG. 8, the map information updating device 10 includes an error inference engine 20, a first aggregation unit 21, a second aggregation unit 22, a position inference engine 23, and an attitude inference engine 24.

[0069] The error inference engine 20 is an inference engine that executes the reprojection error information acquisition step of the map information update method. For each of one or more projection relationships, the error inference engine 20 acquires reprojection error information based on position information associated with one landmark that constitutes the projection relationship, posture information associated with one key frame that constitutes the projection relationship, and projection coordinate information that constitutes the projection relationship, and associates the reprojection error information with the projection relationship.

[0070] In this embodiment, the reprojection error information includes a feature value indicating the result of inferring the reprojection error from one piece of position information, one piece of orientation information, and projection coordinate information corresponding to one of one or more projection relationships. As described above, the reprojection error is an error spatially geometrically calculated from the position information, orientation information, and projection coordinate information.

[0071] The first aggregating unit 21 is a processing unit that executes the landmark-origin error aggregating step of the map information updating method. The first aggregating unit 21 generates, for each of one or more landmarks, a first reprojection error information group in which all reprojection error information associated with the landmark is aggregated, and calculates a first sum based on all the reprojection error information included in the first reprojection error information group.

[0072] The second aggregation unit 22 executes the key frame origin error aggregation step of the map information update method. Processing section The second aggregation unit 22 generates, for each of one or more key frames, a second reprojection error information group in which all of the reprojection error information associated with the key frame is aggregated, and calculates a second sum based on all of the reprojection error information included in the second reprojection error information group.

[0073] Location Inference Engine 23 teeth The inference engine executes the location information update step of the map information update method, inferring, for each of one or more landmarks, a location information update value that is an update value of the location information of the landmark from the first sum, and updating the location information of the landmark using the location information update value.

[0074] The posture inference engine 24 is an inference engine that executes the posture information update step of the map information update method. For each of one or more key frames, the posture inference engine 24 infers a posture information update value that is an update value of the posture information of that key frame from the second sum value, and updates the posture information of that key frame using the posture information update value.

[0075] The map information updating device 10 as described above can realize the predicted map information calculation method of the map information updating method described above.

[0076] [1-4. Inference engine training method] The learning method of each of the above-mentioned inference engines will be described with reference to Fig. 8. As described above, in this embodiment, the position information update value is inferred by the position inference engine 23, the posture information update value is inferred by the posture inference engine 24, and the reprojection error information is inferred by the error inference engine 20.

[0077] The position inference engine 23, the posture inference engine 24, and the error inference engine 20 are engines that have learned using the sum of reprojection errors calculated spatially and geometrically based on the updated values ​​of the map information as a loss function. The updated values ​​of the map information are calculated by the position inference engine 23 and the posture inference engine 24 based on the reprojection error information calculated by the error inference engine 20 based on the map information.

[0078] According to the inference engine used in the map information updating method of this embodiment, by inferring position information and orientation information using separate inference engines, the degrees of freedom of inference can be significantly reduced compared to when the update values ​​for the entire map information are inferred collectively. In other words, the degrees of freedom of inference can be reduced by the difference between the size determined by the entire Hessian matrix used to calculate the update values ​​for the entire map information and the size of the submatrix of the Hessian matrix corresponding to position information and the submatrix corresponding to orientation information. Therefore, the number of steps required for training each inference engine can be significantly reduced. Furthermore, reducing the degrees of freedom of inference can increase the reliability of training the inference engine. This reduces the size of the model (in other words, the amount of calculation) required to obtain the required inference accuracy.

[0079] Furthermore, with regard to the map information update method according to this embodiment, when a solution is inferred using a Hessian matrix composed of a combination of arbitrary numerical values, structures such as simple perceptrons or convolutional neural networks have a very large degree of freedom when obtaining combinations of those numerical values ​​and solutions, resulting in a large scale and computational complexity of the inference model. Accordingly, the scale and computational complexity of the learning of the inference model also increase, increasing the man-hours and difficulty of the learning.

[0080] On the other hand, according to the map information update method of this embodiment, the inference model is divided into an error inference engine 20 that infers reprojection error information in one projection relationship, and a position inference engine 23 and a posture inference engine 24 that infer updated values ​​for position information and posture information, respectively, from the sum of the reprojection error information output from the error inference engine 20. By dividing the inference engines in this manner, changes in the size of the Hessian matrix can be absorbed by changes in the number of projection relationships (related to the number of times the error inference engine 20 is used) and the number of pieces of position information and posture information (related to the number of times the position inference engine 23 and the posture inference engine 24 are used). In other words, the size of the Hessian matrix is ​​reduced to the number of times the same inference engine is used, and the size of each divided inference engine can be kept constant and small. This facilitates learning for each divided inference engine.

[0081] [1-5. Experimental Results] An experiment was conducted to actually update map information using the map information update method according to this embodiment. In this experiment, the results of bundle adjustment performed using the map information update method according to this embodiment were compared with the results of bundle adjustment performed using a conventional map information update method. Here, g2o, which is publicly available software that uses a method of calculating update values ​​spatially geometrically without using inference, was used as the conventional map information update method.

[0082] Experimental results show that, like conventional map information update methods, the map information update method according to this embodiment can reduce the reprojection error in the output information more than the reprojection error in the input information. Furthermore, the map information update method according to this embodiment can reduce the amount of calculation, thereby reducing the time required for bundle adjustment to 1 / 10 or less. Furthermore, the map information update method according to this embodiment can reduce the average value of the reprojection error more than conventional map information update methods.

[0083] Therefore, according to the map information update method of this embodiment, bundle adjustment can be performed more accurately and in a shorter time than with conventional map information update methods.

[0084] (Embodiment 2) A map information updating method according to embodiment 2 will be described. The map information updating method according to this embodiment differs from the map information updating method according to embodiment 1 mainly in the reprojection error information acquisition step. Below, the map information updating method according to this embodiment will be described, focusing on the differences from the map information updating method according to embodiment 1.

[0085] The reprojection error information calculated in the reprojection error information acquisition step of the map information updating method according to this embodiment is a reprojection error calculated spatially geometrically from one piece of position information, one piece of orientation information, and projection coordinate information corresponding to one of one or more projection relationships. In other words, in this embodiment, the reprojection error is not inferred as in the map information updating method according to embodiment 1, but is calculated spatially geometrically.

[0086] As a result, in this embodiment, the reprojection error can be accurately calculated. Also in this embodiment, the position information update value and the attitude information update value are calculated by inference, and in this respect, the same effects as those of the map information update method according to the first embodiment can be achieved.

[0087] Furthermore, in this embodiment, the position information update value is inferred by a position inference engine, the posture information update value is inferred by a posture inference engine, and the re-projection error is calculated spatially geometrically. Similar to the inference engines according to embodiment 1, the position inference engine and the posture inference engine are engines that learn using the sum of re-projection errors calculated spatially geometrically based on the map information update value as a loss function. The position inference engine and the posture inference engine calculate the map information update value by inference based on the re-projection error calculated spatially geometrically based on the map information.

[0088] The position inference engine and posture inference engine according to this embodiment also achieve the same effects as those of position inference engine 23 and posture inference engine 24 according to the first embodiment.

[0089] (Hardware configuration) A hardware configuration for executing the methods according to the above-described embodiments will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of the hardware configuration of a computer 1000 for executing the methods according to the above-described embodiments by software. In other words, the computer 1000 can realize a map information updating device that executes the map information updating methods according to the first and second embodiments.

[0090] 9, the computer 1000 includes an input device 1001, an output device 1002, a CPU 1003, an internal storage 1004, a RAM 1005, a reading device 1007, a transmitting / receiving device 1008, and a bus 1009. The input device 1001, the output device 1002, the CPU 1003, the internal storage 1004, the RAM 1005, the reading device 1007, and the transmitting / receiving device 1008 are connected by the bus 1009.

[0091] The input device 1001 is a user interface device such as a keyboard, mouse, input buttons, touchpad, touch panel display, etc., and accepts user operations. Note that the input device 1001 may be configured to accept voice operations and remote operations using a remote control or the like in addition to accepting user touch operations.

[0092] The output device 1002 is a device that outputs a signal from the computer 1000, and may be a device that serves as a user interface such as a signal output terminal, a display, or a speaker.

[0093] The internal storage 1004 is a flash memory, etc. Furthermore, the internal storage 1004 may store in advance programs for executing the steps of the methods according to the first and second embodiments.

[0094] The RAM 1005 is a random access memory, and is used to store data calculated when a program or application is executed.

[0095] The reading device 1007 reads information from a recording medium such as a USB (Universal Serial Bus) memory. The reading device 1007 reads the above-mentioned programs and applications from a recording medium on which the programs and applications are recorded, and stores the programs and applications in the built-in storage 1004.

[0096] The transmitting / receiving device 1008 is a communication circuit for performing wireless or wired communication. The transmitting / receiving device 1008 communicates with, for example, a server device connected to a network, downloads the above-mentioned programs and applications from the server device, and stores them in the built-in storage 1004.

[0097] The CPU 1003 is a central processing unit that copies programs, applications, etc. stored in the internal storage 1004 to the RAM 1005, and sequentially reads and executes instructions contained in the copied programs, applications, etc. from the RAM 1005.

[0098] (Variations, etc.) While the methods according to the present disclosure have been described above based on the embodiments, the present disclosure is not limited to these embodiments. As long as they do not deviate from the gist of the present disclosure, various modifications conceivable by those skilled in the art to the embodiments and other forms constructed by combining some of the components of the embodiments are also included within the scope of the present disclosure.

[0099] The following embodiments may also be included within the scope of one or more aspects of the present disclosure.

[0100] The methods according to the present disclosure may be realized as a computer program that is executed by a computer, or as a digital signal that is a computer program. Furthermore, the present disclosure may be realized as a non-transitory computer-readable recording medium, such as a CD-ROM, on which the computer program is recorded.

[0101] The present disclosure may also be a computer system including a microprocessor and a memory, the memory storing the computer program, and the microprocessor operating in accordance with the computer program.

[0102] Furthermore, the program or the digital signal may be recorded on the recording medium and transferred, or the program or the digital signal may be transferred via a network or the like, so that the program or the digital signal may be implemented by another independent computer system.

[0103] The above embodiments may be combined with each other. [Industrial Applicability]

[0104] The present disclosure can be used, for example, in VSLAM technology. [Explanation of symbols]

[0105] 10 Map information update device 20 Error Inference Engine 21 First Consolidation Section 22 Second Consolidation Section 23 Location Inference Engine 24 Pose inference engine 1000 computers 1001 Input Device 1002 Output device 1003 CPU 1004 Internal Storage 1005 RAM 1007 Reading device 1008 Transmitting and receiving device 1009 Bus C11, C12, C21, C22 Projected coordinate information KF1, KF2, KF3 keyframes LM1, LM2, LM3 Landmarks

Claims

1. A map information update method, comprising: obtaining one or more projective relationships; obtaining reprojection error information for each of the one or more projection relationships; determining, for each of one or more landmarks, a first sum based on all of the reprojection error information associated with that landmark; for each of one or more key frames, determining a second sum based on all of the reprojection error information associated with that key frame; for each of the one or more landmarks, deducing a location information update value from the first sum, the location information update value being an update value of the location information of the landmark, and updating the location information of the landmark using the location information update value; for each of the one or more key frames, deducing a posture information update value from the second sum, the posture information update value being an update value of the posture information for that key frame, and updating the posture information for that key frame using the posture information update value. How to update map information.

2. 1. A map information updating method for updating map information including one or more pieces of position information each associated with one or more landmarks and one or more pieces of posture information each associated with one or more key frames, the method comprising: each of the one or more key frames is a photographed image that is photographed so as to include at least one of the one or more landmarks; each of the one or more attitude information includes position and orientation information; a projection relationship acquisition step of acquiring one or more projection relationships, wherein each of the one or more projection relationships is a relationship between each of the one or more landmarks, each of the one or more key frames, and projection coordinate information corresponding to coordinates on the key frames of projection points when each of the one or more landmarks is projected onto each of the one or more key frames; a reprojection error information acquisition step of acquiring, for each of the one or more projection relationships, reprojection error information based on position information associated with one of the one or more landmarks constituting the projection relationship among the one or more pieces of position information, orientation information associated with one of the one or more key frames constituting the projection relationship among the one or more pieces of orientation information, and the projection coordinate information constituting the projection relationship, and associating the reprojection error information with the projection relationship; a landmark-origin error aggregation step of generating, for each of the one or more landmarks, a first reprojection error information group in which all of the reprojection error information associated with the landmark is aggregated, and calculating a first total value based on all of the reprojection error information included in the first reprojection error information group; a keyframe-origin error aggregation step of generating a second reprojection error information group in which all of the reprojection error information associated with each of the one or more keyframes is aggregated, and calculating a second total value based on all of the reprojection error information included in the second reprojection error information group; a location information updating step of estimating, for each of the one or more landmarks, a location information update value that is an update value of the location information of the landmark among the one or more pieces of location information from the first sum value, and updating the location information of the landmark using the location information update value; and a posture information updating step of estimating, for each of the one or more key frames, a posture information update value that is an update value of posture information of the key frame among the one or more posture information from the second sum value, and updating the posture information of the key frame using the posture information update value. How to update map information.

3. The reprojection error information includes a feature amount indicating a result of inferring a reprojection error from one of the one or more pieces of position information, one of the one or more pieces of orientation information, and the projection coordinate information, which corresponds to one of the one or more projection relationships, and the reprojection error is spatially geometrically calculated from the position information, the orientation information, and the projection coordinate information. The map information updating method according to claim 2 .

4. The reprojection error information is a reprojection error spatially geometrically calculated from one of the one or more pieces of position information, one of the one or more pieces of orientation information, and the projection coordinate information, which corresponds to one of the one or more projection relationships. The map information updating method according to claim 2 .

5. the location information update is inferred by a location inference engine; the pose information update is inferred by a pose inference engine; the reprojection error information is inferred by an error inference engine; the position inference engine, the posture inference engine, and the error inference engine are engines that are trained using a sum of reprojection errors calculated spatially and geometrically based on the updated values ​​of the map information as a loss function; The updated value of the map information is calculated by the position inference engine and the posture inference engine based on the reprojection error information inferred by the error inference engine based on the map information.

4. The map information updating method according to claim 3.

6. the location information update is inferred by a location inference engine; the pose information update is inferred by a pose inference engine; The reprojection error is calculated spatially geometrically; the position inference engine and the posture inference engine are engines that are trained using a sum of the reprojection errors spatially geometrically determined based on the updated value of the map information as a loss function; The updated value of the map information is calculated by the position inference engine and the posture inference engine based on the reprojection error calculated spatially and geometrically based on the map information.

5. The map information updating method according to claim 4.

7. The feature amount includes a term that is commonly included in blocks that include diagonal elements of a Hessian matrix used to calculate an updated value of the map information.

4. The map information updating method according to claim 3.

Images