Information processing device, information processing method, program, and circuit information

Abstract

The first feature point in the first image and the second feature point in the second image are accurately mapped using a simple process. [Solution] The information processing device according to the embodiment includes a processing unit. The processing unit generates a matching problem, which is a combinatorial optimization problem, based on N first feature points detected from a first image and M second feature points detected from a second image different from the first image, to associate each of the N first feature points with one of the M second feature points that is presumed to represent the same object. Based on the solution to the matching problem, the processing unit generates a correspondence point list representing a correspondence point pair that includes one of the N first feature points and the corresponding second feature point from the M second feature points.

Description

[Technical Field]

[0001] Embodiments of the present invention relate to an information processing device, an information processing method, a program, and circuit information. [Background technology]

[0002] Autonomous mobile systems that autonomously move mobile objects such as self-driving cars and automated guided vehicles are known. These autonomous mobile systems estimate the position of the mobile object within its surrounding environment in order to move it appropriately. SLAM (Simultaneous Localization and Mapping) is a well-known technique for estimating the position of a mobile object.

[0003] One type of SLAM (Simulation-Likely Localization) is Visual SLAM, which uses images captured by a camera to estimate the camera's own position. Visual SLAM tracks feature points detected from captured images and estimates the camera's own position based on the changes in the position of these feature points across a series of consecutive images. When estimating the camera's own position using Visual SLAM, feature point matching must be performed between two captured images. Therefore, when estimating the camera's own position using Visual SLAM, a technique is needed to accurately match feature points between two captured images.

[0004] One known Visual SLAM technique is called ORB SLAM. ORB SLAM selects pairs of feature points with high similarity in their local features as corresponding point pairs. However, when an image contains multiple objects with similar shapes, such as trees, lights, or windows, ORB SLAM may incorrectly match two feature points that are spatially different, even if they are judged to be a pair if they are highly similar.

[0005] Furthermore, a technique is known that uses neural networks to incorporate coordinate information into local features and select pairs of corresponding points. However, such techniques require large-scale computations using large-scale neural networks, and when applied to battery-powered autonomous vehicles and autonomous robots, the resulting high power consumption poses a problem.

[0006] Furthermore, there are known techniques to avoid mismatching, where two feature points that are far apart are mistakenly identified as a pair, by limiting the search range for pairs of corresponding points to a predetermined range. However, since such techniques define the search range uniformly on the image, it is difficult to impose appropriate constraints on both pairs of feature points that are far apart and pairs of feature points that are relatively close together, and there is a possibility of mismatching. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Patent No. 6658001 [Patent Document 2] Japanese Patent Publication No. 2021-060864 [Patent Document 3] Japanese Patent Publication No. 2019-145010 [Patent Document 4] Japanese Patent Publication No. 2019-159566 [Patent Document 5] Japanese Patent Publication No. 2021-043667 [Patent Document 6] Japanese Patent Publication No. 2021-043589 [Non-patent literature]

[0008] [Non-Patent Document 1] Raul Mur-Artal, JMM Montiel, Juan D. Tardos, “ORB-SLAM: a Versatile and Accurate Monocular SLAM System”, IEEE TRANSACTIONS ON ROBOTICS VOL 31, No.5, October 2015

Non-licensed Document 2

Non-licensed Document 4

Non-licensed Document 5

[0009] The problem that this invention aims to solve is to accurately associate a first feature point contained in a first image with a second feature point contained in a second image using a simple process. [Means for solving the problem]

[0010] The information processing apparatus according to this embodiment includes a processing unit. The processing unit generates a matching problem, which is a combinatorial optimization problem, based on N first feature points (N is an integer of 1 or more) detected from a first image and M second feature points (M is an integer of 1 or more) detected from a second image different from the first image, to associate each of the N first feature points with one of the M second feature points that is presumed to represent the same object. Based on the solution to the matching problem, the processing unit generates a correspondence point list representing a correspondence point pair that includes one of the N first feature points and the corresponding second feature point from the M second feature points. [Brief explanation of the drawing]

[0011] [Figure 1] A diagram showing the configuration of a mobile control system according to an embodiment. [Figure 2] A diagram showing the functional configuration of a self-localization device. [Figure 3] A diagram showing an example of the configuration of the estimation unit. [Figure 4] A diagram showing the first image, the second image, and an example of the camera's imaging position. [Figure 5] A flowchart showing the processing flow of the self-position estimation device according to the embodiment. [Figure 6] A diagram showing the functional configuration of the matching unit. [Figure 7] A diagram showing the functional configuration of the problem generation unit. [Figure 8] A diagram showing the functional configuration of the matching unit according to the first modified example. [Figure 9] A diagram showing the functional configuration of a self-position estimation device according to the second modified example. [Figure 10] A diagram showing the functional configuration of a self-position estimation device according to the third modified example. [Figure 11] A diagram showing multiple partial images obtained by dividing an input image. [Figure 12] A flowchart showing the processing flow of a self-position estimation device according to the third modified example. [Figure 13] A diagram showing multiple partial images where some parts overlap. [Figure 14] A diagram showing the functional configuration of the matching unit according to the fourth modified example. [Figure 15] A diagram illustrating the process of selecting and decoding candidate points. [Figure 16] A diagram illustrating a model of the Ising problem. [Figure 17] A diagram showing the internal variables used by the simulated branching algorithm. [Figure 18] A flowchart illustrating the processing flow of a simulated branching machine. [Figure 19] A diagram showing an example of a computer hardware configuration. [Modes for carrying out the invention]

[0012] Embodiments of the present invention will be described below with reference to the attached drawings.

[0013] Figure 1 shows the configuration of a mobile body control system 10 according to an embodiment. The mobile body control system 10 controls the movement of mobile bodies such as autonomous vehicles, unmanned transport vehicles, drones, and autonomous robots.

[0014] The mobile object control system 10 comprises a camera 12, a solver device 14, a self-position estimation device 20, and a control device 22.

[0015] Camera 12 is mounted on a mobile device. Camera 12 captures images of the surrounding environment of the mobile device. Camera 12 captures images of the surrounding environment at predetermined time intervals and generates input images. Camera 12 outputs the generated input images at predetermined time intervals.

[0016] Camera 12 may be a monocular camera or a stereo camera. Camera 12 may be a color camera, a monochrome camera, or an NIR (Near-InfraRed) camera. Camera 12 may be an RGB-D camera or LiDAR capable of measuring the distance to a subject.

[0017] The solver device 14 acquires a combinatorial optimization problem that maximizes or minimizes an objective function, and outputs the solution to the acquired combinatorial optimization problem. The objective function is a function that includes multiple decision variables. The solver device 14 outputs data representing the values ​​of each of the multiple decision variables as the solution.

[0018] The solver device 14 obtains a matching problem, which is a combinatorial optimization problem, from the self-localization device 20. The solver device 14 provides the solution to the matching problem obtained to the self-localization device 20. Note that if the self-localization device 20 solves the matching problem itself, the mobile control system 10 may be configured without the solver device 14.

[0019] The solver device 14 is installed on the mobile body on which the camera 12 is mounted, for example, together with the self-localization device 20. Alternatively, the solver device 14 may be connected to the self-localization device 20 by a network and installed in a location different from the mobile body. For example, the solver device 14 may be implemented on a server device on the network. Alternatively, for example, the solver device 14 may be included inside the self-localization device 20.

[0020] The solver device 14 is, for example, an Ising machine. The Ising machine may also be, for example, a simulated branching machine. The Ising machine and the simulated branching machine will be described in detail later.

[0021] The self-position estimation device 20 is an information processing device that includes a receiving circuit, a processor, ROM (Read Only Memory), RAM (Random Access Memory), and an accelerator.

[0022] The receiving circuit acquires input images captured from the camera 12 at predetermined time intervals. The processor is, for example, a CPU (Central Processing Unit). The processor may be a multiprocessor. Alternatively, the processor may be a DSP (Digital Signal Processor), a GPU (Graphic Processing Unit), or a circuit combining these. The processor may also include an ISP (Image Signal Processor).

[0023] ROM stores programs executed by the processor and parameters used by the processor. RAM stores input images received from camera 12, data executed by the processor and accelerator, etc. The accelerator performs predetermined arithmetic processing such as image processing and signal processing on behalf of the processor. The accelerator can be implemented as a dedicated circuit, a CPU, a DSP, a GPU, or an NPU (Neural Processing Unit), or by any of these circuits.

[0024] A self-position estimation device 20 with this configuration acquires input images from the camera 12 at predetermined time intervals. Based on the input images at predetermined time intervals, the self-position estimation device 20 estimates its own position, which is the position of the camera 12 in three-dimensional space. The self-position estimation device 20 may also estimate the orientation of the camera 12 in three-dimensional space as its own position. Since the camera 12 is mounted on a moving object, the self-position estimation device 20 can estimate the position and orientation of the camera 12 as the position and orientation of the moving object.

[0025] Furthermore, during the process of estimating its own position, the self-position estimation device 20 generates a matching problem, which is a combinatorial optimization problem. The self-position estimation device 20 provides the generated matching problem to the solver device 14 and obtains a solution to the matching problem from the solver device 14. Then, the self-position estimation device 20 performs the self-position estimation process based on the obtained solution to the matching problem. Note that if the matching problem is a problem that can be easily solved, the self-position estimation device 20 may solve the matching problem itself without providing it to the solver device 14.

[0026] Such a self-position estimation device 20 estimates its own position at regular intervals and provides the estimated self-position to the control device 22 at regular intervals.

[0027] The control device 22 acquires the self-position estimated by the self-position estimation device 20, for example, at regular intervals, and controls the drive mechanism of the moving body, etc., based on the acquired self-position.

[0028] Such a mobile object control system 10 can appropriately move a mobile object, for example, from a starting position to a target position.

[0029] Figure 2 shows the functional configuration of the self-position estimation device 20.

[0030] The self-localization device 20 comprises an image processing unit 32, a feature point detection unit 34, a feature quantity calculation unit 36, a past information storage unit 38, a matching unit 40, and an estimation unit 42. The image processing unit 32, the feature point detection unit 34, the feature quantity calculation unit 36, the matching unit 40, and the estimation unit 42 are included in the processing unit of the information processing device.

[0031] The image processing unit 32 acquires input images from the camera 12 at predetermined time intervals. The image processing unit 32 performs image processing on each input image acquired at predetermined time intervals. For example, if the camera 12 outputs an input image that is a RAW image, the image processing unit 32 may convert the RAW image to an RGB image or a monochrome image. Furthermore, the image processing unit 32 may perform processing such as demosaicing, noise reduction, image quality adjustment, distortion correction, and shape transformation on the acquired input images.

[0032] Furthermore, if the self-localization device 20 is equipped with an ISP as hardware, the image processing unit 32 may use the ISP to perform image transformation processing. For example, the image processing unit 32 may temporarily store the acquired input image data in RAM and then perform various processes on the input image stored in RAM.

[0033] The feature point detection unit 34 acquires input images processed by the image processing unit 32 at predetermined time intervals. The feature point detection unit 34 detects one or more feature points from each input image at predetermined time intervals. The number of feature points detected by the feature point detection unit 34 may be the same or different for each input image at predetermined time intervals.

[0034] Each of the one or more feature points represents a characteristic object in the input image. The feature point detection unit 34 detects feature points from the input image using a predetermined algorithm. The feature point detection unit 34 may use any algorithm to detect feature points. For example, the feature point detection unit 34 may detect feature points using the FAST (Features from Accelerated Segment Test) algorithm. Alternatively, the feature point detection unit 34 may use an algorithm such as Harris's corner detector to detect feature points. Furthermore, the feature point detection unit 34 may use a neural network to detect feature points. If the accelerator has the function to execute a feature point detection algorithm, the feature point detection unit 34 provides the accelerator with the input image and detects feature points.

[0035] The feature point detection unit 34 generates a feature point list for each input image at predetermined time intervals, representing the coordinates of one or more feature points on each image. The feature point list may also include a feature point score, which is some parameter calculated during the feature point detection process, for each of the one or more feature points. Furthermore, if feature points are detected using an algorithm that utilizes a hierarchical structure, such as FAST, the feature point list may further include an identifier that identifies the hierarchy for each of the one or more feature points.

[0036] The feature point detection unit 34 then provides the feature quantity calculation unit 36 ​​with a pair of input image and feature point list for each input image at a predetermined time interval.

[0037] The feature calculation unit 36 ​​obtains a pair of an input image and a list of feature points for each input image at a predetermined time interval. For each input image at a predetermined time interval, the feature calculation unit 36 ​​calculates features from the input image using a predetermined algorithm for each of the one or more feature points included in the list of feature points. The feature calculation unit 36 ​​may use any algorithm to calculate the features.

[0038] For example, the feature calculation unit 36 ​​calculates features using algorithms such as SIFT (Scale-Invariant Feature Transform), SURF (Speed-Upped Robust Feature), BRIEF (Binary Robust Independent Elementary Features), or AKAZE (Accelerated KAZE). Alternatively, the feature calculation unit 36 ​​may calculate features using neural networks such as ResNet or Vision_Transformer. Furthermore, the feature calculation unit 36 ​​may include features not only related to the shape of the image, but also features related to the position within the image or the positional relationship with other feature points.

[0039] Furthermore, the feature calculation unit 36 ​​may perform the feature calculation process simultaneously with the feature detection process, rather than performing the feature calculation process after the feature point detection process. For example, the feature point detection unit 34 and the feature calculation unit 36 ​​may perform the feature point detection process and the feature calculation process simultaneously using the ORB (Oriented-BRIEF) algorithm. Alternatively, for example, the feature point detection unit 34 and the feature calculation unit 36 ​​may perform the feature point detection process and the feature calculation process simultaneously using the Super_Point algorithm shown in Non-Patent Literature 3.

[0040] The feature calculation unit 36 ​​generates a feature list containing the feature quantities for each of the one or more feature points in each input image at a predetermined time interval. The feature calculation unit 36 ​​then provides the feature point list and the feature list to the past information storage unit 38, the matching unit 40, and the estimation unit 42 for each input image at a predetermined time interval.

[0041] The past information storage unit 38 stores a list of feature points and a list of feature quantities for past input images. The list of feature points and the list of feature quantities for past input images stored in the past information storage unit 38 are referenced by the matching unit 40.

[0042] For example, the past information storage unit 38 stores a list of feature points and a list of feature quantities for the input image immediately preceding the input image to be processed by the matching unit 40. Alternatively, the past information storage unit 38 may also store a list of feature points and a list of feature quantities for keyframe images prior to the input image currently to be processed by the matching unit 40. A keyframe image is, for example, an input image selected at intervals longer than a predetermined time interval in which the input image is captured. A keyframe image may also be an input image that satisfies predetermined conditions. A keyframe image may also be an input image shown in the map information stored in the estimation unit 42, which will be described later. The past information storage unit 38 may delete the list of feature points and a list of feature quantities for past input images that are no longer referenced by the matching unit 40 after a certain amount of time has passed.

[0043] The matching unit 40 acquires a feature point list and a feature quantity list for each input image at predetermined time intervals.

[0044] Here, the input image currently being processed by the matching unit 40 from among the input images at predetermined time intervals is defined as the first image. Furthermore, input images prior to the first image, but different from the first image currently being processed by the matching unit 40, are defined as the second image. For example, the second image is the input image immediately preceding the first image. Alternatively, for example, the second image may be a keyframe image prior to the first image.

[0045] The matching unit 40 obtains a list of feature points and a list of feature quantities detected and calculated from the first image from the feature quantity calculation unit 36. The matching unit 40 also obtains a list of feature points and a list of feature quantities detected and calculated from the second image from the past information storage unit 38.

[0046] Furthermore, suppose that N first feature points (where N is an integer greater than or equal to 1) are detected as one or more feature points in the first image. Also, suppose that M second feature points (where M is an integer greater than or equal to 1) are detected as one or more feature points in the second image.

[0047] The matching unit 40 generates a matching problem based on N first feature points detected from the first image and M second feature points (where M is an integer greater than or equal to 1) detected from the second image. The matching problem is a combinatorial optimization problem in which each of the N first feature points is associated with one of the M second feature points that is presumed to represent the same object.

[0048] The matching unit 40 provides the generated matching problem to the solver device 14 and obtains a solution to the matching problem from the solver device 14. However, if the matching problem is easily solvable, the matching unit 40 may solve the matching problem itself without providing it to the solver device 14.

[0049] The matching unit 40 associates each of the N first feature points with one of the M second feature points that is presumed to represent the same object, based on the solution to the matching problem. The matching unit 40 associates each of the N first feature points with either zero or one of the M second feature points, and associates each of the M second feature points with either zero or one of the N first feature points. In other words, the matching unit 40 does not associate two or more second feature points with each of the N first feature points, and does not associate two or more first feature points with each of the M second feature points. Furthermore, the matching unit 40 may include first feature points among the N first feature points that are not associated with any second feature points, and may also include second feature points among the M second feature points that are not associated with any first feature points.

[0050] The matching unit 40 then generates a list of corresponding point pairs based on the solution to the matching problem. Each corresponding point pair includes one of the N first feature points and one of the M second feature points that is associated with the same object. The list of corresponding point pairs may represent multiple corresponding point pairs. Alternatively, the list of corresponding point pairs may not represent any corresponding point pairs, i.e., it may represent zero corresponding point pairs.

[0051] The matching unit 40 provides the estimation unit 42 with a list of corresponding points that it has generated for each input image at a predetermined time interval. Further details about the matching unit 40 will be described later.

[0052] The estimation unit 42 acquires a feature point list, a feature quantity list, and a corresponding point list for each input image at predetermined time intervals. Then, at predetermined time intervals, the estimation unit 42 estimates the self-position of the camera 12, which is the position of the camera 12 in three-dimensional space, based on the acquired feature point list, feature quantity list, and corresponding point list. The estimation unit 42 may also estimate the orientation of the camera 12 in three-dimensional space as part of the self-position. Then, the estimation unit 42 provides the estimated self-position to the control device 22.

[0053] Figure 3 shows an example of the configuration of the estimation unit 42.

[0054] The estimation unit 42 includes a map information storage unit 44, a motion estimation unit 46, a tracking unit 48, a keyframe determination unit 50, a keyframe registration unit 52, a local map update unit 54, a global map update unit 56, and an output unit 58.

[0055] The map information storage unit 44 stores map information. The map information includes keyframe information and map point information.

[0056] The keyframe information includes, for each of the one or more keyframe images, a keyframe identifier that identifies the keyframe image, the self-position of the camera 12 in the world coordinate system at the time the keyframe image was captured, and a list of feature points detected from the keyframe image.

[0057] Map point information includes, for each of one or more map points, an identifier for the map point, its position in the world coordinate system, and identification information that identifies the map point's feature quantities. A map point represents an object detected as a feature point. Map point information may also include information indicating the keyframe in which the map point is contained.

[0058] Furthermore, the keyframe information may include information representing the relationship between each of the one or more keyframe images and other keyframe images. For example, the keyframe information may include information representing the relative positional relationship between the camera's own position and the camera's own position at the time of capturing the other keyframe images, for each of the one or more keyframe images. In addition, the keyframe information may include information indicating the included map points and information indicating the relative positional relationship between the included map points and the camera's own position, for each of the one or more keyframe images.

[0059] The motion estimation unit 46 obtains a list of corresponding points for each input image at a predetermined time interval. For each input image at a predetermined time interval, the motion estimation unit 46 estimates the self-position of the camera 12 that captured the target input image. For example, the motion estimation unit 46 estimates the relative amount of movement from the position of the camera 12 that captured the past input image to the position of the camera 12 that captured the input image to be processed by solving a PnP (Perspective-n-Point) problem from the corresponding point pairs included in the list of corresponding points. Then, for example, the motion estimation unit 46 estimates the self-position of the camera 12 that captured the past input image based on the already estimated self-position of the camera 12 that captured the past input image and the estimated relative amount of movement. Note that the motion estimation unit 46 may estimate its self-position by methods other than this estimation method.

[0060] The tracking unit 48 associates each of the one or more map points represented by the map point information with one or more feature points included in the feature point list of the input image to be processed. Then, based on the self-position, feature point list, and feature quantity list calculated by the motion estimation unit 46, the tracking unit 48 optimizes each of the one or more map points and updates the map point information.

[0061] The tracking unit 48 may also update information about one or more map points on the local map. The local map represents a three-dimensional range of the neighborhood of the input keyframe image, which is the most recent keyframe image.

[0062] The keyframe determination unit 50 determines whether the input image to be processed is a keyframe image. For example, the keyframe determination unit 50 determines whether an input image is a keyframe image for each predetermined number of input images. Alternatively, the keyframe determination unit 50 may determine that an input image is a keyframe image if it satisfies predetermined conditions.

[0063] If the keyframe registration unit 52 determines that the input image is a keyframe image, it updates the keyframe information stored in the map information storage unit 44 with the input image as the keyframe image.

[0064] The local map update unit 54 updates the positions of one or more map points on the local map and the position of the keyframe image itself each time a keyframe image is registered, or each time a predetermined number of keyframe images are registered.

[0065] The local map update unit 54 acquires the camera's own position at the time the input keyframe image was captured, the relationship between the input keyframe image and past keyframe images that are temporally close to the input keyframe image, a list of feature points in the input keyframe image, and keyframe information of past keyframe images that are temporally close to the input keyframe image. The input keyframe image is the keyframe image to be updated. Based on this acquired information, the local map update unit 54 calculates the position of keypoints by triangulation and performs bundle adjustments, etc., to optimize the position of map points on the local map and the camera's own position at the time the input keyframe image was captured. Furthermore, the local map update unit 54 may also delete redundant map points and redundant keyframe images.

[0066] The global map update unit 56 updates the position of one or more map points on the global map, and the camera 12's own position at the time of imaging for each of the one or more keyframe images. The global map represents a wider three-dimensional range than the local map, for example, the three-dimensional range from the position where the moving object started moving to its current position. For example, the global map update unit 56 optimizes the camera 12's own position at the time of imaging for each of the one or more map points on the global map and for each of the multiple keyframe images by performing bundle adjustments, etc.

[0067] The local map update unit 54 can perform optimization processing with a relatively small amount of processing power. However, the local map update unit 54 accumulates errors in the self-position of keyframe images, resulting in relatively poor optimization accuracy. On the other hand, the global map update unit 56 has good optimization accuracy, but requires a large amount of processing power. Therefore, the global map update unit 56 performs optimization processing less frequently than the local map update unit 54. For example, the global map update unit 56 may perform optimization processing at predetermined time intervals, or it may perform optimization processing every time a certain number of keyframe images are registered. Furthermore, the global map update unit 56 may perform optimization processing when the self-position of the camera 12 at the time of keyframe image capture returns to an already registered spatial region, that is, when the movement path loops.

[0068] The output unit 58 outputs its current position to the control device 22 at predetermined intervals. The output unit 58 may also output identification information to the control device 22 that identifies the position and feature quantities of one or more map points.

[0069] Figure 4 shows an example of the first image, the second image, and the imaging position of camera 12.

[0070] In Figure 4, the first image is an image of stationary objects q1, q2, and q3 captured by camera 12 at the first time step (t1). In Figure 4, the second image is an image of q1, q2, and q3 captured by camera 12 at the second time step (t2), which is earlier than the first time step (t1). The imaging position of camera 12 is different between the first time step (t1) and the second time step (t2).

[0071] p A 1, p A 2 and p A Each of the three is a feature point included in the first image. B 1, p B 2 and p B Each of the three is a feature point included in the second image. A1 and p B 1 represents q1 on the image. p A 2 and p B 2 represents q2 on the image. p A 3 and p B 3 represents q3 on the image.

[0072] In this embodiment, the self-position estimation device 20 detects N first feature points as one or more feature points from the first image. Then, the self-position estimation device 20 generates a feature point list (P A ) shown in Equation (1).

Number

[0073] Also, in this embodiment, the self-position estimation device 20 calculates N first feature amounts as one or more feature amounts from the first image. The N first feature amounts correspond one-to-one to the N first feature points. Each of the N first feature amounts is the feature amount of the corresponding first feature point among the N first feature points. Then, the self-position estimation device 20 generates a first feature amount list (D A ) shown in Equation (2).

Number

[0074] In Equation (2), d A i represents the feature amount of p A i . i is an arbitrary integer from 1 to N inclusive.

[0075] Also, in this embodiment, the self-position estimation device 20 detects M second feature points as one or more feature points from the second image. Then, the self-position estimation device 20 generates a feature point list (P B ) shown in Equation (3).

Number

[0076] Furthermore, in this embodiment, the self-localization device 20 calculates M second features as one or more feature quantities from the second image. The M second features correspond one-to-one with the M second feature points. Each of the M second features is a feature quantity of the corresponding second feature point among the M second feature points. The self-localization device 20 then calculates the second feature list (D) shown in equation (4). B ) generates.

number

[0077] In equation (4), d B j is, p B j This represents a feature of the model. j is any integer between 1 and M (inclusive).

[0078] In the example shown in Figure 4, the self-localization device 20 uses three feature points (p) detected from the first image. A 1, p A 2, p A 3) Each of the three feature points (p) detected from the second image B 1, p B 2, p B 3) It associates with any of the feature points that are presumed to be the same object. In the example in Figure 4, the self-position estimation device 20 associates the feature point (p A 1) and the feature point (p) which is the image of q1 in the second image. B 1) Correspond with. Also, in the example of Figure 4, the self-localization device 20 identifies the feature point (p) which is the image of q2 in the first image. A 2) and the feature point (p) which is the image of q2 in the second image. B 2) Correspond with. Also, in the example of Figure 4, the self-localization device 20 identifies the feature point (p) which is the image of q3 in the first image. A 3) and the feature point (p) which is the image of q3 in the second image. B 3) Match it with.

[0079] In this embodiment, the self-localization device 20 generates a correspondence point list for the first image. The correspondence point list represents correspondence point pairs. Each correspondence point pair includes one of the N first feature points and the corresponding second feature point from among the M second feature points.

[0080] For example, the correspondence point list may include a value indicating whether or not a pair of correspondence points exists for all combinations of N first feature points and M second feature points.

[0081] In this embodiment, the self-position estimation device 20 calculates the corresponding point list (M) shown in equation (5) for the first image. A,B ) generates.

number

[0082] In equation (5), m A,B i,j is, p A i and p B j This value indicates whether or not a corresponding point pair containing exists. That is, m A,B i,j is, p A i and p B j This indicates whether or not two things are associated with the same object.

[0083] Figure 5 is a flowchart showing the processing flow of the self-position estimation device 20 according to the embodiment. The self-position estimation device 20 performs processing according to the flow shown in Figure 5.

[0084] The self-position estimation device 20 acquires input images from the camera 12 at predetermined time intervals. Each time the self-position estimation device 20 acquires an input image, it uses the acquired input image as the first image and executes the processes from S12 to S21 (loop processing between S11 and S22).

[0085] First, in S12, the self-position estimation device 20 performs image processing on the first image. For example, the self-position estimation device 20 converts the RAW image to an RGB image or a monochrome image, or performs demosaicing, noise reduction, image quality adjustment, distortion correction, and shape transformation processing on the first image.

[0086] Next, in S13, the self-localization device 20 detects N first feature points from the first image. Each of the N first feature points is a feature point included in the first image.

[0087] Next, in S14, the self-localization device 20 calculates N first features from the first image. Each of the N first features corresponds one-to-one with the N first feature points detected in S13. Each of the N first features is a feature at the corresponding first feature point among the N first feature points.

[0088] Next, in S15, the self-localization device 20 acquires M second feature points detected from the second image. The second image is an input image acquired prior to the first image. For example, the second image may be an input image acquired immediately before the first image. Alternatively, for example, the second image may be a keyframe image registered prior to the first image.

[0089] Next, in S16, the self-localization device 20 acquires M second features calculated from the second image. Each of the M second features corresponds one-to-one with the M second feature points acquired in S15. Each of the M second features is a feature at the corresponding second feature point among the M second feature points.

[0090] Next, in S17, the self-localization device 20 generates a matching problem based on N first feature points and N first feature quantities, and M second feature points and M second feature quantities. The matching problem is a combinatorial optimization problem in which each of the N first feature points is associated with one of the M second feature quantities that is estimated to represent the same object.

[0091] Next, in S18, the self-localization device 20 obtains the solution to the matching problem. The self-localization device 20 provides the matching problem to the solver device 14 for solving. Then, the self-localization device 20 obtains the solution to the matching problem from the solver device 14. Alternatively, if the matching problem is easily solvable, the self-localization device 20 may solve the matching problem itself.

[0092] Next, in S19, the self-localization device 20 generates a list of corresponding points representing corresponding point pairs based on the solution to the matching problem. Each corresponding point pair includes one of the N first feature points and the corresponding second feature point from among the M second feature points.

[0093] Next, in S20, the self-position estimation device 20 estimates its own position, which is the position of the camera 12 that captured the first image in three-dimensional space, based on the correspondence point list. The self-position estimation device 20 may also estimate the orientation of the camera 12 in three-dimensional space. Then, the self-position estimation device 20 provides the control device 22 with the self-position estimated for the first image.

[0094] Next, in S21, the self-localization device 20 stores the N first feature points and N first feature quantities detected from the first image as M second feature points and M second feature quantities detected from the second image. As a result, the self-localization device 20 can obtain M second feature points and M second feature quantities in the next processing of the first image.

[0095] Furthermore, if the second image is a keyframe image registered before the first image, the self-localization device 20 stores N first feature points and N first feature quantities as M second feature points and M second feature quantities, provided that the first image was registered as a keyframe image. In other words, if the second image is a keyframe image registered before the first image, the self-localization device 20 will not execute the process in S21 unless the first image is a keyframe image.

[0096] The self-position estimation device 20 repeats the processing steps S12 to S21 for each input image until the movement of the moving object is completed (S22). Then, when the movement of the moving object is completed, the self-position estimation device 20 terminates this flow.

[0097] Figure 6 shows the functional configuration of the matching unit 40.

[0098] The matching unit 40 includes a first acquisition unit 62, a second acquisition unit 64, a problem generation unit 66, a solution unit 68, and a corresponding point list generation unit 70.

[0099] The first acquisition unit 62 acquires a first feature point list representing N first feature points detected from the first image. The first acquisition unit 62 also acquires a first feature quantity list representing N first feature quantities calculated from the first image. The first acquisition unit 62 provides the N first feature points represented by the first feature point list and the N first feature quantities represented by the first feature quantity list to the problem generation unit 66.

[0100] The second acquisition unit 64 acquires a list of second feature points representing the M second feature points detected from the second image. The first acquisition unit 62 acquires a list of second features representing the M second features calculated from the second image. The second acquisition unit 64 provides the M second feature points represented by the second feature point list and the M second features represented by the second feature feature list to the problem generation unit 66.

[0101] The problem generation unit 66 generates a matching problem based on N first feature points and N first feature quantities, and M second feature points and M second feature quantities. The problem generation unit 66 then provides the generated matching problem to the solution unit 68.

[0102] The solution unit 68 provides the matching problem to the solver device 14 and obtains a solution to the matching problem. If the solver device 14 is a QUBO (Quadratic Unconstrained Binary Optimization) solver, the solution unit 68 converts the objective function included in the matching problem into a QUBO equation and provides it to the solver device 14. If the solver device 14 is an Ising problem, the solution unit 68 converts the objective function included in the matching problem into an Ising model and provides it to the solver device 14.

[0103] Furthermore, if the matching problem is easily solvable, the solving unit 68 may solve the matching problem itself without providing it to the solver device 14. The solving unit 68 provides the acquired solution to the corresponding point list generation unit 70.

[0104] The correspondence point list generation unit 70 generates a correspondence point list representing correspondence point pairs based on the solution to the matching problem. The correspondence point list generation unit 70 then provides the generated correspondence point list to the estimation unit 42.

[0105] Figure 7 shows the functional configuration of the problem generation unit 66.

[0106] The problem generation unit 66 includes a feature score generation unit 80, a location score generation unit 82, a co-occurrence score generation unit 84, a constraint score generation unit 86, an objective function generation unit 88, and a matching problem generation unit 90.

[0107] The feature score generation unit 80 obtains N first features calculated from the first image and M second features calculated from the second image. Based on the N first features and M second features, the feature score generation unit 80 generates (N × M) feature scores.

[0108] In this embodiment, the (N×M) feature scores are arranged in an N-row × M-column feature score matrix (R). In this embodiment, the i-row × j-column feature score among the (N×M) feature scores is in R ij It is expressed as follows.

[0109] Feature score of i rows × j columns (R ij This value is determined by the similarity between the feature quantity of the i-th first feature point (i.e., the i-th first feature quantity) and the feature quantity of the j-th second feature point (i.e., the j-th second feature quantity).

[0110] More specifically, if the matching problem is a combinatorial optimization problem that maximizes the objective function, then the feature score of i rows × j columns (R ij The value of ) increases as the similarity between the feature of the i-th first feature point (i.e., the i-th first feature) and the feature of the j-th second feature point (i.e., the j-th second feature) increases. Also, if the matching problem is a combinatorial optimization problem that minimizes the objective function, the feature score of i rows × j columns (R ij The value of ) becomes smaller as the similarity between the feature quantity of the i-th first feature point (i.e., the i-th first feature quantity) and the feature quantity of the j-th second feature point (i.e., the j-th second feature quantity) increases.

[0111] The similarity score, which represents the degree of similarity between the i-th first feature and the j-th second feature, is calculated differently depending on the algorithm used to calculate the features. For example, if the first and second features are calculated using the BRIEF algorithm, the similarity score between the i-th first feature and the j-th second feature may be the Hamming distance between them. Alternatively, the similarity score between the i-th first feature and the j-th second feature may be the cosine similarity between them. Furthermore, the similarity score between the i-th first feature and the j-th second feature may be the likelihood between them, calculated using a neural network or SVM (Support Vector Machine), etc.

[0112] Furthermore, each of the (N×M) feature scores may be normalized to a predetermined range after being calculated using the methods described above. For example, each of the (N×M) feature scores may be normalized to a range of -1 or greater and +1 or less. Alternatively, each of the (N×M) feature scores may be normalized to a predetermined range on a nonlinear curve such as a log curve. Additionally, each of the (N×M) feature scores may have an offset value added to or subtracted from the value calculated using the methods described above. Furthermore, each of the (N×M) feature scores may be discretized to a binary value such as -1 or +1, or to a predetermined number of discrete values.

[0113] The position score generation unit 82 acquires N first feature points detected from the first image and M second feature points detected from the second image. The feature score generation unit 80 generates (N × M) feature scores based on the N first feature points and M second feature points.

[0114] The (N×M) feature scores are arranged in an N x M position score matrix (S). In this embodiment, the i x j position score among the (N×M) position scores is in S ij It is expressed as follows.

[0115] Position score (S) of row i × column j ij This value corresponds to the estimated distance between the estimated position of the j-th second feature point in the first image at time and the position of the i-th first feature point.

[0116] More specifically, if the matching problem is a combinatorial optimization problem that maximizes the objective function, then the position score (S) of row i × column j is... ij The value of ) increases as the estimated distance between the estimated position of the j-th second feature point in the first image at time and the position of the i-th first feature point is smaller. If the matching problem is a combinatorial optimization problem that minimizes the objective function, the position score (S) is i rows × j columns. ij The value of ) becomes smaller the closer the estimated distance between the estimated position of the j-th second feature point in the first image at time and the position of the i-th first feature point.

[0117] For example, the distance between the estimated position of the j-th second feature point in the first image at time and the position of the i-th first feature point is the L2 norm of the difference between the estimated position of the j-th second feature point in the first image at time and the position of the i-th first feature point. Alternatively, the position score generation unit 82 may calculate a provisional list of corresponding points by matching only the features, and estimate the estimated position of the j-th second feature point in the first image at time based on this provisional list of corresponding points. In this case, the position score generation unit 82 may detect the first feature point that is paired with the j-th second feature point in the provisional list of corresponding points, and estimate the estimated position of the j-th second feature point in the first image at time by linear interpolation or the like using the amount of movement of the detected first feature point.

[0118] Furthermore, the position score generation unit 82 may calculate the optical flow from the second image to the first image and estimate the estimated position of the j-th second feature point at the time of the first image based on the optical flow at the position of the j-th second feature point. Alternatively, the position score generation unit 82 may estimate its own position and orientation using information output from sensors other than the camera 12, such as GPS (Global Positioning System) or IMU (Inertial Measurement Unit), and estimate the estimated position of the j-th second feature point at the time of the first image based on the estimated self-position and orientation.

[0119] Furthermore, the position score generation unit 82 may estimate the estimated image position of the j-th second feature point at time of the first image using a Kalman filter or the like, based on the tracking results of feature points in past input images. Alternatively, for example, the position score generation unit 82 may estimate the self-position of the first image from the tracking results of its own position in past frames, and then map the feature points in past frames to the first image whose self-position has been estimated, thereby estimating the estimated image position of the j-th second feature point at time of the first image. Note that the position score generation unit 82 may estimate the estimated position of the j-th second feature point at time of the first image using other estimation methods, not limited to the estimation methods described above.

[0120] Furthermore, the position score generation unit 82 generates a position score (S) in i rows × j columns if the first and second images were captured by the camera 12 which has been translated parallel to the optical axis. ij The position score may include an angular component. For example, the position score generation unit 82 calculates the angular difference between the line connecting the 3D position of the j-th second feature point and the 3D position of the i-th first feature point and the line extending radially from the vanishing point as the angular component. If the matching problem is a combinatorial optimization problem that maximizes the objective function, the position score (S) is calculated as i rows × j columns. ij The value of ) increases as the angular component decreases. If the matching problem is a combinatorial optimization problem that minimizes the objective function, the position score (S) in row i × column j is ij The value of ) becomes smaller as the angular component is smaller.

[0121] Furthermore, each of the (N×M) position scores may be normalized to a value within a predetermined range after being calculated using the method described above. For example, each of the (N×M) position scores may be normalized to a range of -1 or greater and +1 or less. Alternatively, each of the (N×M) position scores may be normalized to a predetermined range on a nonlinear curve such as a logarithmic curve. Additionally, each of the (N×M) position scores may have an offset value added to or subtracted from the value calculated using the method described above. Furthermore, each of the (N×M) position scores may be discretized to a binary value such as -1 or +1, or to a predetermined number of discrete values.

[0122] The co-occurrence score generation unit 84 acquires N first feature points detected from the first image and M second feature points detected from the second image. Based on the N first feature points and M second feature points, the co-occurrence score generation unit 84 generates ((N×M)×(N×M)) co-occurrence scores.

[0123] The ((N×M)×(N×M)) co-occurrence scores are arranged in a (N×M) row × (N×M) column co-occurrence score matrix (T).

[0124] In the co-occurrence score matrix (T), the (N × M) rows represent all combinations of N first feature points and M second feature points. In this embodiment, the row representing the (i,j) pair in the co-occurrence score matrix (T) shows the combination of the i-th first feature point and the j-th second feature point.

[0125] The (N×M) columns in the co-occurrence score matrix (T) represent all combinations of N first feature points and M second feature points. In this embodiment, the column representing the (i',j') pair in the co-occurrence score matrix (T) represents the combination of the i'-th first feature point and the j'-th second feature point. i' represents any integer between 1 and N, and j' represents any integer between 1 and M, inclusive.

[0126] In this embodiment, the co-occurrence scores of the row representing the (i,j) pair and the column representing the (i',j') pair among the ((N×M)×(N×M)) co-occurrence scores are T ij,i´j´ It is expressed as follows.

[0127] Co-occurrence score (T ij,i´j´ ) is a value that represents the likelihood that a pair of corresponding points between the i-th first feature point and the j-th second feature point, and a pair of corresponding points between the i'-th first feature point and the j'-th second feature point, can exist simultaneously.

[0128] For example, co-occurrence score (T ij,i´j´ ) is a value that corresponds to the degree of approximation between the first positional relationship between the image position of the i-th first feature point and the image position of the j-th second feature point, and the second positional relationship between the image position of the i'-th first feature point and the image position of the j'-th second feature point.

[0129] For example, if the matching problem is a combinatorial optimization problem where the objective function is maximized, the co-occurrence score (T ij,i´j´ The co-occurrence score (T) is larger the closer the first positional relationship between the image position of the i-th first feature point and the image position of the j-th second feature point, and the second positional relationship between the image position of the i'-th first feature point and the image position of the j'-th second feature point. Also, if the matching problem is a combinatorial optimization problem that minimizes the objective function, the co-occurrence score (T) is larger.ij,i´j´ The value is smaller the closer the first and second positional relationships are.

[0130] For example, the degree of approximation may be a value indicating that the first and second positional relationships are close if the first line connecting the image position of the i-th first feature point and the image position of the j-th second feature point does not intersect with the second line connecting the image position of the i'-th first feature point and the image position of the j'-th second feature point. Alternatively, for example, the degree of approximation may be a value indicating that the first and second positional relationships are not close if the first and second lines intersect.

[0131] For example, the degree of approximation may be a value corresponding to the difference in vector direction between the direction of the first vector from the image position of the i-th first feature point to the image position of the j-th second feature point and the direction of the second vector from the image position of the i'-th first feature point to the image position of the j'-th second feature point. For example, the degree of approximation may be the cosine similarity between the first vector and the second vector. Alternatively, for example, the degree of approximation may be the dot product of the first vector and the second vector.

[0132] For example, consider the case where two stationary objects, q1 and q2, exist in three-dimensional space, as shown in Figure 4. In the first image, q2 (the first feature point in the first image is p2) A )) is q1 (feature point (p1 in the first image) A It is captured to the right of )). In the second image as well, q2 (in the second image, the second feature point (p2 B )) is q1 (feature point (p1 in the second image) B It is captured to the right of ). When the object is stationary and the amount of movement of camera 12 is small, the positional relationship between q1 and q2 in the image is maintained between the first image and the second image.

[0133] If these conditions are met, p in the first image i A and p j B The positional relationship and p in the second image i´ A and pj´ B If the positional relationships with respect to each other are maintained, it can be said that it is reasonable for the two pairs of corresponding points to hold simultaneously. However, if the positional relationship between p i A and p j B in the first image and the positional relationship between p i´ A and p j´ B in the second image are not maintained, it can be said that it is not reasonable for the two pairs of corresponding points to hold simultaneously. Therefore, the co-occurrence score (T ij,i´j´ ) is set as a value corresponding to the degree of approximation between the first positional relationship between the image positions of the i-th first feature point and the j-th second feature point and the second positional relationship between the image positions of the i'-th first feature point and the j'-th second feature point, so that a score corresponding to the reasonableness of such simultaneous establishment can be expressed.

[0134] Also, for example, the co-occurrence score (T ij,i´j´ ) may be a value obtained by combining the first degree of approximation and the second degree of approximation. The first degree of approximation represents the degree of approximation between the first positional relationship between the image positions of the i-th first feature point and the j-th second feature point and the second positional relationship between the image positions of the i'-th first feature point and the j'-th second feature point. The second degree of approximation represents the degree of approximation between the third positional relationship between the estimated image position at the time of the first image of the j-th second feature point and the image position of the j-th second feature point and the fourth positional relationship between the image positions of the i'-th first feature point and the j'-th second feature point. The co-occurrence score (T ij,i´j´ ) may be a value synthesized by averaging the first degree of approximation and the second degree of approximation. Also, the co-occurrence score (T ij,i´j´ ) may be a value synthesized by weighting and averaging the first degree of approximation and the second degree of approximation. Such a co-occurrence score (T ij,i´j´ ) includes the estimated image position at the time of the first image of the j-th second feature point as an element, so that the reasonableness of simultaneous establishment can be expressed more accurately. Note that the fourth positional relationship may be the same as the second positional relationship.

[0135] Furthermore, if the first condition is met, the co-occurrence score (T ij,i´j´ The co-occurrence score (T) may be a value corresponding to the degree of approximation as described above. Also, if the first condition is met, the co-occurrence score (T) may be a value corresponding to the degree of approximation as described above. ij,i´j´ The co-occurrence score (T) may be a value corresponding to the difference in movement between the first movement from the image position of the i-th first feature point to the image position of the j-th second feature point and the second movement from the image position of the i'-th first feature point to the image position of the j'-th second feature point. Also, if the first condition is met, the co-occurrence score (T) may be a value corresponding to the difference in movement between the first movement from the image position of the i'-th first feature point to the image position of the j'-th second feature point. ij,i´j´ ) may be a value obtained by combining the first displacement difference and the second displacement difference by averaging or weighted averaging, etc. The first displacement difference represents the displacement difference from the image position of the i-th first feature point to the image position of the j-th second feature point. The second displacement difference represents the displacement difference between the third displacement from the estimated image position of the j-th second feature point at time to the image position of the j-th second feature point, and the fourth displacement from the image position of the i'-th first feature point to the image position of the j'-th second feature point.

[0136] The first condition is that the image position of the i-th first feature point and the image position of the i'-th first feature point are less than or equal to a predetermined distance. Alternatively, the first condition may also be that the image position of the i-th first feature point and the image position of the i'-th first feature point are less than or equal to a predetermined distance, AND the image position of the j-th second feature point and the image position of the j'-th second feature point are less than or equal to a predetermined distance.

[0137] For example, if the first condition is met and the matching problem is a combinatorial optimization problem that maximizes the objective function, the co-occurrence score (T ij,i´j´ The co-occurrence score (T) is larger the smaller the difference in displacement. For example, if the first condition is met and the matching problem is a combinatorial optimization problem that minimizes the objective function, the co-occurrence score (T) is larger. ij,i´j´ The smaller the difference in movement, the smaller the co-occurrence score (T ij,i´j´) is a value that indicates that it is not reasonable for the corresponding point pair of the i-th first feature point and the j-th second feature point and the corresponding point pair of the i'-th first feature point and the j'-th second feature point to exist simultaneously.

[0138] Also, when the second condition is satisfied, the co-occurrence score (T ij,i´j´ ) may be a value corresponding to the correlation between the size and the amount of movement from the vanishing point on the image of the first object represented by the corresponding point pair of the i-th first feature point and the j-th second feature point and the size and the amount of movement from the vanishing point on the image of the second object represented by the corresponding point pair of the i'-th first feature point and the j'-th second feature point. The second condition is a condition that the first image and the second image are images captured by a camera 12 that has been translated parallel to the optical axis, and the moving direction from the i-th first feature point to the j-th second feature point and the moving direction from the i'-th first feature point to the j'-th second feature point are radial directions from the vanishing point. For example, the co-occurrence score generation unit 84 determines whether or not the second condition is satisfied based on an external sensor or an estimation result of the self-position of a past input image.

[0139] For example, when the second condition is satisfied and the matching problem is a combinatorial optimization problem that maximizes the objective function, the co-occurrence score (T ij,i´j´ ) is larger as the correlation is larger. Also, for example, when the second condition is satisfied and the matching problem is a combinatorial optimization problem that minimizes the objective function, the co-occurrence score (T ij,i´j´ ) is smaller as the correlation is larger. However, when the second condition is not satisfied, the co-occurrence score (T ij,i´j´ ) is a value that indicates that it is not reasonable for the corresponding point pair of the i-th first feature point and the j-th second feature point and the corresponding point pair of the i'-th first feature point and the j'-th second feature point to exist simultaneously.

[0140] Also, when the third condition is satisfied, the co-occurrence score (T ij,i´j´) may be a value corresponding to the difference in movement between the movement direction and amount of movement of the first object, which is represented by the corresponding point pair between the i-th first feature point and the j-th second feature point, and the movement direction and amount of movement of the second object, which is represented by the corresponding point pair between the i'-th first feature point and the j'-th second feature point. The third condition is that the first and second images are images captured by the camera 12 whose optical axis has been rotated. For example, the co-occurrence score generation unit 84 determines whether the third condition is met based on the estimation result of the self-position of an external sensor or past input images.

[0141] For example, if the third condition is met and the matching problem is a combinatorial optimization problem that maximizes the objective function, the co-occurrence score (T ij,i´j´ The co-occurrence score (T) is larger the smaller the movement difference. Also, for example, if the third condition is met and the matching problem is a combinatorial optimization problem that minimizes the objective function, the co-occurrence score (T) is larger. ij,i´j´ The smaller the movement difference, the smaller the co-occurrence score (T). However, if the third condition is not met, the co-occurrence score (T ij,i´j´ ) is a value that indicates that it is not reasonable for a pair of corresponding points between the i-th first feature point and the j-th second feature point, and a pair of corresponding points between the i'-th first feature point and the j'-th second feature point, to exist simultaneously.

[0142] The second and third conditions are met when the movement of camera 12 is small, the rotation of camera 12 in the roll direction is minimal, and most of the objects are stationary. For example, if the movement of camera 12 is large, the positional relationships of stationary objects in the image will be reversed when comparing two images taken from the front and rear of the object. Similarly, if the rotation in the roll direction is large and camera 12 rolls 180 degrees, the positional relationships of stationary objects in the image will also be reversed. For this reason, the positional relationships between moving objects and stationary objects, or between moving objects and other moving objects, are not guaranteed. However, the positional relationships between feature points within the same moving object are guaranteed. Furthermore, if the shooting interval is short and the movement of the moving object is small, there is a high probability that the relative positional relationships will be guaranteed.

[0143] The second and third conditions assume that the rotation in the roll direction of the position where the first and second images were taken is small. For example, if the moving object is a device that undergoes large roll rotations in a short period of time, such as a drone, and the change in roll direction can be acquired by an external sensor, the self-position estimation device 20 corrects the roll direction of the input image before detecting feature points and calculates the co-occurrence score (T ij,i´j´ ) may be calculated. Also, if it is not possible to obtain the change in roll direction, the self-position estimation device 20 calculates a corrective roll rotation amount from the provisional self-position using the result of matching only the feature quantities, and corrects the roll direction of the input image based on the corrective roll rotation amount to calculate the co-occurrence score (T ij,i´j´ You may calculate ).

[0144] Furthermore, the co-occurrence score generation unit 84 simultaneously determines the second and third conditions, and if each condition is met, it generates a co-occurrence score (T ij,i´j´ ) may be calculated based on correlation and movement difference.

[0145] Furthermore, each of the ((N×M)×(N×M)) co-occurrence scores may be normalized to a value within a predetermined range after being calculated using the methods described above. For example, each of the ((N×M)×(N×M)) co-occurrence scores may be normalized to a range of -1 or greater and +1 or less. Alternatively, each of the ((N×M)×(N×M)) co-occurrence scores may be normalized to a predetermined range on a nonlinear curve such as a logarithmic curve. Additionally, each of the ((N×M)×(N×M)) co-occurrence scores may have an offset value added to or subtracted from the value calculated using the methods described above. Furthermore, each of the ((N×M)×(N×M)) co-occurrence scores may be discretized to a binary value such as -1 or +1, or to a predetermined number of discrete values.

[0146] The constraint score generation unit 86 generates ((N×M)×(N×M)) constraint scores based, for example, on pre-set values. In this embodiment, the ((N×M)×(N×M)) constraint scores are arranged in a constraint score matrix (U) with (N×M) rows and (N×M) columns.

[0147] In the constraint score matrix (U), the (N × M) rows represent all combinations of N first feature points and M second feature points. In this embodiment, the row representing the (i,j) pair in the constraint score matrix (U) represents the combination of the i-th first feature point and the j-th second feature point.

[0148] The (N×M) columns in the constraint score matrix (U) represent all combinations of N first feature points and M second feature points. In this embodiment, the column representing the (i',j') pair in the constraint score matrix (U) indicates the combination of the i'-th first feature point and the j'-th second feature point.

[0149] In this embodiment, the constraint scores of the row representing the pair (i,j) and the column representing the pair (i',j') among the ((N×M)×(N×M)) constraint scores are U ij,i´j´ It is expressed as follows.

[0150] The constraint scores (U) of the row representing the pair (i,j) and the column representing the pair (i',j'). ij,i´j´ This indicates whether a correspondence between the i-th first feature point and the j-th second feature point, and a correspondence between the i'-th first feature point and the j'-th second feature point, can be simultaneously established.

[0151] Constraint score (U ij,i´j´ The value (U) represents the case where i=i', and if j=j', then the constraint is valid, and if j≠j', then the constraint is not valid. ij,i´j´ ) represents a value that indicates that the condition is not met if j=j' when i≠i', and a value that indicates that the condition is met if j≠j'.

[0152] For example, if the matching problem is an optimization problem that maximizes the objective function, then the constraint score (U ij,i´j´The value of ) is a predetermined positive value of 1 if the correspondence between the i-th first feature point and the j-th second feature point and the correspondence between the i'-th first feature point and the j'-th second feature point can be simultaneously established, and is 0 or a predetermined negative value if they cannot be simultaneously established.

[0153] Also, for example, if the matching problem is an optimization problem that minimizes the objective function, the constraint score (U ij,i´j´ The value of ) is a predetermined negative value such as -1 if the correspondence between the i-th first feature point and the j-th second feature point and the correspondence between the i'-th first feature point and the j'-th second feature point can be simultaneously satisfied, and is 0 or a predetermined positive value if they cannot be simultaneously satisfied.

[0154] Such a constraint score matrix (U) can restrict the solution to the matching problem so that each of the N first feature points corresponds to either zero or one second feature point, and each of the M second feature points corresponds to either zero or one first feature point.

[0155] The objective function generation unit 88 obtains an N x M feature score matrix (R) from the feature score generation unit 80. The objective function generation unit 88 obtains an N x M location score matrix (S) from the location score generation unit 82. The objective function generation unit 88 obtains an (N x M) x (N x M) co-occurrence score matrix (T) from the co-occurrence score generation unit 84. The objective function generation unit 88 obtains an (N x M) x (N x M) constraint score matrix (U) from the constraint score generation unit 86.

[0156] The objective function generation unit 88 then generates an objective function (H(x)) as shown in equation (6).

number

[0157] The objective function (H(x)) includes (N×M) decision variables. In this embodiment, the (N×M) decision variables are arranged in an N-row, M-column variable matrix.

[0158] The objective function (H(x)) is minimized or maximized when all (N×M) decision variables are set to their correct values. More specifically, if the matching problem is a combinatorial optimization problem that maximizes the objective function (H(x)), then the objective function (H(x)) is maximized when all (N×M) decision variables are set to their correct values. If the matching problem is a combinatorial optimization problem that minimizes the objective function (H(x)), then the objective function (H(x)) is minimized when all (N×M) decision variables are set to their correct values.

[0159] The i-th row × j-th column of the (N × M) decision variables represents whether the i-th first feature point and the j-th second feature point are a corresponding point pair. If the i-th first feature point and the j-th second feature point represent the same object, the correct value for the i-th row × j-th column of the (N × M) decision variables is a value that indicates the i-th first feature point and the j-th second feature point are a corresponding point pair. If the i-th first feature point and the j-th second feature point do not represent the same object, the correct value for the i-th row × j-th column of the decision variable is a value that indicates the i-th first feature point and the j-th second feature point are not a corresponding point pair.

[0160] In this embodiment, the objective function (H(x)) is given by equation (7), which consists of an i-row × j-column decision variable and the i-row × j-column feature score matrix (R) ij This includes a term that multiplies by ).

number

[0161] C in equation (7) R This is a predetermined constant.

[0162] Feature score of i rows × j columns (R ijThe value ) is a value corresponding to the similarity between the feature quantity of the i-th first feature point and the feature quantity of the j-th second feature point. The similarity between the feature quantity of the i-th first feature point and the feature quantity of the j-th second feature point represents the validity of the i-th first feature point and the j-th second feature point being the same object, that is, the validity of the i-th first feature point and the j-th second feature point being a corresponding point pair.

[0163] Therefore, if the matching problem is a combinatorial optimization problem that maximizes the objective function (H(x)), equation (7) will have its maximum value when all (N × M) decision variables are set to the correct values. If the matching problem is a combinatorial optimization problem that minimizes the objective function (H(x)), equation (7) will have its minimum value when all (N × M) decision variables are set to the correct values.

[0164] In this embodiment, the objective function (H(x)) is given by equation (8), which consists of an i-row × j-column decision variable and the i-row × j-column position score (S) in the position score matrix (S). ij This includes a term that multiplies by ).

number

[0165] C in equation (8) S This is a predetermined constant.

[0166] Position score (S) of row i × column j ij This value corresponds to the estimated distance between the estimated position of the j-th second feature point in the first image at time and the position of the i-th first feature point. The estimated distance between the estimated position of the j-th second feature point in the first image at time and the position of the i-th first feature point represents the validity of the i-th first feature point and the j-th second feature point being the same object, that is, the validity of the i-th first feature point and the j-th second feature point being a corresponding point pair.

[0167] Therefore, when the matching problem is a combinatorial optimization problem that maximizes the objective function (H(x)), Equation (8) reaches its maximum value when all of the (N × M) decision variables are set to the correct values. When the matching problem is a combinatorial optimization problem that minimizes the objective function (H(x)), Equation (8) reaches its minimum value when all of the (N × M) decision variables are set to the correct values.

[0168] In this embodiment, the objective function (H(x)) includes a term that multiplies the decision variable of the i-th row × j-th column, the decision variable of the i'-th row × j'-th column, and the co-occurrence score T of the row representing the pair (i, j) and the column representing the pair (i', j') in the co-occurrence score matrix (T ij,i´j´ ). [Number]

[0169] C in Equation (9) T is a predetermined constant.

[0170] The co-occurrence score T of the row representing the pair (i, j) and the column representing the pair (i', j') ij,i´j´ is a value representing the validity that the corresponding point pair of the i-th first feature point and the j-th second feature point and the corresponding point pair of the i'-th first feature point and the j'-th second feature point exist simultaneously. That is, the co-occurrence score T ij,i´j´ represents the validity that the i-th first feature point and the j-th second feature point represent the same first object, and the i'-th first feature point and the j'-th second feature point represent the same second object.

[0171] Therefore, when the matching problem is a combinatorial optimization problem that maximizes the objective function (H(x)), Equation (9) reaches its maximum value when all of the (N × M) decision variables are set to the correct values. When the matching problem is a combinatorial optimization problem that minimizes the objective function (H(x)), Equation (9) reaches its minimum value when all of the (N × M) decision variables are set to the correct values.

[0172] In this embodiment, the objective function (H(x)) includes a term that multiplies a decision variable of i rows × j columns, a decision variable of i' rows × j' columns, and a row representing the (i, j) pair and a column representing the (i', j') pair in the constraint score matrix (U ij,i´j´ ).

Equation

[0173] C in Equation (10) U is a predetermined constant.

[0174] The constraint score (U ij,i´j´ ) of the row representing the (i, j) pair and the column representing the (i', j') pair represents a value indicating that it is feasible if j = j' when i = i', and represents a value indicating that it is infeasible if j ≠ j'. Also, the constraint score (U ij,i´j´ ) represents a value indicating that it is infeasible if j = j' when i ≠ i', and represents a value indicating that it is feasible if j ≠ j'.

[0175] Therefore, when the matching problem is a combinatorial optimization problem that maximizes the objective function (H(x)), Equation (10) reaches its maximum value when values are substituted such that the number of second feature points corresponding to each of the N first feature points is 0 or 1, and the number of first feature points corresponding to each of the M second feature points is 0 or 1. When the matching problem is a combinatorial optimization problem that minimizes the objective function (H(x)), Equation (10) reaches its minimum value when values are substituted such that the number of second feature points corresponding to each of the N first feature points is 0 or 1, and the number of first feature points corresponding to each of the M second feature points is 0 or 1.

[0176] The objective function (H(x)) may be configured to not include one or two of equations (7), (8), and (9). If the objective function (H(x)) does not include equation (7), the problem generation unit 66 will not include the feature score generation unit 80. If the objective function (H(x)) does not include equation (8), the problem generation unit 66 will not include the location score generation unit 82. If the objective function (H(x)) does not include equation (9), the problem generation unit 66 will not include the co-occurrence score generation unit 84.

[0177] The matching problem generation unit 90 obtains the objective function (H(x)) shown in equation (6) generated by the objective function generation unit 88. The matching problem generation unit 90 then generates an optimization problem that maximizes the objective function (H(x)) as shown in equation (11), or an optimization problem that minimizes the objective function (H(x)) as shown in equation (12), and provides it to the solution unit 68.

number

number

[0178] As described above, the self-localization device 20 generates a matching problem, which is a combinatorial optimization problem, based on the N first feature points detected from the first image and the M second feature points detected from the second image. This matching problem involves associating each of the N first feature points with one of the M second feature points that is estimated to represent the same object. Then, based on the solution to the generated matching problem, the self-localization device 20 generates a list of corresponding points representing pairs of corresponding points, each containing one of the N first feature points and one of the M second feature points that was matched. Thus, the self-localization device 20 can accurately associate the first feature points in the first image with the second feature points in the second image using a simple process.

[0179] (First variation) Next, we will describe the mobile body control system 10 according to the first modified example.

[0180] Figure 8 shows the functional configuration of the matching unit 40 according to the first modified example. The matching unit 40 according to the first modified example further includes an outlier removal unit 92.

[0181] The outlier removal unit 92 obtains a list of corresponding points from the corresponding point list generation unit 70. The outlier removal unit 92 removes outlier pairs of corresponding points from the one or more pairs of corresponding points represented by the obtained list of corresponding points. The outlier removal unit 92 then provides the corresponding point list, from which the outlier pairs of corresponding points have been removed, to the estimation unit 42.

[0182] The outlier removal unit 92 identifies outlier pairs of corresponding points based on the scores of one or more pairs of corresponding points included in the corresponding point list, and removes the identified outlier pairs of corresponding points from the corresponding point list.

[0183] For example, the outlier removal unit 92 may identify correspondence point pairs whose scores are lower than a preset threshold as outlier correspondence point pairs. Alternatively, for example, the outlier removal unit 92 may select a predetermined number or a predetermined percentage of correspondence point pairs with preset scores, and identify the correspondence point pairs excluding the selected predetermined number or percentage as outlier correspondence point pairs.

[0184] The score may be a feature score, which is the average of the features of either the first or second feature point included in the corresponding point pair. Alternatively, the score may be a feature point score, which is the parameter used when detecting either the first or second feature point included in the corresponding point pair, or the average of the parameters. Furthermore, the score may be a value obtained by combining the feature score and the feature point score through a predetermined calculation.

[0185] Further, the outlier removal unit 92 may identify non-matching corresponding point pairs as outlier corresponding point pairs through cross-check processing. As cross-check processing, the outlier removal unit 92 checks whether the first feature point included in the target corresponding point pair is included in corresponding point pairs other than the target corresponding point pair, or whether the second feature point included in the target corresponding point pair is included in corresponding point pairs other than the target corresponding point pair. When the first feature point is included in corresponding point pairs other than the target corresponding point pair, or when the second feature point is included in corresponding point pairs other than the target corresponding point pair, the outlier removal unit 92 identifies the target corresponding point pair as an outlier corresponding point pair.

[0186] The movement control system 10 according to the first modification example can accurately execute the correspondence processing of the corresponding points based on the first feature points and the second feature points that are accurately selected by removing the corresponding point pairs that become outliers from the corresponding point list by such an outlier removal unit 92.

[0187] (Second Modification Example) Next, the movement control system 10 according to the second modification example will be described.

[0188] FIG. 9 is a diagram showing the functional configuration of the self-position estimation device 20 according to the second modification example.

[0189] The self-position estimation device 20 according to the second modification example further includes a selection unit 94.

[0190] In this modification example, it is assumed that L second feature points (L is an integer greater than or equal to M) are detected as one or more feature points in the second image.

[0191] The selection unit 94 acquires the second feature point list and the second feature amount list detected and calculated from the second image from the past information storage unit 38.

[0192] The second feature point list shows the L second feature points detected from the second image. The second feature quantity list shows the L second feature quantities calculated from the second image. The L second feature quantities correspond one-to-one with the L second feature points. Each of the L second feature quantities represents the feature quantity of the corresponding second feature point among the L second feature points.

[0193] The selection unit 94 selects a predetermined number of M feature points from L second feature points. For example, the selection unit 94 may select the M second feature points with the highest feature quantities from among the L second feature points. Alternatively, the selection unit 94 may select the M second feature points with the highest feature point scores, which are parameters used during feature point detection, from among the L second feature points. The selection unit 94 may also use other algorithms to select M second feature points from the L second feature points. Furthermore, the selection unit 94 may select M second feature points based on one or more specific first feature points from among N first feature points. For example, the selection unit 94 may select M second feature points that are close in position to a specific first feature point from among N first feature points. Alternatively, the selection unit 94 may select M second feature points whose feature quantities are similar to those of a specific first feature point.

[0194] The selection unit 94 generates a list of second feature points, each representing M second feature points for the M selected feature points. The selection unit 94 also generates a list of second feature quantities, each representing M second feature quantities for the M selected feature points. If L second feature points and L second feature quantities share a common identifier, the selection unit 94 may generate a list of M identifiers instead of a list of M selected feature points and a list of M feature quantities. The selection unit 94 provides the matching unit 40 with the list of second feature points, each representing M second feature points for the M selected feature points, and the list of second feature quantities, each representing M second feature quantities for the M selected feature points.

[0195] The matching unit 40 then generates a matching problem based on the N first feature points and N feature quantities detected from the first image, and the M second feature points and M second feature quantities selected, and generates a list of corresponding points based on the solution to the matching problem.

[0196] The solver device 14 can be implemented, for example, by a hardware Ising machine. While Ising machines can solve combinatorial optimization problems quickly, the number of spin variables they can handle is limited by the hardware. Therefore, if the solver device 14 is a hardware Ising machine, the number of decision variables included in the matching problem depends on the upper limit of the number of spin variables that the Ising machine can handle.

[0197] For example, if the number of feature points in the first image and the second image is 1000 each, the number of multiple decision variables included in the matching problem will be 1000 × 1000 = 1,000,000. In this case, the number of spin variables handled by the Ising machine will be 1,000,000. However, it is difficult for an Ising machine implemented in hardware to solve an Ising problem with 1,000,000 spin variables. Therefore, if the solver device 14 is implemented by a hardware Ising machine, the self-localization device 20 must limit the number of multiple decision variables included in the matching problem.

[0198] In this modified self-localization device 20, the selection unit 94 limits the number of second feature points to M, thereby limiting the number of multiple decision variables included in the matching problem. As a result, the self-localization device 20 can efficiently solve the matching problem using a solver device 14 implemented by an Ising machine, which can handle a small number of spin variables. The self-localization device 20 in the second modified version may also be applied to the first modified version.

[0199] The selection unit 94 may appropriately change the number of second feature points selected from the L second feature points. In other words, the selection unit 94 may appropriately change M. For example, if a larger feature quantity is preferable, the selection unit 94 may select second feature points that have a second feature quantity equal to or greater than a preset threshold. Alternatively, if a smaller feature quantity is preferable, the selection unit 94 may select second feature points that have a second feature quantity equal to or less than a preset threshold.

[0200] Furthermore, if it is preferable for the distance from a specific location to be small, the selection unit 94 may select M second feature points from the L second feature points that are close to the specific location. For example, the selection unit 94 may select the top M second feature points from the L second feature points that are close to the specific location, such that their distance from the specific location is below a threshold and / or their feature quantity is below a threshold. Alternatively, for example, the selection unit 94 may select two or more second feature points from the L second feature points that are close to the specific location and / or have a feature quantity below a threshold, and then select the top M second feature points with the highest feature quantity from the selected two or more second feature points.

[0201] Alternatively, instead of obtaining a second feature point list and a second feature quantity list, the selection unit 94 may obtain a first feature point list and a first feature quantity list detected and calculated from the first image. In this case, the first feature point list represents K first feature points detected from the first image (where K is an integer greater than or equal to N). The first feature quantity list represents K first features calculated from the first image. The K first features correspond one-to-one with the K first feature points. Each of the K first features represents the feature quantity of the corresponding first feature point among the K first feature points. In such a case, the selection unit 94 selects a predetermined number of N feature points from the K first feature points. For example, the selection unit 94 may select N first feature points based on one or more specific second feature points among M second feature points. For example, the selection unit 94 may select N first feature points that are close in position to a specific second feature point among the M second feature points. For example, the selection unit 94 may select N first feature points whose feature quantities approximate a particular second feature point.

[0202] In this case, the selection unit 94 generates a first feature point list showing N first feature points for the N selected feature points. The selection unit 94 also generates a first feature quantity list showing N first feature quantities for the N selected feature points. The selection unit 94 provides the matching unit 40 with the first feature point list showing N first feature points for the N selected feature points and the first feature quantity list showing N first feature quantities for the N selected feature points. Even with this configuration, the self-localization device 20 according to this modified example can limit the number of first feature points to N by the selection unit 94, thereby limiting the number of multiple decision variables included in the matching problem.

[0203] Furthermore, when obtaining a list of first feature points and a list of first feature quantities detected and calculated from the first image, the selection unit 94 may appropriately change the number of first feature points selected from the K first feature points. That is, the selection unit 94 may appropriately change N. For example, if a larger feature quantity is preferable, the selection unit 94 may select first feature points that have a first feature quantity greater than or equal to a preset threshold. Alternatively, if a smaller feature quantity is preferable, the selection unit 94 may select first feature points that have a first feature quantity less than or equal to a preset threshold.

[0204] Furthermore, if it is preferable for the distance from a specific location to be small, the selection unit 94 may select N first feature points from the K first feature points that are close to the specific location. For example, the selection unit 94 may select the top N first feature points from the K first feature points that are close to the specific location, such that their distance from the specific location is below a threshold and / or their feature quantity is below a threshold. Alternatively, for example, the selection unit 94 may select two or more first feature points from the K first feature points that are close to the specific location and / or have a feature quantity below a threshold, and then select the top N first feature points with the highest feature quantities from the two or more selected first feature points.

[0205] Furthermore, the selection unit 94 may obtain a first feature point list and a first feature quantity list detected and calculated from the first image, as well as a second feature point list and a second feature quantity list detected and calculated from the second image. In this case, the selection unit 94 simultaneously performs the following processes: selecting N first feature points from K first feature points to generate a first feature point list and a first feature quantity list, and selecting M first feature points from L second feature points to generate a second feature point list and a second feature quantity list.

[0206] (Third variation) Next, we will describe the mobile control system 10 according to the third modified example.

[0207] Figure 10 shows the functional configuration of the self-position estimation device 20 according to the third modified example. Figure 11 shows multiple partial images obtained by dividing the input image.

[0208] The self-position estimation device 20 according to the third modified example further comprises an image splitting unit 96 and a list synthesis unit 98.

[0209] The image splitting unit 96 acquires input images at predetermined time intervals, which have been processed by the image processing unit 32. The image splitting unit 96 divides each input image at the predetermined time interval into multiple sub-images corresponding to a predetermined number of sub-regions.

[0210] For example, the image splitting unit 96 generates four partial images by dividing the input image into two vertically and two horizontally, as shown in Figure 11. The image splitting unit 96 can divide the input image into any region, as long as the division is performed at predetermined positions.

[0211] The feature point detection unit 34 detects one or more feature points for each of the multiple partial images for each input image at a predetermined time interval. The feature point detection unit 34 then generates a feature point list for each of the multiple partial images for each input image at a predetermined time interval.

[0212] The feature calculation unit 36 ​​calculates one or more feature quantities corresponding to the one or more feature points detected for each of the multiple sub-images for each input image at a predetermined time interval. The feature calculation unit 36 ​​then generates a feature quantity list for each of the multiple sub-images for each input image at a predetermined time interval.

[0213] The image splitting unit 96 may be located after the feature calculation unit 36. In this case, the feature point detection unit 34 detects feature points from the input image before splitting. The feature calculation unit 36 ​​calculates feature quantities based on the input image before splitting. In this case, the image splitting unit 96 determines which of the multiple regions each of the one or more feature points detected by the feature point detection unit 34 belongs to, and splits the one or more feature points into each of the multiple regions. The image splitting unit 96 also splits the one or more feature quantities calculated by the feature calculation unit 36, corresponding to the one or more feature points. Finally, the image splitting unit 96 generates a feature point list and a feature quantity list for each of the multiple regions.

[0214] The past information storage unit 38 stores a list of feature points and a list of feature quantities for each of the multiple subimages in a past input image.

[0215] The matching unit 40 generates a list of corresponding points for each of the multiple partial images for each input image at a predetermined time interval.

[0216] In this modified example, the input image to be processed is referred to as the first input image. Furthermore, any input image prior to the first input image is referred to as the second input image. The second input image is, for example, the input image immediately preceding the first input image. Alternatively, the second input image may be, for example, a keyframe image prior to the first input image.

[0217] Furthermore, in this modified example, the first image processed by the matching unit 40 is one of the multiple partial images obtained by dividing the first input image. The second image is a second partial image of the same region as the first partial image, obtained by dividing the second input image.

[0218] In this modified example, the matching unit 40 obtains a first feature point list showing N first feature points detected from the first image (i.e., the first partial image among a plurality of partial images obtained by dividing the first input image) and a first feature quantity list showing N first feature quantities calculated from the first image.

[0219] Furthermore, the matching unit 40 obtains from the past information storage unit 38 a list of second feature points indicating M second feature points detected from the second image (i.e., the second sub-image of the same region as the first sub-image, among the multiple sub-images obtained by dividing the second input image) and a list of second feature quantities indicating M second feature quantities calculated from the second image.

[0220] The matching unit 40 then provides the list synthesis unit 98 with a list of corresponding points generated for each of the multiple partial images for each input image at a predetermined time interval.

[0221] The list synthesis unit 98 generates a combined correspondence point list for each input image at a predetermined time interval by combining the correspondence point lists generated for each of the multiple sub-images obtained by dividing the first input image. The list synthesis unit 98 then provides the generated combined correspondence point list to the estimation unit 42.

[0222] In this modified example, the estimation unit 42 acquires a list of composite corresponding points for each input image at a predetermined time interval. Then, at each predetermined time interval, the estimation unit 42 estimates its own position, which is the position of the camera 12 in three-dimensional space, based on the acquired list of composite corresponding points.

[0223] Figure 12 is a flowchart showing the processing flow of the self-position estimation device 20 according to the third modified example. The self-position estimation device 20 according to the third modified example performs processing according to the flow shown in Figure 12.

[0224] The self-position estimation device 20 acquires input images from the camera 12 at predetermined time intervals. Each time the self-position estimation device 20 acquires an input image, it uses the acquired input image as the first input image and executes the processes from S32 to S45 (loop processing between S31 and S46).

[0225] First, in S32, the self-position estimation device 20 performs image processing on the first input image.

[0226] Next, in S33, the self-position estimation device 20 divides the first input image into multiple sub-images corresponding to a predetermined number of sub-regions.

[0227] Next, the self-localization device 20 performs the processing from S35 to S41 for each of the multiple sub-images obtained by dividing the first input image (loop processing between S34 and S41). The self-localization device 20 processes the sub-images to be processed from S35 to S41 as the first image.

[0228] In S35, the self-localization device 20 detects N first feature points from the first image.

[0229] Next, in S36, the self-localization device 20 calculates N first features from the first image.

[0230] Next, in S37, the self-localization device 20 acquires M second feature points detected from the second image.

[0231] Next, in S38, the self-localization device 20 acquires M second features calculated from the second image.

[0232] Next, in S39, the self-localization device 20 generates a matching problem based on N first feature points and N first feature quantities, and M second feature points and M second feature quantities.

[0233] Next, in S40, the self-position estimation device 20 obtains the solution to the matching problem.

[0234] Next, in S41, the self-position estimation device 20 generates a list of corresponding points representing corresponding point pairs based on the solution to the matching problem.

[0235] Then, in S42, if the self-position estimation device 20 has completed the processing from S35 to S41 for all of the multiple partial images obtained by dividing the first input image, it proceeds to processing in S43.

[0236] In S43, the self-position estimation device 20 generates a composite correspondence point list by combining the correspondence point lists generated for each of the multiple partial images obtained by dividing the first input image.

[0237] Next, in S44, the self-position estimation device 20 estimates its own position, which is the position of the camera 12 that captured the first input image in three-dimensional space, based on the composite correspondence point list.

[0238] Next, in S45, the self-localization device 20 stores the N first feature points and N first feature quantities detected from each of the multiple divided images obtained by dividing the first input image as M second feature points and M second feature quantities detected from each of the multiple divided images obtained by dividing the second input image. As a result, the self-localization device 20 can obtain M second feature points and M second feature quantities in the processing of the next first input image.

[0239] Furthermore, if the second input image is a keyframe image specified prior to the first input image, the self-localization device 20 will not execute the process in S45 unless the first input image is a keyframe image.

[0240] The self-position estimation device 20 repeats the processing steps S32 to S45 for each input image until the movement of the moving object is completed (S46). Then, when the movement of the moving object is completed, the self-position estimation device 20 terminates this flow.

[0241] Figure 13 shows multiple partial images obtained by dividing an input image when some parts overlap.

[0242] The image splitting unit 96 divides the input image into multiple sub-images corresponding to a predetermined number of sub-regions. In this case, the first and second regions, which are adjacent to each other among the predetermined number of regions, may partially overlap, as shown in Figure 13.

[0243] In this case, the list synthesis unit 98 determines whether the synthesized correspondence point list contains two or more correspondence point pairs in which at least one of the first or second feature points is duplicated. Then, the list synthesis unit 98 deletes some of the two or more correspondence point pairs in which at least one of the first or second feature points is duplicated, so that the list does not contain two or more correspondence point pairs in which at least one of the first or second feature points is duplicated.

[0244] For example, the list synthesis unit 98 may delete all pairs of corresponding points in which at least one of the first or second feature points overlaps from the list of synthesized corresponding points. Alternatively, for example, the list synthesis unit 98 may retain the pair of corresponding points with the highest feature score, or the pair of corresponding points with the highest feature score, from among the pairs of corresponding points in which at least one of the first or second feature points overlaps, and delete the other pairs of corresponding points.

[0245] Alternatively, for example, the list synthesis unit 98 may cause the matching unit 40 to generate a new list of corresponding points, as follows.

[0246] When the matching unit 40 is instructed to generate a new list of corresponding points, the list synthesis unit 98 identifies a new subregion that includes two or more pairs of corresponding points in which at least one of the first or second feature points overlaps. The new subregion is a portion of the input image.

[0247] Next, the list synthesis unit 98 provides the matching unit 40 with information indicating the location of the new subregion. The matching unit 40 acquires a plurality of first feature points and a plurality of first feature quantities detected and calculated from the new subregion in the first input image. The matching unit 40 also acquires a plurality of second feature points and a plurality of second feature quantities detected and calculated from the new subregion in the second input image.

[0248] Next, the matching unit 40 regenerates the matching problem based on the acquired multiple first feature points and multiple first feature quantities, and the acquired multiple second feature points and multiple second feature quantities. Subsequently, the matching unit 40 generates a new list of corresponding points, which represents one or more new corresponding point pairs, where any first feature point from the multiple first feature points is associated with any second feature point from the multiple second feature points, based on the solution to the regenerated matching problem.

[0249] Then, the list synthesis unit 98 updates the synthesized correspondence point list by replacing two or more pairs of correspondence points in the synthesized correspondence point list where at least one of the first or second feature points is duplicated with one or more new pairs of correspondence points in the new correspondence point list.

[0250] Two feature points located far apart in an image are unlikely to significantly affect the simultaneous existence of each other as a pair. Therefore, performing matching operations on two feature points located far apart in an image separately is unlikely to affect the results.

[0251] However, two neighboring feature points located on either side of the division boundary may influence the co-occurrence score. However, if two neighboring feature points located on either side of the division boundary are matched separately, their relationship will not be reflected in the co-occurrence score. Therefore, the self-localization device 20 according to this modified example can reflect two neighboring feature points located on either side of the division boundary in the co-occurrence score by generating multiple partial images that partially overlap.

[0252] However, by generating multiple partial images with some overlap, the composite correspondence pair may contain two or more correspondence point pairs in which at least one of the first or second feature points overlaps. However, since the list synthesis unit 98 performs the deletion process described above or generates a new correspondence point list, it is possible to generate a composite correspondence point list that does not contain two or more correspondence point pairs in which at least one of the first or second feature points overlaps. Note that the list synthesis unit 98 may generate a composite correspondence point list that does not contain two or more correspondence point pairs in which at least one of the first or second feature points overlaps, not limited to the deletion process described above or the generation of a new correspondence point list.

[0253] As described above, the self-localization device 20 according to this modified example divides the input image into multiple sub-images and generates a matching problem for each of the sub-images. This allows the self-localization device 20 according to this modified example to reduce the number of decision variables included in the matching problem. Therefore, the self-localization device 20 can efficiently solve the matching problem using a solver device 14 realized by an Ising machine, which can handle a small number of spin variables.

[0254] Furthermore, the self-position estimation device 20 according to the third modified example may also be applied to the second modified example. This allows the self-position estimation device 20 according to this modified example to further reduce the number of decision variables included in the matching problem.

[0255] (Fourth variation) Next, we will describe the mobile control system 10 according to the fourth modified example.

[0256] Figure 14 is a diagram showing the functional configuration of the matching unit 40 in the fourth modified example. Figure 15 is a diagram illustrating the contents of the candidate point selection process and the decoding process in the fourth modified example.

[0257] The matching unit 40 according to the fourth modified example further comprises a candidate point selection unit 100 and a decoding unit 102.

[0258] The candidate point selection unit 100 obtains the first feature point list and the first feature quantity list from the first acquisition unit 62. The candidate point selection unit 100 also obtains the second feature point list and the second feature quantity list from the second acquisition unit 64.

[0259] The candidate point selection unit 100 selects G candidate points (G being an integer between 1 and M) from the M second feature points represented in the second feature point list for each of the N first feature points represented in the first feature point list. In this case, the candidate point selection unit 100 may select G candidate points of different combinations for each of the N first feature points. For example, the candidate point selection unit 100 may select G second feature points for each of the N first feature points that have the smallest difference in feature quantities from the first feature point. For example, the candidate point selection unit 100 may select G second feature points for each of the N first feature points that are closest to the first feature point. For example, the candidate point selection unit 100 may select G first feature points from the M first feature points that have a distance from a specific location below a threshold and a feature quantity below a threshold, and that have the highest feature quantities. Alternatively, for example, the candidate point selection unit 100 may select two or more first feature points from among M first feature points whose distance from a specific location is below a threshold and whose feature quantity is below a threshold, and then select the top G first feature points that are closest to the specific location from among the two or more selected first feature points.

[0260] Furthermore, the candidate point selection unit 100 generates correspondence information. The correspondence information represents G candidate points selected for each of the N first feature points. For example, Figure 15 shows correspondence information represented by an N row and G column matrix. For example, in the example in Figure 15, the candidate point selection unit 100 selects one of the first feature points, p A Regarding point 1, there are G candidate points, p B 1, p B 4,…,p B 21It has selected p A Regarding point 2, there are G candidate points, p B 3,p B 5,…,p B 9 is selected. Also, the candidate point selection unit 100 selects one of the first feature points, p A Regarding point 3, there are G candidate points, p B 2,p B 6,…,p B 27 It has selected p A N Regarding G candidate points, p B 7, p B 25 ,…,p B 33 You have selected this option.

[0261] The candidate point selection unit 100 provides the first feature point list, the second feature quantity list, and correspondence relationship information to the problem generation unit 66. The candidate point selection unit 100 also provides the correspondence relationship information to the decoding unit 102.

[0262] The problem generation unit 66 generates a matching problem. In this modified example, the matching problem is a combinatorial optimization problem in which each of the N first feature points is associated with one of the G candidate points selected for the corresponding first feature point that is presumed to represent the same object. In this modified example, the objective function of the matching problem includes (N × G) decision variables. The objective function is minimized or maximized when all (N × G) decision variables are set to the correct values.

[0263] For example, when (N × G) decision variables are arranged in N rows and G columns, the value of the decision variable in row i × column j represents whether the i-th first feature point out of N first feature points and the j-th candidate point out of G candidate points selected for the i-th first feature point are a corresponding point pair. If the i-th first feature point and the j-th candidate point selected for the i-th first feature point are the same object, the correct value of the decision variable in row i × column j is a value that indicates they are a corresponding point pair (e.g., -1). If the i-th first feature point and the j-th candidate point selected for the i-th first feature point are not the same object, the correct value of the decision variable in row i × column j is a value that indicates they are not a corresponding point pair (e.g., +1).

[0264] For example, in the example in Figure 15, the decision variable x is placed in the first row and first column. 11 is, p A 1 and p B 1 indicates whether or not they are a corresponding point pair. Also, the decision variable x is placed in the first row and second column. 12 is, p A 1 and p B This indicates whether 4 and are corresponding point pairs. Also, the decision variable x is located in the first row and column G. 1G is, p A 1 and p B 21 This indicates whether and are a corresponding point pair. Also, the decision variable x is placed in the second row and first column. 21 is, p A 2 and p B This indicates whether 3 and are corresponding point pairs. Also, the decision variable x is located in the second row and second column. 22 is, p A 2 and p B This indicates whether 5 and are corresponding point pairs. Then, the decision variable x is placed in the Nth row and G column. NG is, p A N and p B 33 This indicates whether or not two points are a corresponding point pair.

[0265] Then, the problem generation unit 66 provides the generated matching problem to the solution unit 68.

[0266] The solving unit 68 provides the acquired matching problem to the solver device 14 and obtains the solution to the matching problem from the solver device 14. If the matching problem is easily solvable, the matching unit 40 may solve the matching problem itself without providing it to the solver device 14. The solving unit 68 provides the solution to the matching problem to the decoding unit 102.

[0267] The decoding unit 102 converts the solution to the matching problem, which minimizes or maximizes an objective function containing (N × G) decision variables, based on the correspondence information, into a solution that shows which of the M second feature points corresponds to each of the N first feature points. The decoding unit 102 then provides the generated solution to the matching problem to the correspondence point list generation unit 70.

[0268] The self-localization device 20 according to this modified example can associate each of the N first feature points with any of the M second feature points based on a solution to a matching problem with a reduced number of decision variables. As a result, the self-localization device 20 can efficiently find the solution to the matching problem using a solver device 14 realized by an Ising machine that can handle a small number of spin variables. The self-localization device 20 according to the fourth modified example may also be applied to the first and second modified examples.

[0269] In the fourth modified example, the candidate point selection unit 100 may select Z candidate points (where Z is an integer between 1 and N) from the N first feature points represented in the first feature point list for each of the M second feature points represented in the second feature point list. In this case, the candidate point selection unit 100 may select Z different combinations of candidate points for each of the M second feature points.

[0270] In this case, the correspondence information represents the Z candidate points selected for each of the M second feature points. The matching problem is a combinatorial optimization problem in which each of the M second feature points is associated with one of the Z candidate points selected for the corresponding second feature point that is presumed to represent the same object. The objective function of the matching problem contains (M × Z) decision variables. The objective function is minimized or maximized when all (M × Z) decision variables are set to their correct values.

[0271] For example, when (M × Z) decision variables are arranged in M ​​rows and Z columns, the value of the decision variable in row i × column j represents whether the i-th second feature point out of M second feature points and the j-th candidate point out of Z candidate points selected for the i-th second feature point are a corresponding point pair. If the i-th second feature point and the j-th candidate point selected for the i-th second feature point are the same object, the correct value of the decision variable in row i × column j is a value that indicates they are a corresponding point pair (e.g., -1). If the i-th second feature point and the j-th candidate point selected for the i-th second feature point are not the same object, the correct value of the decision variable in row i × column j is a value that indicates they are not a corresponding point pair (e.g., +1).

[0272] Even with this configuration, the self-localization device 20 can efficiently find a solution to the matching problem using a solver device 14 realized by an Ising machine, which can handle a small number of spin variables.

[0273] (Solver device 14) Next, we will describe the solver device 14 that solves combinatorial optimization problems.

[0274] A combinatorial optimization problem is a problem that seeks to find a combination of values ​​for multiple decision variables that minimizes or maximizes an objective function. The objective function is a function of order one or higher that includes multiple decision variables representing the state of the system to be optimized as arguments. For example, the objective function can be expressed as a polynomial summing multiple terms. Each of the multiple terms is a function that multiplies one or more (up to a predetermined number) of the multiple decision variables by a coefficient. The coefficient in each of the multiple terms is a real number and is also called a weight value. In this case, each of the multiple terms that make up the objective function is expressed by multiplying one or more of the multiple decision variables by one of the multiple weight values. Note that the objective function may include terms expressed by functions such as exponential or logarithmic functions.

[0275] Each of the multiple decision variables may be discrete or continuous. Discrete values ​​may represent 0 or 1, or Ising spins representing -1 or +1. Furthermore, some of the decision variables included in the objective function may be discrete, while others are continuous.

[0276] The state of a system, as represented by multiple decision variables, is called a solution. The set of all possible solutions for the state of the system is called the solution space. In a problem of minimizing an objective function, the solution that minimizes the value of the objective function is called an exact solution. In a problem of minimizing an objective function, a solution that brings the value of the objective function close to the minimum is called a good solution.

[0277] Furthermore, combinatorial optimization problems may include constraints. Constraints represent conditions that the solution must satisfy. Constraints are one or more constraint equations expressed using some of the decision variables included in the objective function. Constraint equations may be equalities or inequalities.

[0278] The QUBO problem is an unconstrained quadratic optimization problem with binary decision variables. In the QUBO problem, each of the multiple terms in the objective function can be expressed as a linear or quadratic expression of the decision variables.

[0279] The objective function of the QUBO problem is given by H in equation (13). total_QUBO It is represented by [this].

[0280]

number

[0281] In equation (13), N is an integer greater than or equal to 2 and represents the number of decision variables. In equation (13), i and j represent any integers greater than or equal to 1 and less than or equal to N. i b is either 0 or 1 and represents the i-th decision variable out of N decision variables. j Q is either 0 or 1 and represents the j-th decision variable out of N decision variables. ij This represents the coefficient of the i-th row and j-th column in the N×N coefficient matrix (Q). Note that Q ij =Q ji That is. Q ii This is a coefficient included in the i-th row and i-th column of the coefficient matrix (Q), and is multiplied by the linear term of the i-th decision variable. ii This is called the bias coefficient.

[0282] Figure 16 shows a model of the Ising problem. The Ising problem is a problem of searching for the ground state of the Ising model. The Ising problem is one of the QUBO problems. The decision variables of the Ising problem represent discrete variables of -1 or +1.

[0283] The objective function of the Ising problem is given by H in equation (14). total_Ising It is represented by [this].

[0284]

number

[0285] In equation (14), N is an integer greater than or equal to 2 and represents the number of decision variables. In equation (14), i and j represent any integers greater than or equal to 1 and less than or equal to N. is is either -1 or +1 and represents the i-th decision variable out of N decision variables. j J is either -1 or +1 and represents the j-th decision variable out of N decision variables. ij This represents the coefficient of the i-th row and j-th column in the N×N coefficient matrix (J). ij =J ji h i h is the coefficient multiplied by the linear term of the i-th decision variable. i This is called the bias coefficient.

[0286] The Ising problem corresponds to the problem of searching for the ground state of the Ising model, one of the magnetic material models in statistical mechanics. Therefore, s i This is sometimes called the spin variable. Also, N, which is the number of decision variables, is sometimes called the spin number. total_Ising This is sometimes called Ising energy. total_Ising N s that have the minimum value i The vector represented by this is sometimes called the ground state (ground spin configuration).

[0287] The objective functions of the QUBO problem and the Ising problem differ only in the value of the constant. Therefore, the QUBO problem and the Ising problem are identical as combinatorial optimization problems. In other words, the QUBO problem and the Ising problem can be converted to each other. For example, the Ising problem and the QUBO problem can be converted to each other by equations (15-1), (15-2), (15-3), and (15-4).

[0288]

number

[0289] The QUBO problem and the Ising problem are known to be NP-complete. That is, many NP-hard problems can be transformed into either a QUBO problem or an Ising problem in polynomial time. Therefore, many practical combinatorial optimization problems can be transformed into either a QUBO problem or an Ising problem.

[0290] An Ising machine is a device that solves the Ising problem. Many Ising machines solve the Ising problem using heuristic methods. Various Ising machines based on principles from fields such as electronics, optics, quantum mechanics, and statistical mechanics have been proposed. Many Ising machines can produce an exact solution or a good solution in a short amount of time.

[0291] Simulated bifurcation algorithms are algorithms for solving combinatorial optimization problems. Simulated bifurcation algorithms are heuristic solution algorithms.

[0292] Simulated bifurcation algorithms are shown, for example, in Non-Patent Documents 4-5 and Patent Documents 2-6. Because simulated bifurcation algorithms were discovered inspired by quantum mechanical optimization methods based on the quantum adiabatic theorem, they are also called quantum-inspired algorithms. Simulated bifurcation algorithms can solve combinatorial optimization problems where the objective function is a quadratic function of multiple decision variables. Simulated bifurcation algorithms can also solve combinatorial optimization problems where the objective function is a cubic or higher function of multiple decision variables, i.e., HUBO (Higher Order Binary Optimization) problems. For example, a simulated bifurcation algorithm for solving the HUBO problem is shown in Patent Document 5. Furthermore, simulated bifurcation algorithms can also solve combinatorial optimization problems where some or all of the multiple decision variables include continuous variables. A simulated bifurcation algorithm for solving combinatorial optimization problems where some or all of the multiple decision variables include continuous variables is shown in Patent Document 6.

[0293] A simulated branching machine is a computing device that performs processing according to a simulated branching algorithm. A simulated branching machine that solves the QUBO problem or the Ising problem is an example of an Ising machine. In this embodiment, the simulated branching machine solves the Ising problem.

[0294] Figure 17 shows the internal variables used by the simulated branching algorithm.

[0295] The simulated branching algorithm has N decision variables (s1~s N When solving a combinatorial optimization problem in which the objective function is expressed using ), the internal variables are N position variables (x1~x N ) and N momentum variables (y1~y N The simulated branching algorithm uses 2 × N internal variables.

[0296] N position variables (x i ) consists of N decision variables (s i ) corresponds one-to-one with the i-th position variable (x) among the N position variables. i ) is the i-th decision variable (s) among the N decision variables. i ) corresponds to N momentum variables (y i ) consists of N decision variables (s i It corresponds one-to-one with the i-th momentum variable (y) among the N momentum variables. i ) is the i-th decision variable (s) among the N decision variables. i ) corresponds to.

[0297] Figure 18 is a flowchart showing the processing flow of a simulated branching machine. The simulated branching machine executes processing according to the simulated branching algorithm in the flow shown in Figure 18.

[0298] First, in S111, the simulated branching machine obtains the Ising problem. Specifically, the simulated branching machine obtains J, which is a matrix containing N × N coefficients, and h, which contains N bias coefficients.

[0299] Next, in S112, the simulated branching machine has 2 × N internal variables, i.e., N position variables (x1~x N ) and N momentum variables (y1~y NInitialize the N position variables (x1~x N Initial values ​​of ) and N momentum variables (y1~y N The initial values ​​of both or either of the following can be obtained from an external source. In addition, the simulated branching machine has N position variables (x1~x N Initial values ​​of ) and N momentum variables (y1~y N The initial value of ) may be generated by a random number generation circuit or set to a predetermined value. Note that since the simulated branching machine is a heuristic, even for the same problem, the N position variables (x1~x N Initial values ​​of ) and N momentum variables (y1~y N If at least one of the initial values ​​of ) is different, it may output different good solutions.

[0300] Next, the simulated branching machine repeats the process from S114 to S116 a predetermined number of times (loop processing between S113 and S117). The process from S114 to S116 involves N position variables (x1~x N A matrix operation that performs matrix multiplication of ) and a matrix containing weight values ​​in N rows × N columns, and N position variables (x1~x N ) and N momentum variables (y1~y N This is a time evolution process for evolving ) over time.

[0301] In S114, the simulated bifurcation machine has N momentum variables (y1~y N The simulated branching machine performs a y-update process that updates each of the i-th momentum variable (y) in the y-update process. i In the update process of ), N position variables (x1~x N ) and the i-th position variable (x) of the N×N matrix (J). i ) and other (N-1) position variables (x 1~i-1, x i+1~N N coefficients (J) representing the interaction between ) i,j ) and the i-th bias coefficient (h i ) and the i-th momentum variable (yi ) Update.

[0302] Next, in S115, the simulated branching machine has N position variables (x1~x N The simulated branch machine executes an x ​​update process that updates each of the i-th position variable (x) in the x update process. i In the update process of ), the i-th momentum variable (y i ) determines the i-th position variable (x i ) Update.

[0303] The simulated branching machine may execute the processes of S114 and S115 in a reversed order.

[0304] Next, in S116, the simulated branching machine has N position variables (x1~x N The simulated bifurcation machine performs wall processing on position variables whose absolute value exceeds 1. Furthermore, the simulated bifurcation machine also performs wall processing on momentum variables corresponding to position variables whose absolute value exceeds 1. For example, in wall processing, the simulated bifurcation machine changes the absolute value of position variables whose absolute value exceeds 1 to a value of 1 or less, while keeping the sign the same. Also, for example, in wall processing, the simulated bifurcation machine changes the value of momentum variables corresponding to position variables whose absolute value exceeds 1 to 0.

[0305] Simulated bifurcation algorithms have variations in the x-update, y-update, and wall-update operations. For example, variations of the simulated bifurcation algorithm include the adiabatic simulated bifurcation (aSB) algorithm, the ballistic simulated bifurcation (bSB) algorithm, and the discrete simulated bifurcation (dSB) algorithm.

[0306] When executing processing according to the adiabatic simulated branching algorithm, the simulated branching machine performs the operation shown in equation (16-1) in the y update process (S114) and the operation shown in equation (16-2) in the x update process (S115). Note that when executing processing according to the adiabatic simulated branching algorithm, the simulated branching machine does not perform the wall processing (S115).

[0307]

number

[0308] When performing processing according to the ballistic simulated branching algorithm, the simulated branching machine performs the operation shown in equation (17-1) in the y update process (S114), the operation shown in equation (17-2) in the x update process (S115), and the operation shown in equation (17-3) in the wall processing (S116).

[0309]

number

[0310] When executing a process according to a discrete simulated branching algorithm, the simulated branching machine performs the operation shown in equation (18-1) in the y update process (S114), the operation shown in equation (18-2) in the x update process (S115), and the operation shown in equation (18-3) in the wall process (S116).

[0311]

number

[0312] Note that in equations (16-1), (16-2), (17-1), (17-2), (17-3), (18-1), (18-2), and (18-3), t k and t k+1 t represents time. k+1 is, t kThis is the time obtained by adding a unit of time (Δt) to that value.

[0313] x i (t k ) is time (t k The i-th position variable (x) in ) i The value of x is shown. i (t k+1 ) is time (t k+1 The i-th position variable (x) in ) i The value of ) is shown. i (t k ) is time (t k The i-th momentum variable (y i The value of ) is shown. i (t k+1 ) is time (t k+1 The i-th momentum variable (y i This shows the value of ).

[0314] K, a0, η, and c0 are predetermined constants. k a(t) is a function that changes with time. k ) is, for example, a(t1)=0, a positive real number that increases as time increases, and is a function that becomes a0 at the end time (T) (a(T)=a0). Also, sgn(x i (t k )) is time (t k The i-th position variable (x) in ) i This is a function that outputs the sign of x i (t k If ) is 0 or greater, add +1, x i (t k If the value is less than 0, it becomes -1.

[0315] When the simulated branching machine has executed the processes S114 to S116 a predetermined number of times, that is, when it has performed calculations until time t becomes the final time T, it exits the loop processing between S113 and S117 and proceeds to S118.

[0316] In S118, the simulated branching machine has N position variables (x1~x) at the final time.N ), or N position variables (x1~x) at the final time. N N decision variables (S11~s) calculated based on ) N The simulated branching machine outputs N decision variables (S11~s). N The i-th decision variable (s) among ) i ) to sgn(x i It is calculated based on the following.

[0317] When the simulated branching machine finishes processing S118, it terminates processing according to the simulated branching algorithm.

[0318] The number of iterations of the time evolution process (the loop between S113 and S117) is predetermined according to the application. The computational complexity required for one iteration of the time evolution process (one iteration from S114 to S116) does not vary. Therefore, the simulated branching machine can minimize the variation in solution time. Consequently, even when applied to real-time systems with time constraints that require processing to be completed by a predetermined time, the simulated branching machine can reliably output a solution within that time.

[0319] Furthermore, as shown in Patent Document 2, for example, a simulated branch machine can be configured using a dedicated parallel processing circuit that includes a large number of arithmetic units. This allows the simulated branch machine to extremely reduce the computation time for a single operation in the time evolution process. In addition, unlike software processing, a simulated branch machine implemented in a dedicated hardware circuit does not experience any interrupt processing, so the solution time is strictly fixed. For example, a simulated branch machine implemented in a dedicated hardware circuit can fix the time until a solution is obtained in units of clock cycles. Therefore, when a simulated branch machine implemented in a dedicated hardware circuit is applied to a real-time system, it can output a solution while more reliably adhering to time constraints.

[0320] (Hardware configuration) Figure 19 shows an example of a computer hardware configuration. The self-localization device 20 is implemented by a computer with a hardware configuration such as that shown in Figure 19. The self-localization device 20 comprises a CPU 401, RAM 402, ROM 403, storage device 404, and communication interface device 405. These components are connected by a bus.

[0321] The CPU 401 is one or more processors that perform arithmetic and control processing according to a program. The CPU 401 uses a predetermined area of ​​the RAM 402 as a working area and performs various processes in cooperation with programs stored in the ROM 403 and storage device 404, etc.

[0322] RAM402 is a type of memory such as SDRAM (Synchronous Dynamic Random Access Memory). RAM402 functions as a workspace for CPU401. ROM403 is a memory that stores programs and various information in a non-rewritable format.

[0323] The storage device 404 is a device that writes and reads data to and from a storage medium made of semiconductors such as flash memory, or a storage medium that can record magnetically or optically. The storage device 404 writes and reads data to and from the storage medium in response to control from the CPU 401. The communication interface device 405 communicates with external devices via a network in response to control from the CPU 401.

[0324] The program executed on the computer causes the computer to function as a self-localization device 20. This program is loaded onto RAM 402 and executed by the CPU 401 (processor).

[0325] Furthermore, programs executed on a computer are provided as files in a format that can be installed on a computer or in an executable format, recorded on computer-readable recording media such as CD-ROMs, flexible disks, CD-Rs, and DVDs (Digital Versatile Disks).

[0326] Furthermore, this program may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Alternatively, this program may be configured to be provided or distributed via a network such as the Internet. Furthermore, the program executed by the self-position estimation device 20 may be pre-installed in a ROM 403 or the like and provided in that configuration.

[0327] The program for causing the computer to function as a self-localization device 20 includes, for example, an image processing module, a feature point detection module, a feature quantity calculation module, a matching module, and an estimation module. When this program is executed by the CPU 401, each module is loaded into the RAM 402, causing the CPU 401 to function as a processing unit including an image processing unit 32, a feature point detection unit 34, a feature quantity calculation unit 36, a matching unit 40, and an estimation unit 42. If the CPU 401 consists of multiple processors, these units may be divided among the multiple processors. Note that some or all of these components may be configured by hardware. Furthermore, this program causes the RAM 402 or storage device 404 to function as a past information storage unit 38.

[0328] Furthermore, the processing unit of the self-position estimation device 20 may be implemented by one or more reconfigurable semiconductor devices, such as an FPGA (Field-Programmable Gate Array). Alternatively, the self-position estimation device 20 may be implemented by one or more CPUs, microprocessors, GPUs, ASICs (Application Specific Integrated Circuits), or electronic circuits including these.

[0329] Furthermore, if the processing unit of the self-position estimation device 20 is implemented using a reconfigurable semiconductor device such as an FPGA, the circuit information (configuration data) written to the reconfigurable semiconductor device in order to operate it as the processing unit of the self-position estimation device 20 may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Alternatively, the circuit information (configuration data) written to the reconfigurable semiconductor device in order to operate it as the processing unit of the self-position estimation device 20 may be recorded on a computer-readable recording medium and provided.

[0330] Furthermore, if the processing unit of the self-position estimation device 20 is implemented by a semiconductor device such as an ASIC, the circuit information representing the configuration of the circuit described in a hardware description language used in the design and manufacture of the processing unit of the self-position estimation device 20 may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Alternatively, the circuit information representing the configuration of the circuit described in a hardware description language used in the design and manufacture of the self-position estimation device 20 may be recorded on a computer-readable recording medium and provided.

[0331] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents.

[0332] <Note> Furthermore, this technology can also be configured as follows.

[0333] (Note 1) Based on N first feature points (where N is an integer greater than or equal to 1) detected from the first image and M second feature points (where M is an integer greater than or equal to 1) detected from a second image different from the first image, a matching problem is generated, which is a combinatorial optimization problem in which each of the N first feature points is associated with one of the M second feature points that is presumed to represent the same object. Based on the solution to the matching problem, a list of corresponding points is generated that represents a pair of corresponding points, where one of the N first feature points is a corresponding second feature point among the M second feature points. An information processing device equipped with a processing unit.

[0334] (Note 2) The aforementioned processing unit, The matching problem is given to a solver device that solves the combinatorial optimization problem, and the solver device is made to solve the matching problem. The solution to the matching problem is obtained from the solver. The information processing device described in Appendix 1.

[0335] (Note 3) The aforementioned matching problem is a combinatorial optimization problem that aims to maximize or minimize an objective function. The aforementioned objective function includes (N × M) decision variables and is minimized or maximized when all (N × M) decision variables are considered to be the correct values. When the (N × M) decision variables are arranged in N rows and M columns, the value of the decision variable in row i × column j among the (N × M) decision variables (where i is an integer between 1 and N, and j is an integer between 1 and N) represents whether the i-th first feature point among the N first feature points and the j-th second feature point among the M second feature points are the corresponding point pair. The correct value in the i-row × j-column decision variable is a value that indicates that the i-th first feature point and the j-th second feature point are a corresponding point pair if they are the same object, and a value that indicates that the i-th first feature point and the j-th second feature point are not a corresponding point pair if they are not the same object. The information processing device described in Appendix 2.

[0336] (Note 4) The objective function includes a term that multiplies the i-row × j-column decision variable, the i'-row × j'-column decision variable among the (N × M) decision variables (where i' is an integer between 1 and N, and j' is an integer between 1 and N), and the co-occurrence score. The co-occurrence score, included in the term obtained by multiplying the i-row × j-column decision variable by the i'-row × j'-column decision variable, is a value that represents the validity of the simultaneous existence of the corresponding point pair between the i-th first feature point and the j-th second feature point, and the corresponding point pair between the i'-th first feature point out of the N first feature points and the j'-th second feature point out of the M second feature points. The information processing device described in Appendix 3.

[0337] (Note 5) The co-occurrence score included in the term obtained by multiplying the i-row × j-column decision variable and the i'-row × j'-column decision variable is, This value corresponds to the degree of approximation between the first positional relationship between the image position of the i-th first feature point and the image position of the j-th second feature point, and the second positional relationship between the image position of the i'-th first feature point and the image position of the j'-th second feature point. The information processing device described in Appendix 4.

[0338] (Note 6) The co-occurrence score included in the term obtained by multiplying the i-row × j-column decision variable and the i'-row × j'-column decision variable is, The first positional relationship between the image position of the i-th first feature point and the image position of the j-th second feature point, and the first degree of approximation between the second positional relationship between the image position of the i'-th first feature point and the image position of the j'-th second feature point, This value is a composite of the estimated image position of the j-th second feature point at time in the first image, the third positional relationship between the j-th second feature point and the image position of the j-th second feature point, and the second approximation of the fourth positional relationship between the i'-th first feature point and the j'-th second feature point. The information processing device described in Appendix 4.

[0339] (Note 7) If the first condition is met such that the position of the i-th first feature point and the position of the i'-th first feature point are less than or equal to a predetermined distance, The co-occurrence score included in the term obtained by multiplying the i-row × j-column decision variable and the i'-row × j'-column decision variable is, This value corresponds to the difference in movement between the first movement of the object represented by the corresponding point pair between the i-th first feature point and the j-th second feature point, and the second movement of the object represented by the corresponding point pair between the i'-th first feature point and the j'-th second feature point. An information processing device as described in any one of the appendices 4 to 6.

[0340] (Note 8) If the first and second images satisfy the second condition that they were captured by a camera shifted parallel to the optical axis, The co-occurrence score included in the term obtained by multiplying the i-row × j-column decision variable and the i'-row × j'-column decision variable is, The value is determined by the correlation between the size and displacement from the vanishing point on the image of the first object, represented by the corresponding point pair of the i-th first feature point and the j-th second feature point, and the size and displacement from the vanishing point on the image of the second object, represented by the corresponding point pair of the i'-th first feature point and the j'-th second feature point. An information processing device as described in any one of the appendices 4 to 7.

[0341] (Note 9) If the first and second images satisfy the third condition that they were captured by a camera whose optical axis has been rotated, The co-occurrence score included in the term obtained by multiplying the i-row × j-column decision variable by the i'-row × j'-column decision variable is a value corresponding to the difference in movement between the direction and amount of movement of the object represented by the corresponding point pair between the i-th first feature point and the j-th second feature point, and the direction and amount of movement of the object represented by the corresponding point pair between the i'-th first feature point and the j'-th second feature point. An information processing device as described in any one of the appendices 4 to 7.

[0342] (Note 10) The aforementioned objective function includes a term that multiplies the i-row × j-column decision variable by the feature score, The feature score multiplied by the i-row × j-column decision variable is: This value corresponds to the similarity between the feature quantity of the i-th first feature point and the feature quantity of the j-th second feature point. An information processing device as described in any one of the appendices 3 through 9.

[0343] (Note 11) The aforementioned objective function includes a term that multiplies the i-row × j-column decision variable by the position score, The position score multiplied by the i-row × j-column decision variable is: This value corresponds to the estimated distance between the estimated position of the j-th second feature point at time in the first image and the position of the i-th first feature point. An information processing device as described in any one of the appendices 3 to 10.

[0344] (Note 12) The processing unit selects G candidate points (where G is an integer between 1 and M) from among the M second feature points for each of the N first feature points. The aforementioned matching problem is a combinatorial optimization problem that aims to maximize or minimize an objective function. The aforementioned objective function includes (N × G) decision variables and is minimized or maximized when all (N × G) decision variables are considered to be the correct values. When the (N × G) decision variables are arranged in N rows and G columns, the value of the decision variable in row i × column j among the (N × G) decision variables (where i is an integer between 1 and N, and j is an integer between 1 and G) represents whether the i-th first feature point among the N first feature points and the j-th candidate point among the G candidate points selected for the i-th first feature point are the corresponding point pair. The correct value in the i-row × j-column decision variable is a value that indicates that the i-th first feature point and the j-th candidate point selected for the i-th first feature point are a corresponding point pair if they are the same object, and a value that indicates that the i-th first feature point and the j-th candidate point selected for the i-th first feature point are not a corresponding point pair if they are not the same object. An information processing device as described in any one of the appendices 1 through 11.

[0345] (Note 13) The first image is the first partial image among a plurality of partial images obtained by dividing the first input image into a plurality of predetermined regions, The second image is a second sub-image of the same region as the first sub-image, among the multiple sub-images obtained by dividing a second input image that is different from the first input image. An information processing device as described in any one of the appendices 1 through 12.

[0346] (Note 14) Of the predetermined plurality of regions, the first region and the second region, which are adjacent to each other, partially overlap. The aforementioned processing unit, A combined correspondence point list is generated by combining the correspondence point lists, each of which is generated as the first image, obtained by dividing the first input image into multiple sub-images. To ensure that the composite correspondence point list does not contain two or more correspondence point pairs in which at least one of the first or second feature points overlaps, some of the two or more correspondence point pairs in which at least one of the first or second feature points overlaps are deleted. The information processing device described in Appendix 13.

[0347] (Note 15) Of the predetermined plurality of regions, the first region and the second region, which are adjacent to each other, partially overlap. The aforementioned processing unit, A combined correspondence point list is generated by combining the correspondence point lists, each of which is generated as the first image, obtained by dividing the first input image into multiple sub-images. If the composite correspondence point list includes two or more pairs of correspondence points in which at least one of the first or second feature points overlaps, a new subregion containing the two or more pairs of correspondence points is identified. Based on the plurality of first feature points detected from the new subregion in the first input image and the plurality of second feature points detected from the new subregion in the second input image, the matching problem is regenerated. Based on the solution to the matching problem that was generated again, one or more new pairs of corresponding points are generated by associating one of the first feature points of the plurality of first feature points with one of the second feature points of the plurality of second feature points. The composite correspondence point list is updated by replacing the two or more pairs of corresponding points in which at least one of the first or second feature points overlaps with the one or more new pairs of corresponding points. The information processing device described in Appendix 13.

[0348] (Note 16) The aforementioned processing unit, From the first image captured by the camera, N first feature points are extracted, and from the second image captured by the camera at a different time than the first image, M second feature points are extracted. Based on the aforementioned list of corresponding points, the position of the camera in three-dimensional space is estimated. An information processing device as described in any one of the appendices 1 through 14.

[0349] (Note 17) An information processing method performed by an information processing device, Based on N first feature points (where N is an integer greater than or equal to 1) detected from the first image and M second feature points (where M is an integer greater than or equal to 1) detected from a second image different from the first image, a matching problem is generated, which is a combinatorial optimization problem in which each of the N first feature points is associated with one of the M second feature points that is presumed to represent the same object. Based on the solution to the matching problem, a list of corresponding points is generated that represents a pair of corresponding points, where one of the N first feature points is a corresponding second feature point among the M second feature points. Information processing methods.

[0350] (Note 18) A program that causes a computer to perform information processing, To the aforementioned computer, Based on N first feature points (where N is an integer greater than or equal to 1) detected from the first image and M second feature points (where M is an integer greater than or equal to 1) detected from a second image different from the first image, a matching problem is generated, which is a combinatorial optimization problem in which each of the N first feature points is associated with one of the M second feature points that is presumed to represent the same object. Based on the solution to the matching problem, a list of corresponding points is generated that represents a pair of corresponding points, where one of the N first feature points is a corresponding second feature point among the M second feature points. program.

[0351] (Note 19) Circuit information describing the configuration of a circuit, written in a hardware description language, The circuit described above is made to function as an information processing device. The aforementioned information processing device is Based on N first feature points (where N is an integer greater than or equal to 1) detected from the first image and M second feature points (where M is an integer greater than or equal to 1) detected from a second image different from the first image, a matching problem is generated, which is a combinatorial optimization problem in which each of the N first feature points is associated with one of the M second feature points that is presumed to represent the same object. Based on the solution to the matching problem, a list of corresponding points is generated that represents a pair of corresponding points, where one of the N first feature points is a corresponding second feature point among the M second feature points. Circuit information.

[0352] (Note 20) Circuit information written to a reconfigurable semiconductor device in order to operate the said reconfigurable semiconductor device, The reconfigurable semiconductor device is made to function as an information processing device. The aforementioned information processing device is Based on N first feature points (where N is an integer greater than or equal to 1) detected from the first image and M second feature points (where M is an integer greater than or equal to 1) detected from a second image different from the first image, a matching problem is generated, which is a combinatorial optimization problem in which each of the N first feature points is associated with one of the M second feature points that is presumed to represent the same object. Based on the solution to the matching problem, a list of corresponding points is generated that represents a pair of corresponding points, where one of the N first feature points is a corresponding second feature point among the M second feature points. Circuit information. [Explanation of Symbols]

[0353] 10 Mobile Control System 12 cameras 14. Solver device 20 Self-position estimation device 22 Control device 32 Image Processing Unit 34 Feature Point Detection Unit 36 Feature Calculation Unit 38 Past information storage section 40 Matching Department 42 Estimation part 62 First acquisition part 64 Second acquisition part 66 Problem generation part 68 Solving section 70. Corresponding Point List Generation Unit 80 Feature Score Generation Unit 82 Position Score Generation Unit 84 Co-occurrence score generation unit 86 Constraint Score Generation Unit 88 Objective Function Generation Unit 90 Matching Problem Generation Unit 92 Outlier Removal Section 94 Selection Section 96 Image division section 98 List Synthesis Unit

Claims

1. Based on N first feature points (where N is an integer greater than or equal to 1) detected from the first image and M second feature points (where M is an integer greater than or equal to 1) detected from a second image different from the first image, a matching problem is generated, which is a combinatorial optimization problem in which each of the N first feature points is associated with one of the M second feature points that is presumed to represent the same object. Based on the solution to the matching problem, a list of corresponding points is generated that represents a pair of corresponding points, where one of the N first feature points is a corresponding second feature point among the M second feature points. An information processing device equipped with a processing unit.

2. The aforementioned processing unit, The matching problem is given to a solver device that solves the combinatorial optimization problem, and the solver device is made to solve the matching problem. The solution to the matching problem is obtained from the solver. The information processing apparatus according to claim 1.

3. The aforementioned matching problem is a combinatorial optimization problem that aims to maximize or minimize an objective function. The aforementioned objective function includes (N × M) decision variables and is minimized or maximized when all (N × M) decision variables are considered to be the correct values. When the (N × M) decision variables are arranged in N rows and M columns, the value of the decision variable in row i × column j among the (N × M) decision variables (where i is an integer between 1 and N, and j is an integer between 1 and N) represents whether the i-th first feature point among the N first feature points and the j-th second feature point among the M second feature points are the corresponding point pair. The correct value in the i-row x j-column decision variable is a value that indicates that the i-th first feature point and the j-th second feature point are a corresponding point pair if they are the same object, and a value that indicates that the i-th first feature point and the j-th second feature point are not a corresponding point pair if they are not the same object. The information processing apparatus according to claim 2.

4. The objective function includes a term that multiplies the i-row × j-column decision variable, the i'-row × j'-column decision variable among the (N × M) decision variables (where i' is an integer between 1 and N, and j' is an integer between 1 and N), and the co-occurrence score. The co-occurrence score, included in the term obtained by multiplying the i-row × j-column decision variable by the i'-row × j'-column decision variable, is a value that represents the validity of the simultaneous existence of the corresponding point pair between the i-th first feature point and the j-th second feature point, and the corresponding point pair between the i'-th first feature point out of the N first feature points and the j'-th second feature point out of the M second feature points. The information processing apparatus according to claim 3.

5. The co-occurrence score included in the term obtained by multiplying the i-row × j-column decision variable and the i'-row × j'-column decision variable is, This value corresponds to the degree of approximation between the first positional relationship between the image position of the i-th first feature point and the image position of the j-th second feature point, and the second positional relationship between the image position of the i'-th first feature point and the image position of the j'-th second feature point. The information processing apparatus according to claim 4.

6. The co-occurrence score included in the term obtained by multiplying the i-row × j-column decision variable and the i'-row × j'-column decision variable is, The first positional relationship between the image position of the i-th first feature point and the image position of the j-th second feature point, and the first degree of approximation between the second positional relationship between the image position of the i'-th first feature point and the image position of the j'-th second feature point, This value is a composite of the estimated image position of the j-th second feature point at the time of the first image, the third positional relationship between the j-th second feature point and the image position of the j-th second feature point, and the second degree of approximation of the fourth positional relationship between the i'-th first feature point and the image position of the j'-th second feature point. The information processing apparatus according to claim 4.

7. If the first condition is met, where the position of the i-th first feature point and the position of the i'-th first feature point are less than or equal to a predetermined distance, The co-occurrence score included in the term obtained by multiplying the i-row × j-column decision variable and the i'-row × j'-column decision variable is, This value corresponds to the difference in movement between the first movement of the object represented by the corresponding point pair between the i-th first feature point and the j-th second feature point, and the second movement of the object represented by the corresponding point pair between the i'-th first feature point and the j'-th second feature point. The information processing apparatus according to claim 4.

8. If the first and second images satisfy the second condition that they were captured by a camera shifted parallel to the optical axis, The co-occurrence score included in the term obtained by multiplying the i-row × j-column decision variable and the i'-row × j'-column decision variable is, The value is determined by the correlation between the size and displacement from the vanishing point on the image of the first object, represented by the corresponding point pair of the i-th first feature point and the j-th second feature point, and the size and displacement from the vanishing point on the image of the second object, represented by the corresponding point pair of the i'-th first feature point and the j'-th second feature point. The information processing apparatus according to claim 4.

9. If the first and second images satisfy the third condition that they were captured by a camera whose optical axis has been rotated, The co-occurrence score included in the term obtained by multiplying the i-row × j-column decision variable by the i'-row × j'-column decision variable is a value corresponding to the difference in movement between the direction and amount of movement of the object represented by the corresponding point pair between the i-th first feature point and the j-th second feature point, and the direction and amount of movement of the object represented by the corresponding point pair between the i'-th first feature point and the j'-th second feature point. The information processing apparatus according to claim 4.

10. The aforementioned objective function includes a term that multiplies the i-row × j-column decision variable by the feature score, The feature score multiplied by the i-row × j-column decision variable is: This value corresponds to the similarity between the feature quantity of the i-th first feature point and the feature quantity of the j-th second feature point. The information processing apparatus according to claim 3.

11. The aforementioned objective function includes a term that multiplies the i-row × j-column decision variable by the position score, The position score multiplied by the i-row × j-column decision variable is: This value corresponds to the estimated distance between the estimated position of the j-th second feature point at the time of the first image and the position of the i-th first feature point. The information processing apparatus according to claim 3.

12. The processing unit selects G candidate points (where G is an integer between 1 and M) from among the M second feature points for each of the N first feature points. The aforementioned matching problem is a combinatorial optimization problem that aims to maximize or minimize an objective function. The aforementioned objective function includes (N × G) decision variables and is minimized or maximized when all (N × G) decision variables are considered to be the correct values. When the (N × G) decision variables are arranged in N rows and G columns, the value of the decision variable in row i × column j among the (N × G) decision variables (where i is an integer between 1 and N, and j is an integer between 1 and G) represents whether the i-th first feature point among the N first feature points and the j-th candidate point among the G candidate points selected for the i-th first feature point are the corresponding point pair. The correct value in the i-row x j-column decision variable is a value that indicates that the i-th first feature point and the j-th candidate point selected for the i-th first feature point are a corresponding point pair if they are the same object, and a value that indicates that the i-th first feature point and the j-th candidate point selected for the i-th first feature point are not a corresponding point pair if they are not the same object. The information processing apparatus according to claim 1.

13. The first image is the first partial image among a plurality of partial images obtained by dividing the first input image into a plurality of predetermined regions, The second image is a second sub-image of the same region as the first sub-image, among the plurality of sub-images obtained by dividing a second input image that is different from the first input image. The information processing apparatus according to claim 1.

14. Of the predetermined plurality of regions, the first region and the second region, which are adjacent to each other, partially overlap. The aforementioned processing unit, A combined correspondence point list is generated by combining the correspondence point lists, each of which is generated as the first image, obtained by dividing the first input image into multiple sub-images. To ensure that the composite correspondence point list does not contain two or more correspondence point pairs in which at least one of the first or second feature points overlaps, some of the two or more correspondence point pairs in which at least one of the first or second feature points overlaps are deleted. The information processing apparatus according to claim 13.

15. Of the predetermined plurality of regions, the first region and the second region, which are adjacent to each other, partially overlap. The aforementioned processing unit, A combined correspondence point list is generated by combining the correspondence point lists, each of which is generated as the first image, obtained by dividing the first input image into multiple sub-images. If the composite correspondence point list includes two or more pairs of correspondence points in which at least one of the first or second feature points overlaps, a new subregion containing the two or more pairs of correspondence points is identified. Based on the plurality of first feature points detected from the new subregion in the first input image and the plurality of second feature points detected from the new subregion in the second input image, the matching problem is regenerated. Based on the solution to the matching problem that was generated again, one or more new pairs of corresponding points are generated by associating one of the first feature points of the plurality of first feature points with one of the second feature points of the plurality of second feature points. The composite correspondence point list is updated by replacing the two or more pairs of corresponding points in which at least one of the first or second feature points overlaps with the one or more new pairs of corresponding points. The information processing apparatus according to claim 13.

16. The aforementioned processing unit, From the first image captured by the camera, N first feature points are extracted, and from the second image captured by the camera at a different time than the first image, M second feature points are extracted. Based on the aforementioned list of corresponding points, the position of the camera in three-dimensional space is estimated. The information processing apparatus according to claim 1.

17. An information processing method performed by an information processing device, Based on N first feature points (where N is an integer greater than or equal to 1) detected from the first image and M second feature points (where M is an integer greater than or equal to 1) detected from a second image different from the first image, a matching problem is generated, which is a combinatorial optimization problem in which each of the N first feature points is associated with one of the M second feature points that is presumed to represent the same object. Based on the solution to the matching problem, a list of corresponding points is generated that represents a pair of corresponding points, where one of the N first feature points is a corresponding second feature point among the M second feature points. Information processing methods.

18. A program that causes a computer to perform information processing, To the aforementioned computer, Based on N first feature points (where N is an integer greater than or equal to 1) detected from a first image and M second feature points (where M is an integer greater than or equal to 1) detected from a second image different from the first image, a matching problem is generated, which is a combinatorial optimization problem in which each of the N first feature points is associated with one of the M second feature points that is presumed to represent the same object. Based on the solution to the matching problem, a list of corresponding points is generated that represents a pair of corresponding points, where one of the N first feature points is a corresponding second feature point among the M second feature points. program.

19. Circuit information describing the configuration of a circuit, written in a hardware description language, The circuit described above is made to function as an information processing device. The aforementioned information processing device is Based on N first feature points (where N is an integer greater than or equal to 1) detected from the first image and M second feature points (where M is an integer greater than or equal to 1) detected from a second image different from the first image, a matching problem is generated, which is a combinatorial optimization problem in which each of the N first feature points is associated with one of the M second feature points that is presumed to represent the same object. Based on the solution to the matching problem, a list of corresponding points is generated that represents a pair of corresponding points, where one of the N first feature points is a corresponding second feature point among the M second feature points. Circuit information.

20. Circuit information written to a reconfigurable semiconductor device in order to operate the said reconfigurable semiconductor device, The reconfigurable semiconductor device is made to function as an information processing device. The aforementioned information processing device is Based on N first feature points (where N is an integer greater than or equal to 1) detected from the first image and M second feature points (where M is an integer greater than or equal to 1) detected from a second image different from the first image, a matching problem is generated, which is a combinatorial optimization problem in which each of the N first feature points is associated with one of the M second feature points that is presumed to represent the same object. Based on the solution to the matching problem, a list of corresponding points is generated that represents a pair of corresponding points, where one of the N first feature points is a corresponding second feature point among the M second feature points. Circuit information.

Images