Information processing device, information processing method, and program

Abstract

Provided is an information processing device having: a detection unit that, on the basis of an assumed location and an assumed positioning of an imaging device at a second time, re-projects, into a second image captured at the second time, the three-dimensional locations of a plurality of feature points in a first image captured at a first time, and detects each feature point in the second image by performing matching in a first search range from the respective locations where the feature points were re-projected; and an estimation unit that estimates the location and positioning of the imaging device at the second time on the basis of the detected three-dimensional locations of the plurality of feature points, wherein when, among the feature points that were detected in the second image by performing matching, the number of feature points for which the re-projection error is at or below a threshold value satisfies a specific condition, said re-projection error being based on the location and positioning of the imaging device that were estimated by the estimation unit, the detection unit detects each feature point in the second image again by performing matching in a second search range that is larger than the first search range.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program.

[0002] A technique is known for estimating the position and orientation of an image capturing device (camera) by matching feature points included in a captured image with three-dimensional coordinates (map points) of the feature points calculated in advance (for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2016-091065

[0004] However, the technology described in Patent Document 1 leaves room for improvement in, for example, the accuracy of estimating the position and orientation of the image capturing device.

[0005] In view of the above-described problems, an object of the present disclosure is to provide a technology that can appropriately estimate the position and orientation of an image capturing device.

[0006] In a first aspect of the present disclosure, there is provided an information processing device including: an acquisition unit that acquires each image captured by an image capture device at each time point; a detection unit that assumes a position and orientation of the image capture device at a second time point, and re-projects three-dimensional positions of a plurality of feature points on a first image captured at a first time point prior to the second time point onto a second image captured at the second time point based on the assumed position and orientation of the image capture device, and detects each feature point on the second image by matching in a first search range from the re-projected position of each feature point; and an estimation unit that estimates the position and orientation of the image capture device at the second time point based on the three-dimensional positions of the plurality of feature points detected by the detection unit, wherein if the number of feature points detected by matching in the second image for which a re-projection error based on the position and orientation of the image capture device estimated by the estimation unit is equal to or less than a threshold, the detection unit re-detects each feature point on the second image by matching in a second search range larger than the first search range.

[0007] Furthermore, a second aspect of the present disclosure provides an information processing method that acquires each image captured by an image capture device at each time point, assumes a position and orientation of the image capture device at a second time point, re-projects three-dimensional positions of a plurality of feature points on a first image captured at a first time point prior to the second time point onto a second image captured at the second time point based on the assumed position and orientation of the image capture device, detects each feature point on the second image by matching in a first search range from the re-projected position of each feature point, and estimates the position and orientation of the image capture device at the second time point based on the detected three-dimensional positions of the plurality of feature points, and if the number of feature points detected in the second image by matching, for which a re-projection error based on the estimated position and orientation of the image capture device is equal to or less than a threshold, satisfies a specific condition, detects each feature point on the second image by matching in a second search range larger than the first search range, and re-estimates the position and orientation of the image capture device at the second time point based on the detected three-dimensional positions of the plurality of feature points.

[0008] Furthermore, in a third aspect of the present disclosure, there is provided a program for causing a computer to execute processes of acquiring each image captured by an image capture device at each time point, assuming a position and orientation of the image capture device at a second time point, re-projecting three-dimensional positions of a plurality of feature points on a first image captured at a first time point prior to the second time point onto a second image captured at the second time point based on the assumed position and orientation of the image capture device, detecting each feature point on the second image by matching within a first search range from the re-projected position of each feature point, estimating the position and orientation of the image capture device at the second time point based on the detected three-dimensional positions of the plurality of feature points, and if the number of feature points detected in the second image by matching, for which a re-projection error based on the estimated position and orientation of the image capture device is equal to or less than a threshold, satisfying a specific condition, detecting each feature point on the second image by matching within a second search range larger than the first search range, and re-estimating the position and orientation of the image capture device at the second time point based on the detected three-dimensional positions of the plurality of feature points.

[0009] According to one aspect, the position and orientation of the image capturing device can be appropriately estimated.

[0010] Fig. 1 is a diagram showing an example of the configuration of an information processing device according to an embodiment; Fig. 2 is a diagram showing an example of the configuration of an information processing system according to an embodiment; Fig. 3 is a diagram showing an example of the hardware configuration of an information processing device according to an embodiment; Fig. 4 is a flowchart showing an example of processing of an information processing device according to an embodiment; Fig. 5 is a diagram showing an example of processing of an information processing device according to an embodiment.

[0011] The principles of the present disclosure will be described with reference to some exemplary embodiments. It should be understood that these embodiments are set forth for illustrative purposes only, to aid those skilled in the art in understanding and practicing the present disclosure, without implying any limitation on the scope of the disclosure. The disclosure described herein may be implemented in various ways other than those described below.

[0012] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that each drawing is merely an example for describing one or more embodiments. Each drawing is not related to only one particular embodiment, but may also be related to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessarily required to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0014] (Embodiment 1) <Configuration> The configuration of an information processing device 10 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the information processing device 10 according to an embodiment. The information processing device 10 has an acquisition unit 11, a detection unit 12, and an estimation unit 13. These units may be realized by cooperation between one or more programs installed in the information processing device 10 and hardware such as a processor and memory of the information processing device 10. The acquisition unit 11 acquires each image captured by an imaging device at each point in time.

[0015] The detection unit 12 assumes the position and orientation of the image capture device at the second time point, and reprojects the three-dimensional positions of multiple feature points on a first image captured at a first time point prior to the second time point onto a second image captured at the second time point based on the assumed position and orientation of the image capture device. The detection unit 12 then detects each feature point on the second image by matching within a first search range from the reprojected position of each feature point.

[0016] Furthermore, the detection unit 12 determines whether or not the number of feature points for which the reprojection error based on the position and orientation of the image capturing device estimated by the estimation unit 13 is equal to or less than a threshold satisfies a specific condition, among the feature points detected by matching in the second image. If the specific condition is satisfied, the detection unit 12 then redetects each feature point on the second image by matching in a second search range that is larger than the first search range.

[0017] The estimation unit 13 estimates the position and orientation of the image capturing device at the second time point based on the three-dimensional positions of the plurality of feature points found by the detection unit 12 .

[0018] (Embodiment 2) Next, the configuration of an information processing system 1 according to an embodiment will be described with reference to Fig. 2. <System Configuration> Fig. 2 is a diagram showing an example of the configuration of an information processing system 1 according to an embodiment. In the example of Fig. 2, the information processing system 1 has an information processing device 10 and a mobile object 20. In the example of Fig. 1, the information processing device 10 and the mobile object 20 are connected so as to be able to communicate with each other via a network N. Note that the number of information processing devices 10 and the mobile objects 20 is not limited to the example of Fig. 2.

[0019] Examples of the network N include the Internet, a mobile communication system, a wireless local area network (LAN), a LAN, a bus, etc. Examples of the mobile communication system include a fifth generation mobile communication system (5G), a sixth generation mobile communication system (6G, Beyond 5G), a fourth generation mobile communication system (4G), a third generation mobile communication system (3G), etc.

[0020] The information processing device 10 is, for example, a device such as a cloud server, a server, an ECU (Electronic Control Unit), or a microcomputer. The information processing device 10 may be installed in a facility remote from the mobile object 20, or may be mounted (built-in) on the mobile object 20. The information processing device 10 may perform so-called Visual SLAM (Simultaneous Localization and Mapping), which uses a camera to simultaneously estimate its own position and create environmental map information of the surrounding area (understand the surrounding space).

[0021] The mobile body 20 may be, for example, a vehicle that runs on roads or the like using wheels, a railway vehicle that runs on tracks, a robot that moves on land using wheels or legs, an unmanned aerial vehicle (drone), an aircraft, a ship, etc. In this case, the mobile body 20 may use the information processing device 10 to provide, for example, autonomous movement (flight) without relying on instructions from a pilot, or an advanced driver-assistance system (ADAS) for assisting driving operations.

[0022] The mobile object 20 may also be a wearable device such as a head-mounted display or smart glasses, a smartphone, a tablet terminal, a game console, or a car navigation device. In this case, the mobile object 20 may provide an augmented reality (AR) service using the information processing device 10, for example.

[0023] The moving body 20 has an image capturing device 21. The image capturing device 21 is a camera, a monocular camera, and is mounted on the moving body 20 to capture images at each point in time.

[0024] <Hardware Configuration> Fig. 3 is a diagram showing an example of the hardware configuration of the information processing device 10 according to the embodiment. In the example of Fig. 3, the information processing device 10 (computer 100) includes a processor 101, a memory 102, and a communication interface 103. These components may be connected via a bus or the like. The memory 102 stores at least a part of a program 104. The communication interface 103 includes an interface required for communication with other network elements.

[0025] When the program 104 is executed by the processor 101, memory 102, and other components in cooperation with each other, the computer 100 performs at least some of the processing of the embodiments of the present disclosure. The memory 102 may be of any type suitable for a local technology network. The memory 102 may be, by way of non-limiting example, a non-transitory computer-readable storage medium. The memory 102 may also be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. While only one memory 102 is shown in the computer 100, several physically distinct memory modules may be present in the computer 100. The processor 101 may be of any type. The processor 101 may include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), and, by way of non-limiting example, a processor based on a multi-core processor architecture. The computer 100 may have multiple processors, such as application-specific integrated circuit chips time-slaved to a clock that synchronizes the main processor.

[0026] Embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device.

[0027] The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, that execute on a target real or virtual processor or device to perform the processes or methods of the present disclosure. Program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or divided among program modules as desired in various embodiments. The machine-executable instructions of the program modules may be executed in local or distributed devices. In a distributed device, the program modules may be located in both local and remote storage media.

[0028] The program code for executing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus. When the program code is executed by the processor or controller, the functions / acts in the flowcharts and / or implementing block diagrams are performed. The program code may be executed entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine, or entirely on a remote machine or server.

[0029] The program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible recording media. Examples of non-transitory computer-readable media include magnetic recording media, magneto-optical recording media, optical disk media, and semiconductor memory. Magnetic recording media include, for example, flexible disks, magnetic tapes, and hard disk drives. Magneto-optical recording media include, for example, magneto-optical disks. Optical disk media include, for example, Blu-ray discs, CD (Compact Disc)-ROM (Read Only Memory), CD-R (Recordable), and CD-RW (Rewritable). Semiconductor memory includes, for example, solid-state drives, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, and RAM (Random Access Memory). The program may also be supplied to a computer by various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire or an optical fiber, or via a wireless communication path.

[0030] <Processing> Next, an example of processing by the information processing device 10 according to the embodiment will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a flowchart showing an example of processing by the information processing device 10 according to the embodiment. Fig. 5 is a diagram showing an example of processing by the information processing device according to the embodiment. Note that the processing in Fig. 4 may be executed periodically (for example, every 1 / 60 seconds when the image capturing device 21 captures images at 60 fps (Frames Per Second)). Note that, below, feature points whose three-dimensional positions are recorded in the environmental map information will also be referred to as "landmarks" as appropriate.

[0031] In step S101, the detection unit 12 assumes the position and orientation of the image capturing device 21 at a second time point (for example, the time point when the current processing is performed; the current time point). Here, the detection unit 12 may assume the position and orientation of the image capturing device 21 at the second time point based on, for example, the moving speed and moving direction of the moving object 20 calculated based on an image previously captured by the image capturing device 21.

[0032] In this case, the detection unit 12 may, for example, assume that the moving body 20 is moving at a constant speed in a straight line, and calculate the position and orientation of the image capturing device 21 at the time of the current processing based on the moving speed and moving direction of the moving body 20 at the time of the previous processing (the time when the processing in FIG. 4 was previously performed). Alternatively, the detection unit 12 may, for example, calculate the position and orientation of the image capturing device 21 at the time of the current processing by time series prediction using a Kalman filter.

[0033] Next, based on the assumed position and orientation of the image capturing device 21, the detection unit 12 reprojects the three-dimensional positions of multiple feature points (landmarks) on the first image captured by the image capturing device 21 at a first time point prior to the second time point (e.g., the time point when the process of FIG. 4 was last performed) onto the second image captured by the image capturing device 21 at the second time point (step S102). Note that each feature point may be extracted from the image by, for example, Scale-invariant feature transform (SIFT), Speed-Upped Robust Feature (SURF), Oriented-BRIEF (ORB), Accelerated KAZE (AKAZE), or the like.

[0034] In the example of FIG. 5 , feature points 511A, 512A, and 513A are captured in a first image 501, which was the previous processing target and was captured by the image capturing device 21 when the previous processing was performed. The three-dimensional positions (positions in three-dimensional space) of the landmarks 511, 512, and 513 were calculated when the previous processing was performed and recorded in the environment map information. The environment map information may include information on each key frame and information on the three-dimensional position of each landmark. The key frame may include, for example, an image to be matched with the current processing target image and information indicating each landmark included in the image.

[0035] Then, the detection unit 12 calculates an image (projection surface) to be captured at the assumed position and attitude 522 of the image capturing device 21, based on the image capturing conditions such as the angle of view of the image capturing device 21. Note that the image capturing conditions such as the angle of view of the image capturing device 21 may be set (registered) in advance in the information processing device 10 by an administrator of the information processing device 10, for example.

[0036] Then, the detection unit 12 performs projection transformation such as perspective projection to project (reproject) the three-dimensional positions of the landmarks 511, 512, and 513 onto the second image 521. In the example of Fig. 5, the three-dimensional positions of the landmarks 511, 512, and 513 are projected onto pixel positions 511B, 512B, and 513B, respectively, on the second image 521.

[0037] Next, the detection unit 12 detects each feature point on the second image by matching within a first search range from the reprojected position of each landmark (step S103). The first search range may be, for example, a circle with a radius of M (e.g., 3) pixels centered on the projected pixel position, or a circle with a height M １ ×M ２ The first search range may be a rectangle in which all four corners of the pixel are right angles. The first search range may be set in advance in the information processing device 10 by an administrator of the information processing device 10 or the like. The first search range may also be determined by the detection unit 12, for example, in accordance with the acceleration of the moving object 20 or the like.

[0038] In the example of FIG. 5, pixels included in first search ranges 531B, 532B, and 533B centered around pixel positions 511B, 512B, and 513B, respectively, on second image 521 are targeted for matching.

[0039] Here, the detection unit 12 may calculate, for example, the distance between the vector representation of the feature point on the first image and the vector representation of the feature point on the second image. Then, the detection unit 12 may determine that the matching is successful (the feature points are the same on the same object) if the distance is equal to or less than a threshold, and may determine that the matching is unsuccessful if the distance is not equal to or less than the threshold. In this case, the detection unit 12 may calculate, for example, the distance D in the vector representation of the feature point by n (e.g., 256)-bit ORB (Oriented Fast and Rotated BRIEF) by the following formula (1): D=Σ(a ｉ ==b ｉ )? 0:1 ... (1)

[0040] Here, i is the subscript of Σ and takes values ​​from 0 to n-1. ｉ is the value of the i-th bit of the feature point in the first image, and b ｉ is the value of the i-th bit of the feature point in the second image. ｉ and b ｉ If and are equal, it is not added to D (0 is added), and a ｉ and b ｉ If the distance D is not equal to the distance D, 1 is added to D. Therefore, D can take a value from 0 to n. The detection unit 12 may determine that the matching is successful if the distance D is equal to or less than a threshold value (for example, 50). Note that, for example, if there are multiple feature points that have been successfully matched in the first search range, the detection unit 12 may select the feature point with the smallest value of the distance D as the feature point that has been successfully matched.

[0041] Next, the estimation unit 13 estimates the position and orientation of the image capture device 21 at the second time point based on the three-dimensional positions of the plurality of feature points detected by the detection unit 12 (step S104). Here, the estimation unit 13 estimates the position and orientation of the image capture device 21 at the second time point based on the three-dimensional positions of the plurality of feature points detected by the detection unit 12, for example, by bundle adjustment using the reprojection error μ ｉｊ -μ'(pｉ , q ｊ Alternatively, the position and orientation of the image capturing device 21 may be calculated so that the sum S of the squares of μ ｉｊ is the position of the feature point on the second image, and p ｉ is the position and orientation of the image capturing device 21, and q ｊ is the three-dimensional position of the landmark, and μ'(p ｉ , q ｊ ) is p ｉ and q ｊ The position of the landmark on the second image is calculated from S=Σ(μ ｉｊ -μ'(p ｉ , q ｊ )) ２ ... (2)

[0042] Here, the estimation unit 13 may estimate the position and orientation of the image capture device 21 and generate environmental map information about the surroundings of the image capture device 21 based on the image acquired by the acquisition unit 11. In this case, the estimation unit 13 may add, for example, the second image and information indicating each landmark detected in the second image to the environmental map information as key frames. This makes it possible to realize Visual SLAM.

[0043] Next, the detection unit 12 sets (classifies) each landmark as an inlier or an outlier based on the position and orientation of the image capture device 21 estimated in the process of step S104 (step S105). Here, the detection unit 12 sets (classifies) each landmark as an inlier or an outlier based on the position and orientation of the image capture device 21 estimated in the process of step S104 (step S105). ｉｊ -μ'(p ｉ , q ｊ ) is equal to or less than a threshold (e.g., 3 pixels), the landmark may be set as an inlier (imaged in the second image). Also, for example, if the reprojection error is not equal to or less than the threshold, the detection unit 12 may set the landmark as an outlier (not imaged in the second image).

[0044] Next, the detection unit 12 determines whether the number of landmarks determined to be inliers (reprojection errors equal to or less than a threshold) among the feature points (landmarks) detected by matching in the second image satisfies a specific condition (step S106). The specific condition may include a condition based on the number of landmarks determined to be inliers in the second image and the number of landmarks determined to be inliers in the first image at the first time point. In this case, the specific condition may include at least one of the ratio and the difference between the number of landmarks determined to be inliers in the second image and the number of landmarks determined to be inliers in the first image at the first time point being equal to or less than a threshold. As a result, for example, if the number of landmarks determined to be inliers in the current image suddenly decreases compared to the number of landmarks determined to be inliers last time, it is considered that the deviation in the assumed position and orientation of the image capture device 21 is relatively large, and therefore it can be determined that the specific condition is satisfied.

[0045] The number m of landmarks determined to be inliers in the second image under the specific condition ２ and the number m of landmarks determined to be inliers in the first image at the first time point. １ The ratio of ｐ In this case, the detection unit 12 may determine that the specific condition is satisfied when the following formula (3) is satisfied: m ２ / m １ ≦R ｐ ...(3)

[0046] The specific condition is the number m of landmarks determined to be inliers in the second image. ２ and the target value I of the number of feature points to be detected. ｔ The ratio of ｔ The target value I of the number of feature points to be detected may be less than or equal to the target value I ｔ may be set in advance in the information processing device 10 by an administrator of the information processing device 10 or the like.

[0047] As a result, for example, if the number of landmarks determined to be inliers this time is relatively small, it is considered that the deviation of the assumed position and orientation of the image capturing device 21 is relatively large, and therefore it can be determined that the specific condition is met. In this case, the detection unit 12 may determine that the specific condition is met when the following formula (4) is met. ２ / I ｔ ≦R ｔ ...(4)

[0048] If the number of landmarks determined to be inliers in the second image does not satisfy a specific condition (NO in step S106), the process ends. On the other hand, if the specific condition is satisfied (YES in step S106), the detection unit 12 searches for each feature point on the second image from the reprojected position of each landmark in a second search range that is larger than the first search range, detects each feature point on the second image by matching (step S107), and proceeds to the process of step S104.

[0049] Here, the detection unit 12 determines, for example, the size M of the second search range. Ｅ (radius, or height and width) is calculated as the size of the first search range M Ｎ may be determined to be a value obtained by adding a constant C to M. The constant C (for example, 10 to 20) is an integer of 1 or more. Ｅ = C + M Ｎ ...(5)

[0050] Furthermore, the detection unit 12 may, for example, determine the size M of the second search range. Ｅ is expressed as the size M of the first search range as in the following equation (6): Ｎ Alternatively, the value may be determined by multiplying the coefficient α by the coefficient α, where the value of the coefficient α is greater than 1. Ｅ = α × M Ｎ ...(6)

[0051] Furthermore, the detection unit 12 may determine the second search range to be larger, for example, as the number of landmarks determined to be inliers in the second image is smaller. This allows the size of the second search range to be larger, for example, as the number of landmarks determined to be inliers this time is smaller, since it is considered that the deviation of the assumed position and orientation of the image capture device 21 is relatively larger. In this case, the detection unit 12 may determine the size M of the second search range, for example. Ｅ may be determined as in the following equation (7): Ｅ = (1-m ２ / I ｔ ) × C + M Ｎ ... (7)

[0052] Furthermore, the detection unit 12 may determine the second search range to be larger, for example, as the number of landmarks determined to be inliers in the second image decreases and as the number of landmarks determined to be inliers in the first image at the first time point (for example, during the previous processing) increases. As a result, for example, the larger the number of landmarks determined to be inliers this time becomes, the larger the deviation in the assumed position and orientation of the image capture device 21 is considered to be, and the larger the size of the second search range can be.

[0053] In this case, the detection unit 12 determines, for example, the size M of the second search range. Ｅ may be determined as in the following equation (8) or (9). Note that min(A, B) represents the smaller value of A or B. M Ｅ = (1-m ２ / m １ ) × C + M Ｎ ... (8) M Ｅ = {1-min (m ２ / m １ , m ２ / I ｔ ) × C + M Ｎ ... (9)

[0054] <Others> When the position and orientation of the image capture device 21 are assumed using uniform linear motion or time series prediction using a Kalman filter, deviations from the assumptions become relatively large when the moving body 20 is rapidly accelerating or rapidly rotating. Even when deviations from the assumptions become relatively large, the possibility of detecting feature points increases if the first search range is set relatively large. However, setting the first search range relatively large increases the number of pixels to be matched, resulting in a decrease in processing speed and an increase in false matches.

[0055] On the other hand, according to the present disclosure, if the number of landmarks determined to be inliers in the first search range satisfies a specific condition, feature points on the image are detected by matching in a second search range that is larger than the first search range, thereby enabling the position and orientation of the image capture device to be appropriately estimated.

[0056] <Modifications> The information processing device 10 may be a device contained in a single housing, but the information processing device 10 of the present disclosure is not limited to this. Each unit of the information processing device 10 may be realized, for example, by cloud computing configured with one or more computers. The information processing device 10 may also be mounted on a mobile object 20. In this case, the information processing device 10 and the imaging device 21 may be housed in the same housing and configured as an integrated information processing device. At least some of the processing of each functional unit of the information processing device 10 may also be executed by a computer possessed by at least one of the mobile object 20 and the imaging device 21. Such information processing devices 10 are also included in examples of the "information processing device" of the present disclosure.

[0057] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0058] The present invention is not limited to the above-described embodiment, and can be modified as appropriate within the scope of the invention.

[0059] Some or all of the above embodiments may also be described as, but are not limited to, the following appendices. Note that some or all of the elements (e.g., configurations and functions) described in each appendix dependent on appendix 1 may also be dependent on independent appendices in other categories in a similar dependency relationship. Some or all of the elements described in any appendix may be applied to various hardware, software, recording means for recording software, systems, and methods. an acquisition unit that acquires each image captured by an image capture device at each time point; a detection unit that assumes a position and orientation of the image capture device at a second time point, and re-projects three-dimensional positions of a plurality of feature points on a first image captured at a first time point prior to the second time point onto a second image captured at the second time point based on the assumed position and orientation of the image capture device, and detects each feature point on the second image by matching in a first search range from the re-projected position of each feature point; and an estimation unit that estimates the position and orientation of the image capture device at the second time point based on the three-dimensional positions of the plurality of feature points detected by the detection unit, wherein the detection unit re-detects each feature point on the second image by matching in a second search range larger than the first search range when a specific condition is satisfied for a number of feature points, among the feature points detected by matching in the second image, for which a re-projection error based on the position and orientation of the image capture device estimated by the estimation unit is equal to or less than a threshold value. (Supplementary Note 2) The information processing device according to Supplementary Note 1, wherein the specific condition includes a condition based on the number of feature points in the second image whose reprojection error is equal to or less than a threshold and the number of feature points for the first image whose reprojection error is equal to or less than the threshold at the first time point. (Supplementary Note 3) The information processing device according to Supplementary Note 2, wherein the specific condition includes at least one of a ratio and a difference between the number of feature points in the second image whose reprojection error is equal to or less than a threshold and the number of feature points for the first image whose reprojection error is equal to or less than the threshold at the first time point being equal to or less than a threshold. (Supplementary Note 4) The information processing device according to Supplementary Note 1, wherein the specific condition includes the number of feature points in the second image whose reprojection error is equal to or less than a threshold being equal to or less than a threshold.(Supplementary Note 5) The information processing device according to Supplementary Note 1 or 2, wherein the detection unit calculates a distance between a vector representation of a feature point on the first image and a vector representation of a feature point on the second image, and determines that a re-projection error is equal to or less than a threshold if the distance is equal to or less than a threshold. (Supplementary Note 6) The information processing device according to Supplementary Note 1 or 2, wherein the detection unit determines the second search range to be larger the fewer the number of feature points in the second image whose re-projection error is equal to or less than the threshold. (Supplementary Note 7) The information processing device according to Supplementary Note 6, wherein the detection unit determines the second search range to be larger the greater the number of feature points in the first image whose re-projection error is equal to or less than the threshold at the first time point. (Supplementary Note 8) The information processing device according to Supplementary Note 1 or 2, wherein the estimation unit generates environmental map information of the surroundings of the imaging device based on the image acquired by the acquisition unit, and the detection unit detects three-dimensional positions of the plurality of feature points based on the environmental map information. (Supplementary Note 9) An information processing method comprising: acquiring each image captured by an image capture device at each time point; assuming a position and orientation of the image capture device at a second time point; re-projecting three-dimensional positions of a plurality of feature points on a first image captured at a first time point prior to the second time point onto a second image captured at the second time point based on the assumed position and orientation of the image capture device; detecting each feature point on the second image by matching in a first search range from the re-projected position of each feature point; estimating the position and orientation of the image capture device at the second time point based on the detected three-dimensional positions of the plurality of feature points; if the number of feature points detected in the second image by matching, for which a re-projection error based on the estimated position and orientation of the image capture device is equal to or less than a threshold, satisfies a specific condition, detecting each feature point on the second image by matching in a second search range larger than the first search range; and re-estimating the position and orientation of the image capture device at the second time point based on the detected three-dimensional positions of the plurality of feature points.(Supplementary Note 10) A program that causes a computer to execute processes of acquiring each image captured by an image capture device at each time point, assuming a position and orientation of the image capture device at a second time point, re-projecting three-dimensional positions of a plurality of feature points on a first image captured at a first time point prior to the second time point onto a second image captured at the second time point based on the assumed position and orientation of the image capture device, detecting each feature point on the second image by matching in a first search range from the re-projected position of each feature point, estimating the position and orientation of the image capture device at the second time point based on the detected three-dimensional positions of the plurality of feature points, and if the number of feature points detected in the second image by matching that have a re-projection error based on the estimated position and orientation of the image capture device equal to or less than a threshold satisfies a specific condition, detecting each feature point on the second image by matching in a second search range that is larger than the first search range, and re-estimating the position and orientation of the image capture device at the second time point based on the detected three-dimensional positions of the plurality of feature points.

[0060] This application claims priority based on Japanese Patent Application No. 2023-207423, filed December 8, 2023, the disclosure of which is incorporated herein by reference in its entirety.

[0061] REFERENCE SIGNS LIST 1 Information processing system 10 Information processing device 11 Acquisition unit 12 Detection unit 13 Estimation unit 20 Moving object 21 Imaging device

Claims

1. An information processing device comprising: an acquisition unit that acquires each image captured by an image capture device at each time point; a detection unit that assumes a position and orientation of the image capture device at a second time point; and re-projects three-dimensional positions of a plurality of feature points on a first image captured at a first time point prior to the second time point onto a second image captured at the second time point based on the assumed position and orientation of the image capture device; and detects each feature point on the second image in a first search range from the re-projected positions of each feature point by matching; and an estimation unit that estimates the position and orientation of the image capture device at the second time point based on the three-dimensional positions of the plurality of feature points detected by the detection unit; 2. The information processing device of claim 1, wherein the specific condition includes a condition based on the number of feature points in the second image whose reprojection error is below a threshold and the number of feature points in the first image whose reprojection error is below a threshold at the first point in time.

3. The information processing device described in claim 2, wherein the specific condition includes that at least one of the ratio and the difference between the number of feature points in the second image whose reprojection error is below a threshold and the number of feature points in the first image whose reprojection error is below a threshold at the first time point is below a threshold.

4. The information processing device according to claim 1, wherein the specific condition includes that the number of feature points in the second image having a reprojection error equal to or less than a threshold is equal to or less than a threshold.

5. The information processing device according to claim 1 or 2, wherein the detection unit calculates a distance between a vector representation of a feature point on the first image and a vector representation of a feature point on the second image, and if the distance is equal to or less than a threshold, determines that the reprojection error is equal to or less than a threshold.

6. The information processing device according to claim 1 or 2, wherein the detection unit determines the second search range to be larger as the number of feature points in the second image having a reprojection error equal to or smaller than a threshold value decreases.

7. The information processing device according to claim 6, wherein the detection unit determines the second search range to be larger as the number of feature points having a reprojection error equal to or smaller than a threshold value for the first image at the first time point increases.

8. An information processing device as described in claim 1 or 2, wherein the estimation unit generates environmental map information of the surroundings of the photographing device based on the image acquired by the acquisition unit, and the detection unit detects three-dimensional positions of the multiple feature points based on the environmental map information.

9. An information processing method comprising: acquiring each image taken by an image capture device at each time point; assuming a position and orientation of the image capture device at a second time point; re-projecting three-dimensional positions of a plurality of feature points on a first image taken at a first time point prior to the second time point onto a second image taken at the second time point based on the assumed position and orientation of the image capture device; detecting each feature point on the second image in a first search range from the re-projected position of each feature point by matching; estimating the position and orientation of the image capture device at the second time point based on the three-dimensional positions of the detected plurality of feature points; if the number of feature points detected in the second image by matching that have a re-projection error based on the estimated position and orientation of the image capture device below a threshold satisfies a specific condition, detecting each feature point on the second image by matching in a second search range larger than the first search range; and re-estimating the position and orientation of the image capture device at the second time point based on the detected three-dimensional positions of the plurality of feature points.

10. A program that causes a computer to execute the following processes: acquiring each image taken by a camera at each time point; assuming a position and orientation of the camera at a second time point; re-projecting three-dimensional positions of a plurality of feature points on a first image taken at a first time point prior to the second time point onto a second image taken at the second time point based on the assumed position and orientation of the camera; detecting each feature point on the second image in a first search range from the re-projected position of each feature point by matching; estimating the position and orientation of the camera at the second time point based on the three-dimensional positions of the detected plurality of feature points; and, if the number of feature points detected in the second image by matching that have a re-projection error based on the estimated position and orientation of the camera below a threshold value satisfies a specific condition, detecting each feature point on the second image by matching in a second search range larger than the first search range; and re-estimating the position and orientation of the camera at the second time point based on the detected three-dimensional positions of the plurality of feature points.

Images