Information processing device, information processing method, and program

The information processing device uses multiple matching processes to stabilize and enhance the accuracy of camera position and orientation calculations under varying lighting conditions by increasing the number of correctly matched feature points.

JP7581016B2Active Publication Date: 2024-11-12CANON KK
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2020182541
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-10-30
Publication Date
2024-11-12
Estimated Expiration
2040-10-30

AI Technical Summary

Technical Problem

Existing methods for calculating the position and orientation of a camera fail to maintain accuracy and stability due to changes in environmental lighting conditions or camera exposure, leading to reduced posture accuracy and stability.

Method used

An information processing device that associates characteristic regions between images using multiple matching processes, including a first and a second association process, to stabilize and enhance the accuracy of position and orientation calculations even under varying lighting conditions.

Benefits of technology

The device achieves stable and highly accurate acquisition of camera position and orientation by increasing the number of correctly matched feature points, even when image brightness changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007581016000001
    Figure 0007581016000001
  • Figure 0007581016000002
    Figure 0007581016000002
  • Figure 0007581016000003
    Figure 0007581016000003
Patent Text Reader

Abstract

To allow stable and highly accurate acquisition of a position attitude.SOLUTION: An information processing apparatus (1) according to the present invention has first associating means (12) for associating positions of feature regions of a first image and a second image input from an imaging device (10) on a time serial basis with each other, first determination means (13) for determining whether or not association by the first associating means of the positions of the feature regions is successful, second associating means (14) for, if the association by the first associating means of the positions of the feature regions is not successful, associating the positions of the feature regions of the first image and the second image with each other by an association method different from that of the first association means, and acquiring means (15) for acquiring a positional attitude of the imaging device based on the positions of the feature regions on the associated second image.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing technique for acquiring the position and orientation of an imaging device. [Background technology]

[0002] Measurement of the position and orientation of a camera based on image information is used for various purposes, such as alignment between real space and virtual objects in mixed reality (MR) / augmented reality (AR), self-position estimation of robots and automobiles, 3D modeling of objects and spaces, etc. Patent Document 1 discloses a method of calculating the position and orientation of a camera by matching feature points detected from an image between images that are adjacent in time. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-9918 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when the brightness of an image changes due to a change in environmental lighting conditions or camera exposure, the technology of Patent Document 1 is likely to fail to match feature points in successive images, resulting in a problem of reduced accuracy and stability of the position and orientation.

[0005] Therefore, an object of the present invention is to make it possible to acquire a position and orientation stably and with high accuracy. [Means for solving the problem]

[0006] The information processing device of the present invention includes a first matching means for matching positions of characteristic regions between a first image and a second image input in time series from an imaging device, and a second matching means for matching positions of the characteristic regions between the first image and the second image using a method different from the method used by the first matching means when the first matching means has failed to match the positions of the characteristic regions; a position of a feature region where the first matching means has successfully matched, and a position of a feature region where the second matching means has successfully matched and acquiring means for acquiring the position and orientation of the imaging device based on the position of the feature region. Effect of the Invention

[0007] According to the present invention, it is possible to obtain the position and orientation stably and with high accuracy. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 illustrates an example of a functional configuration of an information processing apparatus according to the first embodiment; [Diagram 2] FIG. 1 illustrates an example of a hardware configuration of an information processing apparatus according to a first embodiment. [Diagram 3] 4 is a flowchart showing a flow of information processing according to the first embodiment. [Figure 4] FIG. 13 is a diagram showing an example of a GUI used for setting a result of association success or failure and threshold values, etc. [Diagram 5] FIG. 11 illustrates an example of a functional configuration of an information processing apparatus according to a second embodiment. [Figure 6] 10 is a flowchart showing a flow of information processing according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the present invention. Although the embodiments describe a number of features, not all of these features are essential to the present invention, and the features may be combined in any manner. Furthermore, in the accompanying drawings, the same or similar configurations or processes are given the same reference numbers, and duplicated descriptions are omitted.

[0010] <First embodiment> As a first embodiment, an information processing device that matches feature regions between images input in time series from an imaging device (hereinafter referred to as a camera) and acquires the position and orientation of the camera will be described. In this embodiment, the position and orientation acquired by the information processing device is used, for example, to control a mobile device. In this embodiment, a mobile robot is taken as an example of the mobile device. Note that a method for controlling a mobile device such as a mobile robot based on the position and orientation of a camera is publicly known, and therefore a detailed description thereof will be omitted.

[0011] The information processing device of this embodiment estimates the position and orientation of the camera for each time on the time series by associating feature regions between images input from the camera in a time series (captured by the camera) and tracking the positions of the associated feature regions. In this embodiment, among the images input from the camera in a time series, an image input at a first time on the time series is defined as a first image, and an image input at a second time on the time series after the first time is defined as a second image. In this embodiment, the feature region of an image is an image region having a geometric structure such as a corner of a subject captured in the image, and in the following description, the feature region is called a feature point. In this embodiment, the information on the position and orientation of the camera is represented by six parameters, including three parameters representing the position of the camera in world coordinates defined in the real space and three parameters representing the orientation of the camera. For example, the three parameters representing the position of the camera are represented by xyz coordinates, and the three parameters representing the orientation of the camera are represented by the rotation angles of the Yaw axis, the Roll axis, and the Pitch axis.

[0012] In the information processing device of this embodiment, when the luminance of an image input from a camera changes due to, for example, a change in the lighting conditions of the environment or a change in the exposure of the camera, and matching of the positions of the image feature points fails, matching of the feature points is performed using another matching method. In this embodiment, the image feature points are first matched by a predetermined first matching process, and when matching of the feature point positions fails in the first matching process, a second matching process is performed by a method different from the first matching process. In this way, the information processing device of this embodiment performs matching of the feature points using a plurality of matching methods in a complementary manner, thereby increasing the number of correctly matched feature points (the number of matching of the feature points). As a result, the information processing device of this embodiment can stably acquire the camera position and orientation with high accuracy even if the luminance of the input image changes.

[0013] FIG. 1 is a functional block diagram showing the configuration of a control system including an information processing device 1 according to the first embodiment, an imaging device 10, and a robot control device 100 that controls a mobile robot. The information processing device 1 calculates the position and orientation of a camera based on an image captured and input by a camera serving as an imaging device 10, and outputs the position and orientation information to a robot control device 100 built into or connected to a mobile robot (not shown). The robot control device 100 controls the mobile robot based on the position and orientation information.

[0014] The imaging device 10 is a stereo camera. In this embodiment, the left camera constituting the stereo camera is called the left camera, and the right camera is called the right camera. In this embodiment, unless otherwise specified, an example will be described in which the information processing device 1 calculates the position and orientation of the camera (imaging device 10) based on an image captured by, for example, the left camera out of the left and right cameras of the stereo camera. Of course, the information processing device 1 may calculate the position and orientation of the camera based on an image captured by the right camera. In this embodiment, the image captured by the left camera is called image L, and the image captured by the right camera is called image R, and both the left and right images are collectively called stereo images. In this embodiment, when specifying the time when image L was captured, for example, image L captured at time t is expressed as image L(t).

[0015] As shown in FIG. 1, the information processing device 1 includes an image input unit 11, a first association unit 12, a first determination unit 13, a second association unit 14, and a calculation unit 15. The image input unit 11 is connected to a camera which is an image capture device 10, and acquires stereo images captured by the camera in time series at, for example, 60 frames per second in time series order, and outputs the images to the first association unit 12 and the second association unit 14. Here, it is assumed that the image capture device 10 acquires monochrome stereo images.

[0016] The first matching unit 12 detects feature points from a first image at a first time and a second image at a second time different from the first time, which are input to the image input unit 11 in chronological order, and performs a process of matching identical feature points between the first image and the second image. In the case of the example of this embodiment, the first matching unit 12 performs a first matching process of matching identical feature points between an image L(t) at a time t and an image L(t-1) at a time t-1, which are input in chronological order. Although the details will be described later, the first matching unit 12 performs a process of matching the positions of feature points in the first image with the positions of corresponding feature points on the second image. Then, the result of the first matching process is output to the first determination unit 13 and the calculation unit 15.

[0017] The first determination unit 13 determines whether the feature points associated by the first matching unit 12 between image L(t) and image L(t-1) are the same feature points, that is, performs a success / failure determination as to whether the first matching process has been successful or unsuccessful. As will be described in detail later, the first determination unit 13 determines whether the positions (corresponding positions) at which the first matching unit 12 associated the feature points between the first image and the second image are correct. Then, the first determination unit 13 outputs a first success / failure determination result indicating the result of the success / failure determination of the first matching process to the second matching unit 14.

[0018] The second matching unit 14 executes a second matching process based on the first success / failure determination result input from the first determination unit 13. For example, when the first success / failure determination result indicates a failure of matching, the second matching unit 14 executes a second matching process using a method different from the first matching process to match feature points that have failed in the first matching process. Details of the second matching process executed by the second matching unit 14 will be described later. Then, the result of the second matching process by the second matching unit 14 is output to the calculation unit 15.

[0019] The calculation unit 15 calculates the position and orientation of the camera based on the positional relationship of the feature points between the image L(t) and the image L(t-1) for which the first matching process has been successful or the positional relationship of the feature points when the second matching process has been performed. Details of the position and orientation calculation process in the calculation unit 15 will be described later.

[0020] Fig. 2 is a diagram showing an example of a hardware configuration of the information processing device 1 according to the present embodiment. As shown in Fig. 2, the information processing device 1 has, as its hardware configuration, a CPU 21, a ROM 22, a RAM 23, an external memory 24, an input unit 25, a display unit 26, a communication I / F 27, an I / O 28, and a system bus 29.

[0021] The CPU 21 controls various devices connected to the system bus 29 . The ROM 22 stores a BIOS program and a boot program. The RAM 23 is used as a main storage device for the CPU 21. The external memory 24 stores various programs including the information processing program according to this embodiment executed by the CPU 21 of the information processing device 1.

[0022] The input unit 25 performs processes such as converting information input by a user via a keyboard, mouse, etc. into information that can be used internally. The display unit 26 generates information such as images to be displayed on a display device according to instructions from the CPU 21. The display device may be a liquid crystal display device, a projector, an LED indicator, or the like, and any type of device may be used.

[0023] The communication I / F 27 is an interface unit that performs information communication via a network. The interface standard may be a network communication standard such as Ethernet (registered trademark), or may be of any type such as serial communication such as USB, wireless communication, etc. The position and orientation information calculated by the information processing device 1 is output to a robot control device 100 built into or connected to the mobile robot via the communication I / F 27. The I / O 28 is connected to a camera that is the imaging device 10, and receives as input a stereo image captured by the camera.

[0024] Fig. 3 is a flowchart showing the flow of information processing performed in the information processing device 1 according to the first embodiment. The processing of each step shown in the flowchart in Fig. 3 is realized, for example, by the CPU 21 executing the information processing program according to this embodiment and operating as each functional unit of the functional block in Fig. 1.

[0025] First, in step S11, the CPU 21 initializes the system. That is, the CPU 21 reads the information processing program according to this embodiment from the external memory 24, and makes the information processing device 1 operable. The CPU 21 also reads various setting parameters from the external memory 24 into the RAM 23 as necessary. The various setting parameters include camera parameters. The camera parameters are assumed to have been calibrated in advance, for example, by Zhang's method. The Zhang method is disclosed in Z. Zhang, "A flexible new technique for camera calibration, Trans. PAMI2000. In this embodiment, the relative position and orientation between the left and right cameras of the stereo camera (external parameters of the cameras) are assumed to have been calibrated in advance by photographing a pattern whose three-dimensional shape is known and calculating the relative position based on the pattern.

[0026] Next, in step S12, the image input unit 11 acquires stereo images captured by the camera in a time series. At this time, the image input unit 11 also performs a process of converting each of the stereo images inputted from the camera in a time series into an image from which the influence of lens distortion has been removed, using the camera parameters.

[0027] Next, in step S13, the first matching unit 12 performs a first matching process to match feature points between a first image and a second image input in time series from the camera via the image input unit 11. For example, the first matching unit 12 searches for the position of a feature point detected from an image L(t-1) input from the camera at a first time t-1 in an image L(t) input at a second time t. Then, the first matching unit 12 performs a position matching process to match the position of the feature point in the image L(t-1) with the position of the same feature point in the image L(t).

[0028] Specifically, the first matching unit 12 first detects feature points from the image L(t-1) at the first time t-1 input to the image input unit 11. For example, the method of Shi et al. (J. Shi and C. Tomasi, Good Features to Track, Proc. CVPR1994) can be used to detect feature points. Next, the first matching unit 12 searches for where the feature points detected on the image L(t-1) are located in the image L(t) at the second time t. In this case, the method of searching for feature points includes the KLT method of Tomasi et al. The KLT method of Tomasi et al. is disclosed in C. Tomasi and T. Kanade, "Detection and Tracking of Point Features", Carnegie Mellon University Technical Report CMU-CS-91-132, 1991. In this way, the first matching unit 12 matches feature points between a first image at a first time and a second image at a second time, thereby obtaining a correspondence relationship between feature point positions between the two images at the first and second times.

[0029] In this embodiment, the first matching unit 12 also performs a matching process of feature points between stereo images, and calculates depth information of each feature point relative to the camera using the matched feature points between the stereo images according to the principle of triangulation. Specifically, the first matching unit 12 searches for corresponding feature points between stereo images by a block matching method that scans the right camera image R(t-1) using a small patch around the feature point detected on the left camera image L(t-1) as a reference. The first matching unit 12 then converts the positions of the corresponding feature points between stereo images into depth values ​​in the camera coordinate system based on the external parameters of the camera, and holds them.

[0030] Next, in step S14, the first determination unit 13 determines whether the result of the first association process by the first association unit 12 is successful or not. For example, the first determination unit 13 determines whether the first association process by the first association unit 12 is successful or not based on the distance between the predicted position of the feature point calculated by the calculation unit 15 based on the history of the calculated position and orientation of the camera and the feature point position associated in step S13.

[0031] Specifically, the first determination unit 13 first calculates the three-dimensional position of the feature point relative to the camera at time t-1 based on the feature point position and the depth value associated in step S13. Next, the first determination unit 13 extrapolates the position and orientation (position and orientation history) calculated based on the images at time t-2 and time t-1 to the three-dimensional position of the feature point calculated for time t-1, and predicts the position and orientation of the camera at time t. Furthermore, the first determination unit 13 projects the feature point onto the image L(t) that is to be captured by the left camera at the predicted time t, and calculates the predicted position of the feature point. Then, the first determination unit 13 calculates the Euclidean distance between the calculated predicted position and the feature point position calculated in the first association process in step S13, and if the distance is equal to or greater than a predetermined threshold, determines that the first association process has failed. In this embodiment, the first determination unit 13 determines that the matching process of the feature point positions between the first image L(t-1) and the second image L(t) has failed if the number of feature points for which the matching is determined to be successful is equal to or less than a predetermined threshold number. In this embodiment, if the first determination unit 13 determines that the matching process is successful, the processing of the information processing device 1 proceeds to step S16 by the calculation unit 15, whereas if it determines that the matching process has failed, the processing proceeds to step S15 by the second matching unit 14.

[0032] When the process proceeds to step S15, the second matching unit 14 matches the feature points for which the matching process has been determined to have failed in step S14 using a second matching process that is different from the first matching process in step S13. In this embodiment, in order to match the feature points even if there is a luminance change between images taken at different times, the second matching unit 14 performs template matching after subtracting the average luminance of the image patches in the second matching process.

[0033] Specifically, the second matching unit 14 first sets the position of the feature point detected from the image L(t-1) at time t-1 as an initial value, and sets a predetermined range including the position of the initial value in the image L(t) at time t as a search range. Next, the second matching unit 14 calculates the average brightness value of small patches around the feature point detected from the image L(t-1) at time t-1, and subtracts the average brightness value from each pixel value of the small patch to create a reference patch. Next, the second matching unit 14 sequentially generates small patches so as to scan the search range set as described above on the image L(t), and generates a search patch by subtracting the average brightness for each of the sequentially generated small patches. Furthermore, the second matching unit 14 calculates the sum of squares of the difference between the reference patch and the search patch (Zero-mean Sum of Squared Distance: ZSSD value), and sets it as the error value of the search patch. Then, the second matching unit 14 sets the coordinates in the search range where the error value is the smallest as the position of the corresponding feature point. After the process of step S15, the process of the information processing device 1 proceeds to step S17.

[0034] Also, when the process proceeds to step S16, the calculation unit 15 calculates the position and orientation of the camera based on the correspondence and depth value of the feature points associated in step S13 or step S15 among the feature points of the images at time t-1 and time t. That is, the calculation unit 15 calculates the position and orientation of the camera based on the feature points associated in step S13 and determined to be successfully associated in step S14, or the correspondence and depth value of the feature points associated in step S15. In the case of this embodiment, the calculation unit 15 calculates the position and orientation of the camera so that an error is minimized when the three-dimensional position of the feature point in the image L(t-1) at time t-1 is projected onto the image L(t) at time t. Specifically, the calculation unit 15 calculates the position and orientation of the camera by solving a so-called PNP problem (Perspective-n-Pont Problem). Then, the information on the position and orientation of the camera calculated by the calculation unit 15 is sent to the robot control device 100. After the process of step S16, the process of the information processing device 1 proceeds to step S17.

[0035] When the process proceeds to step S17, the CPU 21 judges whether or not to end the process. In this embodiment, when a mobile robot (not shown) arrives at a destination or assumes a target posture, the robot control device 100 sends a command to end the system to the information processing device 1. If the end command is input, the CPU 21 of the information processing device 1 ends the process of the flowchart in Fig. 3, and if not, returns the process to step S12 to continue the process of acquiring the position and posture.

[0036] <Effects of the First Embodiment> As described above, in the first embodiment, the success or failure of the first matching process is determined, and feature points for which matching is determined to have failed are matched by a second matching process different from the first matching process. As a result, according to the second embodiment, even if the luminance of the image changes, the number of matching feature points can be increased, and the position and orientation of the camera can be calculated stably with high accuracy.

[0037] <Modification of the first embodiment> In this embodiment, the imaging device 10 is a stereo camera that captures monochrome images, but it may be a stereo camera that captures color images. The imaging device 10 is not limited to a stereo camera, and may be any camera, such as a monocular camera or a depth camera. When a monocular camera is used as the imaging device 10, a depth value is required to calculate the position and orientation. In this case, the first association unit 12 and the second association unit 14 calculate the depth from the movement of the monocular camera. An example of a method for calculating the depth from the movement of the monocular camera is the method of Klein et al. The method of Klein et al. is disclosed in G. Klein and D. Murray, "Parallel tracking and mapping for small AR workspaces," Proc. ISMAR, 2007.

[0038] In this embodiment, the KLT method is used as the matching method in the first matching unit 12, and the template matching method using the ZSSD value is used as the matching method in the second matching unit 14. Here, the method of the second matching process performed by the second matching unit 14 may be another method as long as it can match the same feature points between two different images. For example, a method of cutting out a small patch of an image and searching by template matching may be used, or a method of performing feature point matching using features such as SIFT features and ORB features may be used. In addition, in the second matching process, for example, the Gain robust KLT feature Tracks method of Bergmann et al., which estimates the exposure of the camera and performs brightness correction, may be used. The Gain robust KLT feature Tracks method is disclosed in P. Bergmann. et.al, Online Photometric Calibration of Auto Exposure Video for Realtime Visual Odometry and SLAM, In Proc. RA-L2018. Also, for example, if the camera is configured to acquire exposure and gain values, or the camera exposure can be estimated by the method of Bergmann et al., a pre-processing may be performed to correct the images to have the same exposure and gain before the second matching process is performed. If the exposure and gain cannot be acquired, the pre-processing may be performed by subtracting the average or median of the luminance or flattening the histogram for the entire image or for each local region (small region) to reduce the luminance difference of the image before the second matching process is performed. That is, the second matching process may be performed after correcting the luminance values ​​of the entire image or small regions so that at least one of the average luminance difference, median difference, histogram difference, exposure difference, and sensitivity difference of the imaging device of the entire image or small regions is reduced. In the case of this embodiment, the correction of the luminance values ​​in the entire image or small regions is performed during the second matching process when the first matching process fails.Each of these methods applicable as the second matching process may be used in the first matching process, but in this case too, different methods are used for the first matching process and the second matching process.

[0039] The information processing device may also be configured to perform multiple matching processes, such as matching feature points for which the second matching process has failed in a third matching process, and matching feature points for which the third matching process has failed in a fourth point matching process. In this case, the information processing device has multiple matching units, such as a third matching unit that performs the third matching process and a fourth matching unit that performs the fourth point matching process. Alternatively, the second matching unit may be configured to change the matching process in sequence, such as the third matching process and the fourth matching process, when the second matching process has failed. In this way, by configuring the device to be able to perform multiple matching processes, it becomes possible to match feature points more stably.

[0040] In addition, in this embodiment, the first determination unit 13 determines whether the first matching process has been successful based on the predicted positions of the feature points obtained by projecting three-dimensional feature point positions onto an image at a camera position predicted from the history of the camera position and orientation, but the method of determining whether the first determination unit 13 has been successful is not limited to this example. That is, the process of determining whether the first determination unit 13 has been successful may use other methods as long as it is possible to detect whether the matching of feature points has failed (succeeded).

[0041] For example, the first determination unit 13 may hold the feature point positions on the image at time t-2 and time t-1 as a history, calculate the moving speed of the feature point on the image from time t-2 to time t-1, and predict the feature point position on the image at time t based on the moving speed. For example, the first determination unit 13 may hold the history of the feature point positions on the image, calculate the average movement vector of the feature point based on the held positions, and determine that the matching has failed if the difference between the average movement vector and the movement amount of the feature point is a predetermined amount or more. For example, the first determination unit 13 may calculate a geometric correspondence between images such as an F matrix from the correspondence of the feature point positions, and determine that the matching has failed for inconsistent feature points whose residual of the geometric correspondence is a predetermined threshold or more. For example, the first determination unit 13 may calculate the relative position and orientation of the camera once from the correspondence of the feature points, and determine that the matching has failed for feature points whose reprojection error value is greater than a predetermined value. Furthermore, the first determination unit 13 may determine that the matching has failed if, for example, the amount of variation in the distribution of feature points is equal to or greater than a predetermined amount. For example, the first determination unit 13 determines that the matching has failed if the number of matched feature points decreases or increases by a predetermined number or more between the images at the two times, or if the average position of the matched feature points changes by a predetermined distance or more between the images at the two times.

[0042] The first determination unit 13 may be configured to observe a luminance change of an image and estimate the success or failure of the matching process of the feature point positions based on the luminance change. For example, the first determination unit 13 estimates (determines) that matching of the feature point positions will fail if the average luminance change amount of the entire image or a local region is equal to or greater than a predetermined amount, or if the change amount of the histogram is equal to or greater than a predetermined amount. The first determination unit 13 can also obtain, for example, the exposure and gain values ​​of the camera, and estimate (determine) that matching of the feature points will fail if the fluctuation amount of these values ​​is greater than a predetermined amount.

[0043] Although the information processing device of the first embodiment detects feature points in an image at time t-1 and searches for corresponding feature points in an image at time t, feature points do not have to be detected from an image at a certain time. The information processing device may be configured to track feature point positions associated at each time in a chronological order, for example. That is, the information processing device may be configured to track feature points detected from an image at time t by sequentially associating them with an image at time t+1, an image at time t+2, an image at time t+3, and so on. With such a configuration, it is not necessary to detect feature points in all images, and therefore the processing speed is improved.

[0044] In the present embodiment, the first matching unit 12 and the second matching unit 14 both perform feature point matching between images at a first time and a second time, but the first and second matching units may be configured to match feature points between images at different times. That is, the first matching unit 12 may perform feature point matching between images at time t-2 and time t, and the second matching unit 14 may perform feature point matching between images at time t-1 and time t. With such a configuration, feature points are matched between images at a plurality of times, the number of feature points that are matched increases, and it becomes possible to stably calculate the position and orientation with high accuracy.

[0045] The information processing device may also be configured to hold three-dimensional map information including three-dimensional geometric information that indicates the three-dimensional coordinates of feature points detected from an image, and to hold the feature points in the three-dimensional map in association with the feature points of the input image. In this way, it is possible to reduce accumulated errors in position and orientation calculation compared to when position and orientation calculation is performed based on the correspondence between feature points of previous and next images.

[0046] The information processing device can also apply the feature point matching method of the present embodiment to matching feature points stored in a map, for example. In this way, the number of feature point matchings in the map increases, the map accuracy during map optimization described in the method of Klein et al. is improved, and the accuracy of position and orientation calculation can be improved.

[0047] In this embodiment, an example is assumed in which the movement, posture, etc. of a mobile robot, which is an example of a mobile device, is controlled based on the position and posture calculated by the calculation unit 15. Examples of mobile robots include automated guided vehicles (AGVs), autonomous mobile robots (AMRs), drones, and self-driving cars. In addition, the mobile robot is not limited to one that moves autonomously, and may be configured as a measurement device that measures the position and posture of a mobile device equipped with a moving mechanism that carries the information processing device of this embodiment.

[0048] In the present embodiment, the position and orientation calculated by the calculation unit 15 is applied to the control of a moving device, but the application of the position and orientation is not limited to a moving device as long as the device uses the position and orientation. For example, the information processing device of the present embodiment can be used as a position and orientation estimation device when a virtual object such as mixed reality (MR) or augmented reality (AR) is synthesized with a real image and presented. The information processing device can also be used as a device that inputs the current position and orientation in a virtual reality (VR) that moves in a virtual space based on the position and orientation of the real space. In addition, the information processing device can also be used as a device that measures the tip position of a robot hand equipped with a camera.

[0049] Furthermore, the information processing device of this embodiment may be configured to include an input display unit having a function of displaying an input image via the image input unit 11, a function of displaying the result of matching of feature points, and for adjusting a threshold value used to determine whether the matching is successful or not. FIG. 4 is a diagram showing an example in which the function of the input display unit is configured as a GUI (Graphical User Interface) displayed on a display having a touch panel.

[0050] In FIG. 4, a window 40 shows an example of a window screen on which a GUI that realizes the function of the input display unit is displayed. A camera image 41 is an example of a captured image inputted via the image input unit 11. A circle 42 in the camera image 41 represents a feature point that the first matching unit 12 has successfully matched among the feature points detected from the image. A triangle 43 in the camera image 41 represents a feature point that the second matching unit 14 has matched among the feature points detected from the image. A cross 44 in the camera image 41 represents a feature point that has failed to be matched among the feature points detected from the image. A character string 45 in the camera image 41 is a message indicating that the first determination unit 13 has detected a luminance change in the image.

[0051] An input form 46 in the window 40 is an area for inputting a threshold value for the error between the feature point position predicted by the first determination unit 13 and the feature point position associated by the first association unit 12. If the error between the predicted feature point position and the associated feature point position is equal to or greater than the threshold value inputted in the input form 46, the first determination unit 13 determines that the feature point association has failed. When a numerical value is inputted into the input form 46 via the input unit 25 in Fig. 2 from a keyboard, mouse or the like, the information processing device sets a threshold value according to that numerical value. By using such a UI, the user can adjust the threshold value for determining whether the association has been successful.

[0052] An input form 47 in the window 40 is an area where a threshold value for the exposure variation value of the camera is input. When the exposure variation value of the camera is equal to or greater than the threshold value input in the input form 47, the first determination unit 13 determines that a change in the brightness of the image has occurred. When a numerical value is input in the input form 47 via the input unit 25 in FIG. 2 from a keyboard, mouse, or the like, the information processing device sets a threshold value according to that numerical value. By using such a UI, the user can recognize that a brightness change has occurred and can adjust the threshold value for detecting the brightness change.

[0053] Second Embodiment In the first embodiment, during the second matching process, first, an initial value for matching is predicted, and the appearance of the image is predicted based on the position and orientation at the time of the matching process to calculate feature amounts. In the second embodiment, the second matching process is further terminated after a predetermined time, and feature points that are incorrectly matched in the second matching process are removed. As a result, the second embodiment achieves feature point matching processing with high accuracy in a short time compared to the first embodiment. Information processing according to the second embodiment will be described below.

[0054] In the second embodiment, as in the first embodiment, a stereo camera consisting of a left camera and a right camera is used as the imaging device 10, and the position and orientation of the left camera is the position and orientation of the imaging device unless otherwise specified. Also, as in the first embodiment, both left and right images, an image L captured by the left camera and an image R captured by the right camera, are called stereo images. Also, as in the first embodiment, an image captured at time t, for example, is represented as image L(t).

[0055] 5 is a diagram showing the functional configuration of a control system having an information processing device 5, an imaging device 10, and a robot control device 100 according to the second embodiment. The information processing device 5 of the second embodiment has a prediction unit 51, a correction unit 52, a processing stop unit 53, and a second determination unit 54 in addition to the configuration described in the first embodiment. In the case of the second embodiment, the first success / failure determination result by the first determination unit 13 is also input to the prediction unit 51 and the correction unit 52.

[0056] When the first success / failure determination result indicates failure, the prediction unit 51 acquires the predicted positions of the feature points for the second image based on at least one of the movement amount of the camera and the movement amount of the feature points. For example, when the second image is an image L(t) at time t, if the first success / failure determination result is determined to be failure, the prediction unit 51 first predicts the position and orientation of the camera at time t based on the history of the position and orientation calculated by the calculation unit 15. Furthermore, the prediction unit 51 predicts the positions of the feature points in the image L(t) at time t based on the position and orientation of the camera predicted at time t. Then, the prediction result by the prediction unit 51 is output to the correction unit 52. Details of the prediction process performed by the prediction unit 51 will be described later.

[0057] If the first success / failure determination result indicates failure, the correction unit 52 performs correction based on the position and orientation calculated by the calculation unit 15 so as to reduce the feature amount at the feature point of the first image. If the second image is an image L(t) at time t, the correction unit 52 first predicts the position and orientation of the camera at time t based on the history of the position and orientation calculated by the calculation unit 15. Furthermore, the correction unit 52 generates, from the image L(t-1), a small patch around the feature point where the feature point will be observed on the image L(t) captured at time t, based on the position and orientation of the camera predicted at time t. Then, the correction unit 52 outputs the generated small patch to the second association unit 14.

[0058] The processing stop unit 53 measures the processing time of the second matching unit 14, and when the processing time exceeds a predetermined time corresponding to the processing rate (e.g., 1 / 60th of a second) when the calculation unit 15 calculates the position and orientation, outputs a processing stop signal to the second matching unit 14.

[0059] The second matching unit 14 according to the second embodiment sets the predicted position predicted by the prediction unit 51 as an initial position, and calculates a corresponding position on the second image in the time series with respect to the initial position. The second matching unit 14 according to the second embodiment calculates a corresponding position on the second image with respect to the feature point based on the feature amount corrected by the correction unit 52 for the feature point. Taking the first image L(t-1) and the second image L(t) as an example, the second matching unit 14 performs a second matching process using the images L(t) and L(t-1) based on the feature point position predicted by the prediction unit 51 and the small patch created by the correction unit 52. The second matching unit 14 then matches the feature points that are determined to be the same by the second matching process, and outputs the result of the matching process to the second determination unit 54. Details of the matching process performed by the second matching unit 14 in the second embodiment will be described later. Furthermore, in the second embodiment, when a processing stop signal is input from the processing stop unit 53, the second matching unit 14 discontinues the matching process in accordance with the input even if matching processing has not been completed for all feature points.

[0060] The second determination unit 54 determines whether the matching process of feature points between images executed by the second matching unit 14 has been successful, that is, whether the feature points matched by the second matching process are the same feature points. If the second determination unit 54 determines that the second matching process has been successful, that is, if it determines that the feature points between the images are the same, it outputs the correspondence of the feature points to the calculation unit 15.

[0061] Calculation unit 15 in the second embodiment calculates the position and orientation of the camera using feature points successfully matched by first matching unit 12 and feature points successfully matched by second matching unit 14. The process of calculating the position and orientation of the camera by calculation unit 15 is similar to the example in the first embodiment, and therefore description thereof will be omitted.

[0062] FIG. 6 is a flowchart showing a process flow in the information processing device 5 of the second embodiment. As in the flowchart of FIG. 3 described above, the process of each step shown in the flowchart of FIG. 6 is realized by, for example, the CPU 21 executing the information processing program according to the second embodiment and operating as each functional unit of the functional block of FIG. 1. The flowchart shown in FIG. 6 has steps S21, S22, S23, and S24 in addition to the process steps described in the first embodiment. Step S21 is a process in which the prediction unit 51 predicts the feature point position, and step S22 is a process in which the correction unit 52 corrects the feature amount. Step S23 is a process in which the process stop unit 53 determines whether to stop the process, and step S24 is a process in which the second determination unit 54 determines whether the second association process is successful.

[0063] In the second embodiment, if it is determined in step S14 that the first association process has failed, the process proceeds to step S21. In step S21, the prediction unit 51 predicts the position and orientation of the camera based on the history of the position and orientation calculated by the calculation unit 15, and predicts the coordinates on the image of the feature point determined to have failed in step S14. To explain this with a specific example, the prediction unit 51 first calculates the three-dimensional position of the feature point relative to the camera at time t-1 based on the feature point position and depth value detected in step S13. Next, the prediction unit 51 predicts the position and orientation of the camera at time t based on the position and orientation calculated by the calculation unit 15 at times t-2 and t-1 (i.e., the history of the position and orientation). Then, the prediction unit 51 projects the feature point onto the image L(t) of the predicted position and orientation of the camera at time t, and calculates the predicted position of the feature point. This calculated predicted position is used as an initial value for the second association process in step S15.

[0064] Next, in step S22, the correction unit 52 creates a small patch around the feature point determined to have failed in step S14 at which the feature point will be observed in the image L(t) based on the history of the position and orientation calculated by the calculation unit 15. Specifically, the correction unit 52 first predicts the position and orientation of the camera at time t based on the position and orientation (history of the position and orientation) of the camera calculated by the calculation unit 15 for time t-2 and time t-1. Next, the correction unit 52 calculates the relative position and orientation between the position and orientation at time t-1 and the position and orientation at time t. Then, based on the calculated relative position and orientation, the correction unit 52 performs homography transformation on the small patch in the image L(t-1) at time t-1 to create a small patch around the feature point at which the feature point will be observed in the image L(t).

[0065] Next, in the case of the second embodiment, in step S15, the second matching unit 14 sets the predicted position of the feature point calculated in step S21 as an initial value, and searches for the feature point position on the image L(t) by referring to the small patch around the feature point calculated in step S22. Specifically, the second matching unit 14 sets a predetermined area around the predicted position of the feature point calculated in step S21 as a search window range, and performs template matching by using the small patch around the feature point calculated in step S22 as a reference patch. Note that in this case, the second matching unit 14 subtracts the average luminance of the small patch as described in the first embodiment in order to deal with the change in luminance between the image L(t-1) and the image L(t), and performs template matching. Then, the second matching unit 14 sets the coordinates at which the ZSSD value is minimum within the search window range as the corresponding position of the feature point.

[0066] Next, in step S23, the process stopping unit 53 stops (terminates) the process so that the processing time of the second matching process in step S15 does not exceed a predetermined time. Specifically, if the processing time of the second matching process exceeds a predetermined time (e.g., 8 ms) set based on a position and orientation calculation rate specified in advance, the process stopping unit 53 terminates the matching process of the feature points thereafter. Note that, if the processing time of the second matching process does not exceed the predetermined time, the process of the information processing device 5 returns to step S15, and if the processing time exceeds the predetermined time, the process proceeds to the next step S24.

[0067] When the process proceeds to step S24, the second determination unit 54 determines whether the feature point positions calculated in step S15 are valid, that is, determines whether the second matching process is successful. Specifically, the second determination unit 54 calculates the Euclidean distance between the coordinates of the feature points predicted in step S21 and the feature point positions calculated in step S15. If the calculated Euclidean distance is equal to or greater than the threshold distance, the second determination unit 54 determines that the second matching process has failed. On the other hand, if the calculated Euclidean distance is less than the threshold distance and the second matching process is determined to be successful, the second determination unit 54 sends the coordinates of the feature points determined to be successful to the calculation unit 15. As a result, in step S16, the calculation unit 15 calculates the position and orientation of the camera based on the coordinates of the feature points.

[0068] <Effects of the Second Embodiment> As described above, in the second embodiment, before the second matching process is performed, the position and orientation of the camera are predicted to predict the positions of feature points, and the feature amounts of the predicted feature points are corrected to match the predicted camera position and orientation. Also, in the second embodiment, the second matching process is terminated after a predetermined time, and erroneous feature point matching relationships that have failed in the second matching process are removed. As a result, according to the second embodiment, even if feature point matching fails due to a change in image luminance, for example, it is possible to increase the number of feature point associations, and it becomes possible to stably calculate the position and orientation with high accuracy and in a short time.

[0069] <Modification of the second embodiment> In the case of the second embodiment, in the process of predicting the position of the feature point, the predicted position of the feature point is calculated by projecting the three-dimensional position of the feature point onto the image from the camera position predicted from the history of the position and orientation of the camera, but the present invention is not limited to this example.For example, the information processing device may hold the history of the positions of the feature points on the images, calculate the moving speed from the feature point on the image at time t-2 to the feature point on the image at time t-1, and predict the position of the feature point on the image at time t.

[0070] In the present embodiment, in the feature correction process, patches around feature points are calculated by homography transformation based on a predicted value of the camera position and orientation, but the present invention is not limited to this example. For example, as long as it is possible to correct changes in image feature amounts due to changes in the camera position and orientation, it is not limited to homography transformation, and Affine transformation, skew transformation, rotation transformation, etc. may be used. In addition, the information processing device may be equipped with an acceleration sensor, and the image may be rotated based on the sensor output so that the gravity axis coincides, and then patches may be cut out.

[0071] Further, the success / failure determination process of the second determination section 54 in the second embodiment may employ the success / failure determination method described as a modified example of the first determination section 13 in the first embodiment. In the second embodiment, the second matching process is stopped (aborted) when the processing time of the second matching process exceeds a predetermined time, but the present invention is not limited to this example. For example, any method may be used as long as the output interval of the position and orientation calculation can be maintained at a predetermined value, and the second matching process may be aborted when a predetermined number of feature points have been processed. In this case, the information processing device may be configured to perform a process of determining whether a predetermined number of feature points have been processed after step S24, for example, instead of step S23, and to stop the second matching process when the matching process of the predetermined number of feature points is successful.

[0072] In the second embodiment, an example has been described in which feature point position prediction and feature amount correction are performed for all feature points in advance, and then the processing is stopped each time the second matching processing is performed for one feature point, but the present invention is not limited to this example. For example, the information processing device may be configured to perform the feature point position prediction processing in step S21, the feature amount correction processing in step S22, and the second matching processing in step S15 for each feature point, and determine whether to stop the processing. In this configuration, it is no longer necessary to perform the feature point position prediction processing and feature amount correction processing for the feature point for which the processing has been stopped, and it becomes possible to perform the second matching processing for more feature points within a specified time.

[0073] Although the information processing device of the second embodiment has been shown as having an example including a prediction unit 51, a correction unit 52, a processing stop unit 53, and a second determination unit 54, it is not necessary to have all of these, and a configuration including only some of them may be used. Even if the information processing device is equipped with all of the prediction unit 51 to the second determination unit 54, it may be configured to selectively use only some of them, for example. For example, the information processing device may be configured to predict the position of a feature point or correct a feature amount only when the amount of movement of the camera is greater than a predetermined amount.

[0074] The information processing device may be configured to present the GUI described in the modified example of the first embodiment. When presenting a GUI in the second embodiment, the information processing device may be configured to present all or part of the information calculated by the prediction unit 51, the correction unit 52, the processing stop unit 53, and the second determination unit 54. Furthermore, the information processing device may be configured so that the parameters as described in the first embodiment can be adjusted via a GUI. In this case, it becomes possible to stably calculate the position and orientation of the camera with high accuracy even if a change in luminance occurs.

[0075] Furthermore, the information processing device may display a line segment or the like connecting the predicted positions of the feature points predicted by the prediction unit 51 and the positions of the feature points calculated by the second association unit 14. When such a display is performed, the user can easily check the error from the predicted position of each feature point. The information processing device may also display the feature amount corrected by the correction unit 52, i.e., small patches around the feature points in this embodiment. In this case, the user can easily check whether the feature points are associated with the correct feature points. Furthermore, the information processing device may display the number of feature points executed before the process is stopped (aborted) by the process stopping unit 53. In this case, the user can know the number of feature points executed before the process is stopped. Furthermore, the information processing device may display feature points for which the second determination unit 54 has determined that the matching process has failed. In this case, the user can understand which points in the image were not used in calculating the position and orientation.

[0076] <Third embodiment> In the third embodiment, a configuration will be described in which the moving speed or the speed of attitude change of the mobile robot is reduced to realize stable control when the brightness of an image changes due to changes in the environmental lighting conditions or the exposure of the camera. In the case of the information processing device of the third embodiment, the calculation unit 15 also has a function as a control value calculation unit that calculates a robot control value that controls the moving speed or the speed of attitude change of the mobile robot. Note that the third embodiment will be described by reusing the functional block diagram of FIG. 1 and the flowchart of FIG. 3 used in the first embodiment.

[0077] In the third embodiment, the calculation unit 15 also receives the success / failure determination result from the first determination unit 13. In addition to the camera position / orientation calculation process described above, the calculation unit 15 in the third embodiment also performs a process of calculating a robot control value for controlling the moving speed or the speed of attitude change of the mobile robot, based on the success / failure determination result from the first determination unit 13. In particular, in this embodiment, when the success / failure determination result indicates failure due to a change in the lighting conditions of the environment or the exposure of the camera changing and thus changing the brightness of the image, the calculation unit 15 calculates and outputs a robot control value for decreasing the moving speed or the speed of attitude change of the mobile robot.

[0078] In the third embodiment, the robot control value output from the calculation unit 15 is sent to the robot control device 100 built in or connected to the mobile robot. The robot control device 100 performs drive control of the motor of the mobile robot (not shown) based on the robot control value from the calculation unit 15. In the present embodiment, the motor controlled based on the robot control value is a motor that moves the mobile robot or a motor that changes the posture of the mobile robot. The robot control device 100 controls the rotation speed of the motor based on the robot control value. This controls the moving speed of the mobile robot, the speed of the movement of the mobile robot, the speed of the movement of the posture change, etc. That is, in this embodiment, when the brightness of the image changes due to a change in the lighting conditions or exposure and the result of the success / failure judgment is failure, the moving speed of the mobile robot is controlled to be decreased, or the speed of the movement of the posture change is controlled to be decreased.

[0079] In the third embodiment, the calculation and output process of the robot control value by the calculation unit 15 based on the success / failure determination result is executed after the position and orientation calculation process of step S16 shown in FIG. 3 is performed, for example. Here, the calculation unit 15 calculates a robot control value for controlling the moving robot by a known method using the position and orientation of the camera. At this time, when the success / failure determination result of the first determination unit 13 indicates a failure of the association process, the calculation unit 15 calculates a robot control value by multiplying the robot control value calculated based on the position and orientation of the camera by a predetermined coefficient (for example, 0.5). Then, the calculation unit 15 outputs the robot control value to the robot control device 100. As a result, the robot control device 100 controls the moving robot based on the robot control value multiplied by the coefficient of 0.5, that is, controls the moving speed and the attitude change operation speed of the moving robot slower than in the control based only on the position and orientation.

[0080] <Effects of the Third Embodiment> According to the third embodiment, when the brightness of an image changes due to changes in lighting conditions or exposure, the moving speed and the speed of posture change of the moving robot are reduced, thereby making it possible to control the moving robot with high accuracy and stable position and posture.

[0081] <Modification of the third embodiment> The robot control value in the third embodiment may be any signal capable of changing the speed and angular velocity of the mobile robot, and is not limited to the signal controlling the rotational speed of the motor described above. For example, the robot control value may be any voltage value or PWM (pulse width modulation) period. For example, if the mobile robot is driven by a pneumatic actuator rather than a motor, the robot control value is a value that controls the air pressure. For example, if the mobile robot is driven by a linear drive using electromagnetism, the robot control value is a value that controls the frequency of the current.

[0082] As for the method of adjusting the robot control value, any method can be used as long as it can reduce the speed and angular velocity of the moving robot when the brightness of the image changes due to changes in the lighting conditions and exposure. For example, the information processing device may be configured to reduce the speed and angular velocity of the moving robot as the number of feature points associated by the second association unit 14 decreases. For example, when the number of associated feature points is 30 or more, the coefficient to be multiplied to the robot control value based on the position and orientation is set to 1, and when the number of associated feature points is less than 30, the coefficient to be multiplied to the robot control value based on the position and orientation is set to the product of the number of associated feature points and 1 / 30. Alternatively, for example, the robot control value based on the position and orientation may be multiplied by a coefficient inversely proportional to the number of associated feature points.

[0083] The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0084] 1: information processing device, 11: image input unit, 12: first matching unit, 13: first determination unit, 14: second matching unit, 15: calculation unit, 51: prediction unit, 52: correction unit, 53: processing stop unit, 54: second determination unit

Claims

1. a first matching means for matching positions of feature regions between a first image and a second image inputted in time series from an imaging device; a second matching means for matching the positions of the characteristic regions between the first image and the second image using a method different from the method used by the first matching means when the first matching means has failed to match the positions of the characteristic regions; an acquisition means for acquiring a position and orientation of the imaging device based on a position of a feature region for which the first association has been successfully performed and a position of a feature region for which the second association has been successfully performed; 13. An information processing device comprising:

2. a storage means for storing three-dimensional map information including geometric information representing three-dimensional coordinates of the positions of the feature regions; The information processing apparatus according to claim 1 , wherein the three-dimensional map information holds information about the characteristic regions.

3. The information processing device according to claim 1 or claim 2, characterized in that the second matching means corrects the luminance values ​​of at least one of the first image and the second image or local regions of the first image and the second image so as to reduce at least one of the average luminance difference, the median difference, the histogram difference, the exposure difference of the imaging device, and the sensitivity difference of the imaging device, and then matches the positions of the characteristic regions.

4. 4. The information processing apparatus according to claim 1, wherein the second matching means matches the positions of the characteristic regions in such a manner that the feature amount of the characteristic regions decreases.

5. a prediction means for acquiring a predicted position of the second image based on at least one of a movement amount of the imaging device and a movement amount of the characteristic area, 5 . The information processing apparatus according to claim 1 , wherein the second matching means matches the positions of the characteristic regions in the second image using the predicted positions as initial positions.

6. a correction unit that corrects a feature amount of the feature region of the first image based on the position and orientation acquired by the acquisition unit, 6. The information processing device according to claim 1, wherein the second matching means performs the position matching using the feature area whose feature amount has been corrected by the correction means and the feature area of ​​the second image.

7. The information processing device according to any one of claims 1 to 6, characterized in that, when the matching of the positions of the characteristic regions fails, the second matching means performs a process of matching the positions of the characteristic regions between the first image and the second image using a method that is further different from the method used when the matching fails.

8. 8. The information processing apparatus according to claim 1, further comprising a determination unit that determines whether the association of the positions of the characteristic regions is successful.

9. 9. The information processing device according to claim 8, wherein the determination means determines that the association has failed when a distance between a position predicted based on an amount of movement of the imaging device or an amount of movement of the feature region and the associated position is equal to or greater than a threshold value.

10. The information processing device according to claim 8, characterized in that the determination means obtains a geometric correspondence between the first image and the second image based on the matched positions, and determines that the matching has failed if a residual of the geometric correspondence is equal to or greater than a predetermined threshold.

11. The information processing device according to claim 8, characterized in that the determination means determines that the matching has failed when at least one of the average luminance difference, median difference, histogram difference, exposure difference of the imaging device, and sensitivity difference of the imaging device of the entire image or local region of the first image and the second image is equal to or greater than a corresponding threshold value.

12. 12. The information processing apparatus according to claim 1, further comprising a stop means for interrupting the association process when a processing time for the association by the second association means exceeds a predetermined time.

13. A control value calculation means for calculating a control value of a moving device, 13. The information processing device according to claim 1, wherein the control value calculation means calculates a control value for reducing at least one of the moving velocity or angular velocity of the moving device when the matching by the first matching means fails.

14. An information processing device according to any one of claims 1 to 13; a control device that controls a moving device based on the position and orientation of the imaging device acquired by the information processing device; A system comprising:

15. An information processing method executed by one or more processors, comprising: a first matching step of matching positions of feature regions between a first image and a second image input in time series from an imaging device; a second matching step of matching the positions of the characteristic regions between the first image and the second image using a method different from the method used in the first matching step when the first matching step has failed to match the positions of the characteristic regions; an acquisition step of acquiring a position and orientation of the image capturing device based on positions of feature regions for which the first matching step has succeeded and positions of feature regions for which the second matching step has succeeded; 13. An information processing method comprising:

16. A program for causing a computer to function as each of the means included in the information processing device according to any one of claims 1 to 14.

Citation Information

Patent Citations

  • Method for tracking moving object on monitoring picture

    JP1994231252A

  • Position / attitude estimation apparatus, position / attitude estimation method, and program

    JP2016091065A

  • Video processing apparatus, video processing method, and program

    JP2017130042A

  • Self-position detection device, moving body device, and self-position detection method

    JP2018009918A

  • Image processing device, imaging device, image processing method, and image processing program

    JP2019185404A