Information processing apparatus, information processing method, and program

Abstract

To enable positioning with satisfactory accuracy to be continued even if a position attitude of an object in a keyframe image is changed.SOLUTION: An information processing apparatus according to an embodiment of the present invention has acquisition means for acquiring an image captured by an imaging device, selection means for selecting a key frame image obtained by capturing a range close to a captured range of the image from a plurality of key frame images previously prepared, and control means for, based on a degree of similarity of the image and the key frame image selected by the selection means, controlling invalidation processing invalidating at least a part of the key frame image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Mixed reality (MR) and augmented reality (AR) are known as technologies that fuse real space with virtual space in real time. These technologies seamlessly fuse real space with a virtual space created by a computer. For example, MR and AR technologies are expected to be applied to various fields, such as assembly support in factories that superimposes the work procedures and wiring status of assembly work on the real world, and are being applied to the real world.

[0003] MR is achieved by precisely aligning and placing computer graphics (CG) objects at any position in real space. As a result of the alignment, when a user looks around through the display, they can see the CG objects that exist in a unique location in real space, and experience it as if the object were actually in that location.

[0004] Conventionally, the most common technique for achieving alignment was to define a spatial coordinate system using markers with distinctive black and white patterns, and to detect markers whose positions were already known by observing them with a camera through image processing. In recent years, there have been techniques for achieving alignment without using known markers, such as Simultaneous Localization (SLO). A method called Surface Layered Mapping (SLAM) has been established. The mechanism of SLAM is explained in Non-Patent Document 1. In SLAM, one or more stationary features are detected in an input camera image, and the position and orientation of the camera are updated by determining the relative position and orientation of the detected features. The features detected from the camera image are points or edges that have a relatively significant change in brightness in the image and form corners. For example, if a cube of a different color from the desk is placed on a plain desk, each vertex of the cube is a feature point, and each contour line of the cube is an edge. Various applications can provide images in which CG is placed at a specific position in real space using the position and orientation of the camera and information on the detected features.

[0005] As the name suggests, SLAM is a technology that performs tracking of the self-position and orientation, and mapping to create a map of the surrounding environment from camera images. Specifically, mapping is a process that creates information called a keyframe when the field of view changes significantly due to a change in the camera's position and orientation. A keyframe is information that is stored in association with a camera image (also called a keyframe image), the position and orientation of the camera when the camera image was captured, and information about the features in the camera image. Keyframe images are used for optimization processes that minimize position and orientation errors when alignment fails and is restored, and when a loop is detected in the camera trajectory. Optimization processes that are performed when a loop is detected are aimed at reducing the accumulated position and orientation errors.

[0006] Non-Patent Document 1 explains pose graph optimization as an example of optimization processing. In pose graph optimization, when a camera trajectory loop is detected due to conditions such as the similarity of keyframe images, the features of the current keyframe and adjacent keyframes and the camera position and orientation are modified based on the graph structure of the keyframes. However, keyframes are created at intervals of about once every few seconds, and each contains image data, which increases memory consumption. Therefore, Non-Patent Document 1 compares, for example, three keyframes, and calculates the image data stored in each keyframe. If more than 90% of the feature points in a keyframe overlap, one keyframe is retained and the others are deemed redundant and deleted.

[0007] Patent Document 1 also discloses a configuration for deleting key frames to organize memory. For example, memory organization is performed when memory usage exceeds a reference capacity, or when a reference time has passed since deleting deletable data. Memory organization is a process for deleting key frames except for one when there are multiple key frames observing the vicinity, or for deleting feature points included in the key frames that have an error larger than a reference error.

[0008] In addition, a moving object captured in a key frame image becomes noise for the key frame. For example, if an image of a person walking in front of a camera is stored as a key frame image, the positioning accuracy may be reduced in the process of referencing the key frame image because the moving object that no longer exists is captured in the image.

[0009] Non-Patent Document 2 discloses a method of estimating the position and orientation by registering keyframe images with excluding moving objects in a normal RGB camera configuration and an RGB-depth camera configuration. The RGB camera uses a method of deep learning to segment objects that may move, such as hands and cars, and extracts areas that should not be stored as keyframe images. In addition, in a configuration with a depth camera, moving objects are detected by a multi-view geometry method, and the area of ​​the detected moving object is combined with the area of ​​the moving object detected by the RGB camera. When storing a keyframe image, the moving object area is painted white, or in a configuration with a depth camera, the corresponding area of ​​the most recent frame is synthesized, making it possible to perform mapping and self-orientation estimation in a state without moving objects. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] Japanese Patent Application Laid-Open No. 2019-159354

Non-Patent Literature

[0011]

Non-Patent Literature 1

Non-Patent Literature 2

Non-Patent Literature 3

Non-Patent Literature 4

[0012] However, if an object captured in a keyframe image moves (the position and orientation of the object changes) after the keyframe is registered, the moved object may cause noise. Specifically, in the method shown in Patent Document 1, if the estimation of the position and orientation of the camera fails (is lost), the position and orientation estimation is performed by comparing the similarity between the current image and a search keyframe image captured at a nearby position and orientation. The estimation is resumed based on the similarity. Whether or not the trajectory of the position and orientation of the camera has generated a loop may be determined based on the similarity between the current image and the search keyframe image. When the current image after the object has moved is compared with the search keyframe image before the object moved, even if the positions and orientations are close to each other, the estimation of the position and orientation may not be resumed or a loop of the trajectory of the position and orientation may not be detected. In an environment in which the position and orientation of an object changes due to the movement of the object, the accuracy of self-location estimation and mapping may be reduced because the keyframe image in which the object before the movement is captured remains.

[0013] An object of the present invention is to continue accurate registration even when the position and orientation of an object captured in a key frame image changes. [Means for solving the problem]

[0014] A first aspect of the present disclosure is an information processing device characterized by having an acquisition means for acquiring an image captured by an imaging device, a selection means for selecting a keyframe image from a plurality of keyframe images prepared in advance, the keyframe image capturing a range approximating the imaging range of the image, and a control means for controlling an invalidation process for invalidating at least a portion of the keyframe image based on the similarity between the image and the keyframe image selected by the selection means.

[0015] A second aspect of the present invention is an information processing method comprising: an acquisition step of acquiring an image captured by an imaging device; a selection step of selecting a keyframe image from a plurality of keyframe images prepared in advance, the keyframe image capturing a range approximating the imaging range of the image; and a control step of controlling an invalidation process of invalidating at least a portion of the keyframe image based on the similarity between the image and the keyframe image selected in the selection step.

[0016] A third aspect of the present invention is a program for causing a computer to function as each of the means of the information processing device described above. Effect of the Invention

[0017] According to the present invention, accurate registration can be continued even if the position and orientation of an object captured in a key frame image changes. [Brief description of the drawings]

[0018] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. [Diagram 2] 1 is a block diagram showing an example of a functional configuration of an information processing device according to a first embodiment. [Diagram 3] 13 is a flowchart illustrating an example of a key frame creation process. [Figure 4] FIG. 11 is a diagram illustrating an example of extraction of feature points. [Diagram 5] FIG. 11 is a diagram illustrating an example of key frame information. [Figure 6]FIG. 11 is a diagram showing an example of information on feature points. [Figure 7] 13 is a flowchart illustrating an example of a key frame update process in the first embodiment. [Figure 8] FIG. 13 is a diagram illustrating a search key frame. [Figure 9] FIG. 13 is a diagram illustrating an example of a similarity score. [Figure 10] 13 is a flowchart illustrating an example of a key frame update process according to the second embodiment. [Figure 11] FIG. 13 is a diagram illustrating a comparison of visual words histograms. [Figure 12] 13 is a flowchart illustrating an example of a key frame update process according to the third embodiment. [Figure 13] FIG. 13 is a block diagram showing an example of a functional configuration of an information processing device according to a fourth embodiment. [Figure 14] 13 is a flowchart illustrating an example of a key frame update process according to the fourth embodiment. [Figure 15] FIG. 13 is a diagram illustrating extraction of a predetermined area in the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0020] <Embodiment 1> [Hardware configuration] 1 is a diagram showing an example of the hardware configuration of an information processing device 100. The information processing device 100 includes an imaging device 101, an input unit 102, a storage unit 103, a display unit 104, a control unit 105, and a bus .

[0021] The imaging device 101 captures a series of images. The captured images are utilized as digital information. The imaging device 101 is, for example, a monocular camera, a stereo camera, or a camera equipped with a depth sensor. The imaging device 101 may be disposed inside the information processing device 100, or may be disposed in a separate entity physically separated from the information processing device 100.

[0022] The input unit 102 is a device or operating member for giving instructions to the control unit 105, and specifically includes input devices such as a keyboard, a keypad, a button, a switch, a mouse, a touchpad, and a gesture input device.

[0023] The storage unit 103 is a storage medium such as a semiconductor memory, a hard disk, or a solid state drive, and stores programs and data for executing processing by the information processing device 100. The data stored in the storage unit 103 includes, for example, camera parameters of the imaging device 101 used to calculate the current position and orientation of the imaging device 101 from an image captured by the imaging device 101. The data stored in the storage unit 103 also includes key frame information. Note that some of the programs and data used by the information processing device 100 are not limited to the storage unit 103, and may be acquired from an external data source (for example, a data server, a network storage, or an external memory).

[0024] The display unit 104 is a display. The control unit 105 is, for example, a CPU (Central Processing Unit) or a DSP (Digital Signal The control unit 105 realizes various functions of the information processing device 100 by reading out a program stored in the storage unit 103 or another storage medium into a memory such as a random access memory (RAM) and executing the program.

[0025] The bus 106 interconnects the imaging device 101, the input unit 102, the storage unit 103, the display unit 104, and the control unit 105. The bus 106 is not limited to connecting the components of each unit in close physical proximity via wiring inside the information processing device 100, and may connect the components via a transmission path such as a LAN, a WAN, or the Internet. The method of providing information via the bus 106 may be a method of continuously transmitting information, or a method of transmitting information buffered in a file or memory to the control unit 105 or the like at a desired timing.

[0026] [Function configuration] 2 is a block diagram showing an example of a functional configuration of the information processing device 100 according to embodiment 1. The information processing device 100 according to embodiment 1 includes an imaging unit 111, a position and orientation calculation unit 112, a key frame creation unit 113, a key frame storage unit 114, a matching unit 116, and a key frame update unit 117.

[0027] The imaging unit 111 (imaging device 101) constantly captures images and converts them into digital information. The imaging unit 111 transmits the captured images to the position and orientation calculation unit 112. The imaging unit 111 may continuously transmit the captured images in order, or may accumulate the images in a format such as a file and then transmit them.

[0028] The position and orientation calculation unit 112 acquires an image captured by the imaging unit 111 and calculates the position and orientation of the imaging unit 111. The position and orientation can be calculated by, for example, a method called SLAM (Simultaneous Localization and Mapping). The position and orientation calculation unit 112 extracts a plurality of features from a space, and calculates the relative position and orientation of the imaging unit 111 with respect to the extracted features using camera parameters that are values ​​specific to the imaging unit 111. The position and orientation calculation unit 112 transmits the calculated position and orientation to an application, and the application can draw a CG object according to the received position and orientation.

[0029] Furthermore, the position and orientation calculation unit 112 transmits the image acquired from the imaging unit 111 and the position and orientation of the imaging unit 111 calculated from the image to the key frame creation unit 113, and instructs the creation of a key frame (KF). It is assumed that the position and orientation calculation unit 112 instructs the creation of a key frame according to creation conditions based on a general algorithm. For example, the position and orientation calculation unit 112 may instruct the creation of a key frame at regular time intervals, or may instruct the creation of a key frame when there is a movement of a predetermined distance or a predetermined change in orientation from the position and orientation when the previous key frame was created.

[0030] Furthermore, the position and orientation calculation unit 112 transmits the current captured image acquired from the imaging unit 111 to the matching unit 116, and instructs the matching unit 116 to search for a keyframe image capturing a range similar to the imaging range of the image. The position and orientation calculation unit 112 may give the instruction to the matching unit 116 for each frame, or may give the instruction when it is determined that there is a keyframe observing a place similar to the place to be observed from the current position and orientation of the imaging unit 111 (imaging device 101).

[0031] The key frame creation unit 113 creates a key frame upon receiving a key frame creation instruction from the position and orientation calculation unit 112. The key frame creation unit 113 creates a key frame by using an image received from the position and orientation calculation unit 112 as a key frame image. In addition to the key frame image, the key frame includes information on the position and orientation of the imaging unit 111 when the key frame image was captured, and position information of features extracted from the key frame image. The key frame creation unit 113 can receive the information on the position and orientation of the imaging unit 111 and the position information of the features from the position and orientation calculation unit 112. The created key frames have a unique Index and can be uniquely distinguished from each other.

[0032] Note that the features extracted from the image are not limited to feature points, and may be features of other forms such as lines (edges, feature lines), etc. In the following description, the features extracted are described as feature points, but are not limited to points and may be features of other forms.

[0033] The key frame storage unit 114 stores key frame information (KF information) of the key frames created by the key frame creation unit 113. The storage unit 103 may be a temporary storage such as a memory, or may be a non-volatile disk such as a hard disk. In response to an inquiry from another device, the key frame storage unit 114 can provide the other device with all or part of the key frame information.

[0034] The matching unit 116 compares the current captured image received from the position and orientation calculation unit 112 with a keyframe image capturing an area similar to the imaging area of ​​the current image. First, the matching unit 116 selects a keyframe observing an area (location) similar to the imaging area (location observed from the current position and orientation) of the current image acquired from the imaging unit 111 as a search keyframe. Next, the matching unit 116 evaluates the similarity between the current captured image and a keyframe image of the search keyframe (hereinafter also referred to as a search keyframe image), and calculates the similarity. The matching unit 116 calculates a similarity score (hereinafter, simply referred to as similarity) for each search keyframe, and determines the search keyframe whose calculated similarity score (hereinafter, simply referred to as similarity) is below a predetermined threshold (corresponding to a first threshold) as the keyframe to be updated. The matching unit 116 notifies the keyframe update unit 117 of the determined keyframe to be updated.

[0035] The key frame updating unit 117 updates the key frame information of the key frame to be updated determined by the matching unit 116. The key frame updating unit 117 updates the key frame information by invalidation processing that invalidates at least a part of the key frame image of the key frame to be updated.

[0036] The key frame updating unit 117 may update the key frame information by deleting the update target key frame (the key frame image of the update target key frame) as the invalidation process. The key frame updating unit 117 may also update the key frame information by lowering the reliability of the update target key frame.

[0037] [Keyframe (KF) creation process] A process flow for calculating the position and orientation of the image capturing device 101 (image capturing unit 111) and creating a key frame will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of the key frame creation process.

[0038] In step S301, the imaging unit 111 steadily captures images and converts them into digital information. The imaging unit 111 transmits the captured images to the position and orientation calculation unit 112. The imaging unit 111 continues to capture images at, for example, 60 fps.

[0039] In step S302, the position and orientation calculation unit 112 acquires an image captured by the imaging unit 111, and calculates the position and orientation of the imaging unit 111 from the acquired image. The position and orientation calculation unit 112 can calculate the position and orientation of the imaging unit 111 by using, for example, one of the SLAM techniques disclosed in Patent Document 1.

[0040] Specifically, the position and orientation calculation unit 112 extracts, as feature points, positions that are stationary in space and serve as intersection points from among the luminance components of the image captured by the imaging unit 111. FIG. 4 is a diagram showing an example of feature point extraction. An image 401 in FIG. 4 is an image captured of an environment in which a desk is placed. The position and orientation calculation unit 112 extracts, as feature points, corners 402 of the desk and boundary points 403 of a pattern on the desk, and calculates the positions of the extracted feature points. The position and orientation calculation unit 112 calculates the relative position and orientation between the extracted feature points and the imaging unit 111, and adds the relative position and orientation to the initial position and orientation of the imaging unit 111 to calculate the self-position and orientation of the imaging unit 111.

[0041] The position and orientation calculation unit 112 may, for example, determine the position and orientation at which the calculation is started as the origin, and set the origin as the initial position and orientation. Alternatively, the position and orientation calculation unit 112 may use the image capture unit 111 to recognize a marker prepared for determining the origin, and may calculate the self-position and orientation of the image capture unit 111 by setting the recognized marker position as the initial position and orientation and adding the relative position and orientation. A method of calculating the position and orientation between the image capture unit 111 and the marker by recognizing the marker is disclosed in, for example, Non-Patent Document 3.

[0042] In step S303, the key frame creation unit 113 creates a key frame upon receiving a key frame creation instruction from the position and orientation calculation unit 112. The condition for creating a key frame may be, for example, that the distance and angle have changed from the position and orientation when the previous key frame was created, and the ratio of feature points registered in the existing key frame among the feature points included in the key frame image has become equal to or less than a threshold. The threshold may be determined based on the processing load of the information processing device 100 and the accuracy of self-location estimation and environmental map creation.

[0043] 5 is a diagram showing an example of key frame information. The key frame information is stored in the key frame storage unit 114. Index 501 is an identifier that uniquely distinguishes each key frame. Index 501 is usually numbered in the order of 0, 1, ..., but when a key frame generated in the past is read and used, the index is numbered from the continuation of Index 501 of the existing key frame.

[0044] Image 502 is a key frame image. The size of image 502 may be the same as that of the key frame image, or may be an image obtained by reducing or compressing the size of the key frame image from the viewpoint of storage capacity. Image 502 may be an image captured by an RGB camera, or an image converted into a luminance component.

[0045] The position and orientation 503 is a value indicating the position and orientation of the image capturing unit 111 when a key frame image is created. The position and orientation 503 is indicated by, for example, a 4×4 matrix. The feature point information 504 is information on one or more feature points captured in the key frame image. The feature point information 504 is information on the index of the feature points included in the key frame image.

[0046] Fig. 6 is a diagram showing an example of information on feature points. Feature point index 601 is an identifier that uniquely distinguishes each feature point. Coordinates 602 are the coordinates of the feature point in the world coordinate system. If the feature extracted from the image is not a point but a line (edge, feature line), coordinates 602 can be, for example, information on the coordinates of both ends of the feature line. Feature point information 504 in the key frame information in Fig. 5 is represented by feature point index 601 shown in Fig. 6.

[0047] 6 is created when the position and orientation calculation unit 112 extracts and tracks feature points in image processing for each frame. For example, when a new feature point is extracted, the position and orientation calculation unit 112 assigns an index 601 and registers the feature point information. Furthermore, when the key frame creation conditions are satisfied and a key frame is created, the key frame creation unit 113 stores the index 601 of the feature point present in the key frame image, among the feature points registered in the feature point information in FIG. 6, as feature point information 504.

[0048] Since the information on feature points extracted from each frame captured by the imaging unit 111 generally becomes large in number, it is not necessarily preferable to continue storing all of the feature points from the viewpoint of memory usage. The position and orientation calculation unit 112 may not store information on some feature points based on various conditions. For example, when a new feature point is extracted and the presence of the same feature point is confirmed in one or more other key frame images, the position and orientation calculation unit 112 stores information on the extracted feature point as information on the feature point shown in FIG. 6. On the other hand, when the presence of the same feature point as the new feature point is not confirmed in other key frame images, the position and orientation calculation unit 112 does not store the feature point. In addition, without being limited to this example, the position and orientation calculation unit 112 may select whether or not to store information on the feature point by various methods.

[0049] The information on the feature point may include various information for improving the alignment accuracy and the user experience in addition to the coordinates 602 exemplified in Fig. 6. For example, the information on the feature point may include (1) the number of times that the feature point is stationary and processed as a valid feature point, and (2) information indicating whether the feature point is a stationary valid feature point or a moving invalid feature point. Furthermore, when the imaging device 101 is a stereo camera, the information on the feature point may include (3) disparity information between the left camera image and the right camera image for the feature point, and (4) disparity information of the previous frame. By using these pieces of information, the information processing device 100 can improve the alignment accuracy and the user experience.

[0050] [Keyframe (KF) update process] The key frame update process by the information processing device 100 will be described with reference to Fig. 7. The key frame update process is an invalidation process that invalidates at least a part of key frame images that are not similar to the captured image, among key frame images captured in a range similar to the imaging range of the captured image. In the first embodiment, the invalidation process is a process of deleting key frame images (key frames). Fig. 7 is a diagram showing an example of the key frame update process in the first embodiment.

[0051] In step S701, the matching unit 116 first acquires a captured image captured by the imaging unit 111. Next, the matching unit 116 selects a keyframe image (keyframe) capturing an image of a range similar to the imaging range of the current captured image from a plurality of keyframe images prepared in advance. The keyframe image selected by the matching unit 116 is also called a search keyframe image.

[0052] The multiple key frame images prepared in advance are key frame images of the key frame created by the key frame creation process of Fig. 3, and are stored in the key frame storage unit 114. The key frame creation process shown in Fig. 3 may be executed in parallel with the key frame update process shown in Fig. 7. In this case, the matching unit 116 may select a search key frame image from the multiple key frame images including the newly created key frame image.

[0053] The matching unit 116 selects a keyframe image in which the position and orientation of the imaging unit 111 when the keyframe image is captured is in a nearby position and orientation to the position and orientation of the imaging unit 111 when the current captured image is captured, as a search keyframe image. Whether or not the position and orientation of the imaging unit 111 when the keyframe image is captured is in a nearby position and orientation to the position and orientation of the imaging unit 111 when the current captured image is captured can be determined based on, for example, the calculation results of the following Equations 1 to 3.

number

[0054] In formula 1, Pc is the position of the imaging unit 111, and Pki is the position of the i-th key frame. Distance is the difference in three-dimensional position between the position Pc of the imaging unit 111 and the position Pki of the i-th key frame. If condition 1 that Distance is less than a threshold Tt is satisfied, the matching unit 116 determines that the position of the imaging unit 111 when the current captured image was captured and the position of the imaging unit 111 when the key frame image was captured are neighboring positions.

[0055] In Equation 2, Qc is the orientation of the image capture unit 111, and Qki is the orientation of the i-th key frame. DiffRot is an orientation difference vector between the orientation Qc of the image capture unit 111 and the orientation Qki of the i-th key frame. Qc, Qki, and DiffRot are values ​​expressed by quaternion expressions that represent the orientations.

[0056] The orientation difference vector DiffRot is converted into an angle DiffAngle by Equation 3, which converts a quaternion representation into an angle representation. If condition 2 that the angle DiffAngle is less than a threshold value Tr is satisfied, the matching unit 116 determines that the angle DiffAngle is the angle at which the current captured image was captured. The orientation of the imaging unit 111 when the key frame image is captured is determined to be close to the orientation of the imaging unit 111 when the key frame image is captured.

[0057] When Condition 1 and Condition 2 are satisfied, the matching unit 116 determines that the position and orientation of the imaging unit 111 when the keyframe image was captured is in the vicinity of the position and orientation of the imaging unit 111 when the current captured image was captured. In other words, the matching unit 116 determines that the keyframe image that satisfies Condition 1 and Condition 2 captures a range that is similar to the imaging range of the current captured image, and selects it as a search keyframe image.

[0058] The search key frames will be described with reference to Fig. 8. Fig. 8 shows the current position and orientation 801 of the imaging unit 111 (imaging device 101), and a search range 802 in the vicinity of the imaging unit 111. The search range 802 is a range determined based on a threshold Tt of condition 1 and a threshold Tr of condition 2. In the example of Fig. 8, the search range 802 includes four search key frames (KF1 to KF4).

[0059] Assume that KF1 to KF4 correspond to key frames with indexes 501 of 1 to 4 in FIG. 5, for example. Images 502 of KF1 to KF4 are images of a desk with a pattern on the tabletop. The key frame image of KF1 is captured at a position and orientation closer to the current position and orientation than the other key frames. The key frame image of KF2 is captured from a position and orientation slightly different from KF1, but the feature points appearing in the key frame image have not changed much. The key frame image of KF3 is captured from the same position and orientation as KF2, but the key frame image contains still and moving objects that do not exist in the currently captured image. The key frame image of KF4 is captured from the same position and orientation as KF2, but the pattern on the tabletop has changed from that of the key frame image of KF2.

[0060] The matching unit 116 executes a loop process L1 for each of the search key frames i (i=1, ..., N) selected in step S701 in Fig. 7. The loop process L1 includes the processes of steps S702 and S703. Hereinafter, a case will be described in which the matching unit 116 selects KF1 to KF4 described with reference to Figs. 5 and 8 as search key frames in step S701. Also, the captured image at the current position and orientation is assumed to be image 401 shown in Fig. 4.

[0061] In step S702, the matching unit 116 calculates a similarity score between the current captured image and a search keyframe image (a keyframe image of a search keyframe i). For example, the matching unit 116 can calculate and evaluate the similarity score by a Bag of visual words (BoW) method.

[0062] Fig. 9 is a diagram showing an example of similarity scores. The vertical axis represents the similarity score between the current captured image and the search key frame image, and the similarity scores for each of the search key frame images KF1 to KF4 are shown in a bar graph. According to Fig. 9, the magnitude relationship of the similarity scores of each search key frame image is KF1>KF2>KF3>KF4.

[0063] In step S703, the matching unit 116 sets a search keyframe whose similarity score is below a threshold 901 (first threshold) illustrated in Fig. 9 as a keyframe to be updated. The first threshold is a threshold for determining whether the search keyframe image is a target for updating, and can be set based on, for example, a desired alignment accuracy.

[0064] In the example of FIG. 9, the search keyframe whose similarity score is below the threshold 901 is KF4. The matching unit 116 sets KF4 as the keyframe to be updated. When the loop process L1 is executed for each search keyframe i (i=1, . . . , N), the process proceeds to step S704. Proceed to.

[0065] In step S704, the key frame update unit 117 updates KF4, which is the key frame to be updated. For example, the key frame update unit 117 deletes KF4 stored in the key frame storage unit 114. That is, the key frame image of KF4 is deleted and becomes invalid.

[0066] The multiple key frames created by the key frame creation unit 113 have a graph structure, and for example, an edge is established between key frames that have a certain number or more of common feature points. When the key frame update unit 117 deletes a key frame to be updated, it re-establishes an edge between the deleted key frame and the key frames that previously had an edge between them.

[0067] According to the above-mentioned embodiment 1, when the similarity score between the current captured image and a key frame image of a previously created key frame is smaller than a first threshold, the information processing device 100 deletes the key frame image. Therefore, even if an object placed in the real environment moves, the information processing device 100 can continue suitable alignment by deleting the key frame image capturing the real environment before the object moves.

[0068] 7, the matching unit 116 compares the entire current captured image with the entire search keyframe image to calculate the similarity score, but may set an invalid area that is not a target of comparison, as in the fourth embodiment. For example, when a moving object is captured in the current captured image, the matching unit 116 sets the area of ​​the moving object as an invalid area, and also sets the corresponding area of ​​the search keyframe image as an invalid area. By excluding the invalid area and calculating the similarity score between the current captured image and the search keyframe image, the matching unit 116 can reduce the effect of the moving object on the similarity score and continue suitable alignment.

[0069] <Embodiment 2> In the first embodiment, the information processing device 100 deletes a key frame image (key frame) when the similarity score between the current captured image and a key frame image of a previously created key frame is smaller than a first threshold value. In contrast, in the second embodiment, the information processing device 100 does not delete the key frame image, but invalidates at least a part of the key frame image to reduce the impact on the accuracy of alignment. In the second embodiment, the key frame update process (invalidation process of the key frame image) by the key frame update unit 117 is different from the key frame update process of the first embodiment. The hardware configuration and functional configuration of the information processing device 100 according to the second embodiment are similar to those of the first embodiment, and therefore description thereof will be omitted.

[0070] 3 and 10, the processing steps different from those in the first embodiment will be described. In the second embodiment, the key frame creation processing in step S303 in FIG.

[0071] In step S303 in Fig. 3, the key frame creation unit 113 creates a unique feature vector corresponding to the key frame image as key frame information. The key frame creation unit 113 can create the feature vector using a method called Bag of visual words (BoW). BoW is a method of compressing the feature amount of an image into a one-dimensional vector.

[0072] According to the method disclosed in Non-Patent Document 1, the key frame creation unit 113 first extracts feature vectors of each feature point from the key frame image when creating the key frame. Next, the key frame creation unit 113 clusters the feature vectors and sets the center vector as a feature vector called visual words. As a result, A unique histogram is obtained for each visual word. When creating a keyframe, keyframe creation unit 113 creates a histogram for each visual word for the keyframe image, and stores the histogram in keyframe storage unit 114 (storage unit 103) as part of the keyframe information.

[0073] The key frame update process of the second embodiment will be described with reference to Fig. 10. The key frame update process is an invalidation process that invalidates at least a part of a key frame image that is not similar to a captured image, among key frame images captured in a range similar to the imaging range of a captured image. The invalidation process in the second embodiment is a process (partial key frame update process) that invalidates a part of a key frame image (key frame). The same processes as those in Fig. 7 of the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0074] In step S701, the matching unit 116 selects a search keyframe image in the same manner as in the first embodiment. The matching unit 116 executes a loop process L2 for each of the search keyframes i (i=1, ..., M) selected in step S701 of Fig. 10. The loop process L2 includes the processes of steps S702, S703, and S1001 to S1003. The processes of steps S702 and S703 are the same as those in the first embodiment. In step S703, the matching unit 116 sets a search keyframe whose similarity score is lower than a first threshold as a keyframe to be updated.

[0075] For a search key frame i that has not been set as a key frame to be updated, the key frame update unit 117 does not execute the processes of steps S1001 to S1003, and proceeds to loop process L2 for the next search key frame i+1.

[0076] In step S1001, the key frame updating unit 117 reads a histogram of visual words for the key frame image of the key frame to be updated from the key frame storage unit 114. The key frame updating unit 117 also creates a histogram of visual words for the currently captured image using the method explained in step S303 of Fig. 3, and compares it with the histogram of visual words for the key frame image of the key frame to be updated.

[0077] Fig. 11 is a diagram for explaining a comparison of histograms of visual words. Fig. 11 shows a histogram 1102 of visual words corresponding to a current captured image 1101, and a histogram 1104 of visual words corresponding to a key frame image 1103. The horizontal axis of the histograms 1102 and 1104 is visual words 1105. The histograms 1102 and 1104 each show the frequency of four visual words.

[0078] 10, the key frame update unit 117 selects visual words based on the difference between a histogram 1102 of a current captured image 1101 and a histogram 1104 of a key frame image 1103. Because the current captured image 1101 and the key frame image 1103 are captured by the same imaging unit 111 and have the same resolution, their histogram frequencies can be compared equally.

[0079] In the example of Fig. 11, a histogram 1106 is created from the absolute value of the difference between a histogram 1102 and a histogram 1104. The key frame update unit 117 selects visual words whose absolute value of the difference is greater than the other visual words. For example, the key frame update unit 117 selects visual words whose number of types is half the number of types of visual words. In the example of FIG. 11, half of the four visual words (the two visual words surrounded by a frame 1107) are selected in descending order of absolute value of the difference. Also, for example, the key frame update unit 117 may select visual words whose absolute value of the difference is greater than a predetermined threshold value.

[0080] In step S1003 of Fig. 10, key frame update unit 117 identifies a position in the key frame image that corresponds to the visual word selected in step S1002. Key frame update unit 117 sets the vicinity of the image coordinates of the identified position as an invalid area. In the example of Fig. 11, key frame update unit 117 sets area 1108 surrounded by a dotted line as an invalid area. Since there are multiple positions that correspond to the visual words selected in step S1002, key frame update unit 117 combines the invalid areas that are the vicinity of each position. Once loop process L2 has been executed for each search key frame i (i = 1, ..., M), the process proceeds to step S1004.

[0081] In step S1004, the key frame updating unit 117 updates the update target key frame in which the invalid area was set in step S1003, and stores the update target key frame in the key frame storage unit 114. Specifically, the key frame updating unit 117 paints the invalid area in the key frame image of the update target key frame in black, and overwrites the original key frame image.

[0082] According to the above-mentioned second embodiment, the information processing device 100 can continue suitable alignment by invalidating the area of ​​the object not appearing in the current captured image in the key frame image without deleting the key frame to be updated. Furthermore, the information processing device 100 can suppress the decrease in the density of the key frames and reduce the deterioration of the alignment accuracy by invalidating a part of the key frame image without deleting the key frame.

[0083] <Embodiment 3> In the first and second embodiments, the first threshold for determining whether or not a search keyframe image is to be updated is assumed to be a predetermined constant value. In contrast, in the third embodiment, the first threshold is set based on the density of the search keyframe corresponding to the search keyframe image selected in step S701 of Fig. 7. The first threshold varies depending on the density of the search keyframe, so that the matching unit 116 can weight the search keyframe when determining whether or not it is to be updated.

[0084] When the density is higher (a sufficient number of search keyframes exist), the keyframe updating unit 117 sets more search keyframes as update targets by setting the first threshold higher. On the other hand, when the density is lower (a sufficient number of search keyframes do not exist), the keyframe updating unit 117 sets the first threshold lower to reduce the number of search keyframes to be set as update targets. In a position and orientation where the density of search keyframe images is lower, the keyframe updating unit 117 can reduce a decrease in alignment accuracy by maintaining the number of keyframes.

[0085] The hardware configuration and the functional configuration of the information processing device 100 according to the third embodiment are the same as those of the first embodiment, and therefore the description thereof will be omitted. Also, the key frame creation process shown in FIG. 3 is the same as that of the first embodiment, and therefore the description thereof will be omitted.

[0086] The key frame update process of the third embodiment will be described with reference to Fig. 12. In the example of Fig. 12, the first threshold is set based on the density of the search key frame. Since the density of the search key frame varies depending on the position and orientation of the imaging unit 111 when the current captured image is captured, the first threshold varies according to the position and orientation of the imaging unit 111. The same processes as those in Fig. 7 of the first embodiment are denoted by the same reference numerals and detailed description thereof will be omitted.

[0087] In step S701, the matching unit 116 performs a search key frame similar to the first embodiment. The matching unit 116 executes a loop process L3 for each of the search key frames i (i=1, . . . , L) selected in step S701 of FIG. 12. The loop process L3 includes the processes of steps S702 and S1201. The process of step S702 is the same as that in the first embodiment.

[0088] In step S1201, the key frame update unit 117 sets the similarity score calculated in step S702 in Fig. 12 to the key frame information of the search key frame i. The set similarity score indicates the reliability of the search key frame i. After loop processing L3 is executed for each search key frame i (i = 1, ..., L), the process proceeds to step S1202.

[0089] In step S1202, the key frame update unit 117 calculates and normalizes the density of the search key frames selected in step S701 (key frames corresponding to key frame images captured in a range similar to the imaging range of the current captured image). In the example of Fig. 8, there are four key frames in the search range 802. For example, if the upper limit of the number of key frames in the search range 802 is set to 5, the normalized key frame density is calculated as 4÷5=0.8. The key frame update unit 117 sets the value of the key frame density as the key frame information of each search key frame.

[0090] In step S1203, the key frame update unit 117 updates the search key frame having a lower similarity score than the key frame density calculated in step S1201 as a key frame to be updated. Specifically, the key frame update unit 117 deletes the key frame to be updated stored in the key frame storage unit 114.

[0091] According to the above-mentioned third embodiment, the information processing device 100 actively deletes key frame images captured at positions and orientations with higher key frame density, while making it difficult to delete key frame images captured at positions and orientations with lower key frame density. The information processing device 100 can maintain the key frame density and suppress a decrease in the alignment accuracy by not deleting many key frame images captured at positions and orientations with lower key frame density.

[0092] Note that the keyframe update unit 117 determines whether or not to update the search keyframe by comparing the similarity score with the normalized keyframe density, but this is not limited to the above, and the similarity score may be used for various processes of the information processing device 100.

[0093] For example, the information processing device 100 can use the magnitude of the similarity score for the correction amount of the feature position during pose graph optimization (bundle adjustment). A pose graph is a graph structure consisting of nodes and edges, where the nodes are configured as the positions and postures of key frames, and the edges are configured as the relative positions and postures between key frames. Pose graph optimization is a process of correcting the position of each feature so that when each feature is reprojected from the estimated current position and posture onto a specific plane, the difference (reprojection error) between the position of the feature on the image in the current camera image is as small as possible for the entire feature. Details of pose graph optimization are disclosed in Non-Patent Document 1 and Non-Patent Document 4. In the process of pose graph optimization, the information processing device 100 can weight the contribution rate of the key frame according to the similarity score so that the contribution rate when calculating the correction amount of the feature position is lower for a key frame with a lower similarity score.

[0094] <Embodiment 4> In the first embodiment, the information processing device 100 calculates a similarity score by comparing the entire current captured image with the entire search keyframe image. In contrast, in the fourth embodiment, the information processing device 100 calculates a similarity score by comparing the entire current captured image with the entire search keyframe image. A similarity score with an image (second image) obtained by removing an area corresponding to the predetermined area from the search keyframe image is calculated. The information processing device 100 performs a keyframe update process (a process of invalidating a keyframe image) based on the similarity score calculated from the image obtained by removing the predetermined area. The hardware configuration of the information processing device 100 according to the fourth embodiment is the same as that of the first embodiment, and therefore a description thereof will be omitted.

[0095] 13 is a block diagram showing an example of a functional configuration of an information processing device 100 according to embodiment 4. The information processing device 100 according to embodiment 4 includes an imaging unit 111, a position and orientation calculation unit 112, a key frame creation unit 113, a key frame storage unit 114, a region extraction unit 115, a matching unit 116, and a key frame update unit 117. The processing of the region extraction unit 115 added in embodiment 4, and processing of the functional configurations having the same reference numerals as in FIG. 2 that differ from embodiment 1 will be described.

[0096] The position and orientation calculation unit 112 transmits the current captured image acquired from the imaging unit 111 to the area extraction unit 115, not to the matching unit 116. In addition, the position and orientation calculation unit 112 instructs the area extraction unit 115 to extract a predetermined area (two-dimensional area).

[0097] The area extraction unit 115 extracts a predetermined area from the current captured image received from the position and orientation calculation unit 112, and transmits the first image with the predetermined area removed to the matching unit 116. The predetermined area is an area in the current captured image that may become noise when calculating the position and orientation. The area extraction unit 115 generates the first image by filling the predetermined area with a specific color or leaving the area other than the predetermined area as an image. The area extraction unit 115 transmits the generated first image to the matching unit 116.

[0098] The matching unit 116 compares the first image received from the region extraction unit 115 with a keyframe image capturing a range similar to the imaging range of the first image. First, the matching unit 116 selects a keyframe observing a range (location) similar to the imaging range (location observed from the current position and orientation) of the current captured image acquired from the imaging unit 111 as a search keyframe. Next, the matching unit 116 evaluates the similarity between the first image and a second image obtained by excluding an area corresponding to a predetermined area from the search keyframe image, and calculates a similarity score. The matching unit 116 determines a search keyframe whose calculated similarity score is below a predetermined threshold (first threshold) as an update target keyframe. The matching unit 116 notifies the keyframe update unit 117 of the determined update target keyframe.

[0099] The key frame update process of the fourth embodiment will be described with reference to Fig. 14. In the fourth embodiment, the matching unit 116 calculates a similarity score by comparing a first image obtained by removing a predetermined area from the current captured image with a second image obtained by removing an area corresponding to the predetermined area from the search key frame image. The same processes as those in Fig. 7 of the first embodiment are denoted by the same reference numerals and detailed description thereof will be omitted.

[0100] In step S1401, the area extraction unit 115 receives a current captured image from the position and orientation calculation unit 112, extracts a predetermined area, and generates a first image other than the predetermined area. The area extraction unit 115 transmits the generated first image to the matching unit 116.

[0101] Extraction of a predetermined region will be described with reference to Figs. 15(a) to 15(d). The region extraction unit 115 can acquire information on error of feature points from the position and orientation calculation unit 112. The key frame storage unit 114 stores the three-dimensional positions of feature points as key frame information, and the position and orientation calculation unit 112 can calculate where a known feature point should be drawn on an image based on the amount of movement from the previous frame to the current frame. The position and orientation calculation unit 112 calculates the extent to which the feature point actually deviates in the captured image from the calculated image coordinate value of the feature point. The amount of error of the feature point can be obtained depending on whether it is detected.

[0102] A feature point whose error amount exceeds a predetermined threshold (second threshold) is treated as an invalid feature point and is not used in calculating the position and orientation. In an image 1501 shown in Fig. 15(a), valid feature points are indicated by black stars and invalid feature points by white squares. When a moving object such as a person is captured, the error amount of the feature point becomes large. When the error amount of a feature point exceeds the second threshold, the area extraction unit 115 sets the feature point as an invalid feature point.

[0103] An image 1502 in Fig. 15(b) shows a state in which a circle (dotted circle) of a predetermined radius is drawn with each invalid feature point as its center. The predetermined radius can be set, for example, based on the distance between the invalid feature points. The predetermined radius may be a predetermined value, or may be a specific value according to the average distance between the invalid feature points. Moreover, the predetermined radius may be different for each invalid feature point.

[0104] Image 1503 in Fig. 15(c) shows an area (hatched area) obtained by combining areas surrounded by dotted circles drawn in image 1502. Area extraction unit 115 extracts the area hatched in image 1503 from the current captured image as a predetermined area. Area extraction unit 115 paints the extracted predetermined area black, and transmits the image 1503 with the blackened area removed therefrom to matching unit 116 as a first image.

[0105] The region extraction unit 115 may extract the predetermined region by other methods. For example, if the predetermined region covers up a valid feature point, the predetermined radius of a circle centered on the invalid feature point may be reduced and adjusted. The predetermined region may also be the region of a moving object detected in the current captured image.

[0106] In step S701, the matching unit 116 selects a search keyframe image in the same manner as in the first embodiment. The matching unit 116 executes a loop process L4 for each of the search keyframes i (i=1, ..., K) selected in step S701 in Fig. 14. The loop process L4 includes the processes of steps S1402, S1403, and S703. The process of step S703 is the same as in the first embodiment.

[0107] In step S1402, the matching unit 116 temporarily applies the predetermined area extracted in the current captured image in step S1401 to the key frame image of the search key frame i. Specifically, the matching unit 116 acquires a second image obtained by removing the area corresponding to the predetermined area from the key frame image of the search key frame i. Image 1504 in Fig. 15(d) shows an example of the second image obtained by removing the area corresponding to the predetermined area from the key frame image of KF2 with Index=2 in Fig. 5.

[0108] In step S1403, the matching unit 116 calculates a similarity score between the current captured image (first image) excluding the predetermined area and the keyframe image (second image) of the search keyframe i to which the predetermined area has been applied. For example, the matching unit 116 can calculate and evaluate the similarity score by the BoW method. The second image is an image to which the predetermined area has been temporarily applied to the search keyframe image in order to calculate the similarity score, and after the calculation of the similarity score, the application of the predetermined area to the search keyframe image is released.

[0109] In step S703, the matching unit 116 sets a search keyframe whose similarity score is below the first threshold as a keyframe to be updated, as in the first embodiment. After loop process L4 is executed for each search keyframe i (i=1, . . . , K), the process proceeds to step S704. In step S704, the keyframe update unit 117 sets a search keyframe whose similarity score is below the first threshold as a keyframe to be updated, as in step S704 in the first embodiment or step S1004 in the second embodiment. Update the frame.

[0110] According to the above-mentioned embodiment 4, even if a moving object is captured in the current captured image, the information processing device 100 calculates the similarity score excluding a predetermined area extracted based on the moving object, so that the key frame can be appropriately updated even in an environment where people or objects move. Even if a moving object is captured in the current captured image, the information processing device 100 can suppress a decrease in the similarity score with the key frame image and reduce a decrease in the accuracy of alignment.

[0111] <Modification> In each of the above embodiments, an example has been described in which live images captured by the imaging unit 111 are processed sequentially, but the information processing device 100 may store images captured by the imaging unit 111 in a file on a computer, and execute the processing described in each embodiment while playing the file. In this case, the description of "current captured image" should be read as "current image of the file being played."

[0112] In addition, in each of the above embodiments, an example in which the key frame update process is executed sequentially has been described, but the information processing device 100 may execute the key frame update process at various timings that a person skilled in the art can imagine. For example, the information processing device 100 may execute the key frame update process during recovery processing after alignment has failed and processing for detecting a loop in the trajectory of the position and orientation of the image capture unit 111.

[0113] In addition, in each of the above embodiments, the key frame update unit 117 updates the search key frame set by the matching unit 116 as the key frame to be updated. However, the determination of whether or not to update the search key frame may be made in combination with other conditions. That is, even if the search key frame is set by the matching unit 116 as the key frame to be updated, the key frame update unit 117 can determine that the search key frame should not be updated based on the key frame information of the search key frame. On the other hand, even if the search key frame is not set by the matching unit 116 as the key frame to be updated, the key frame update unit 117 can determine that the search key frame can be updated (to be updated) based on the key frame information of the search key frame. The key frame update unit 117 leaves the search key frame that can improve the accuracy of alignment as not updateable, and deletes or invalidates a part of the key frame image of the search key frame that may reduce the accuracy of alignment as updateable.

[0114] For example, when the keyframe image of the search keyframe is the first generated keyframe image among the keyframe images stored in the keyframe storage unit 114, the keyframe updating unit 117 may not update the search keyframe. Whether the keyframe image of the search keyframe is the first generated can be determined by, for example, whether the index of the keyframe information is the first number among the keyframes stored in the keyframe storage unit 114.

[0115] Also, for example, when the search keyframe is a keyframe read from a saved file, the keyframe update unit 117 may be able to update the search keyframe. The saved file here is a map file including keyframe information, and is a file saved in the storage unit 103 of the information processing device 100 or in the cloud, etc. For example, after a map file is created and saved in a specified area, when the information processing device 100 continues to create a map file in the same specified area the next day, it reads the file saved the previous day. If an object such as a desk is moved in the specified area after the map file is saved, the information on the keyframe read from the saved file may be out of date. For this reason, the keyframe update unit 117 may update the keyframe read from the saved file. The system can be updated (subject to update).

[0116] Furthermore, for example, when the key frame information includes the creation date and time of the key frame image, the key frame update unit 117 may not update the search key frame whose creation date and time is later than a predetermined date and time that is a threshold value.

[0117] Also, for example, when the search key frame exists at a position where the loop of the trajectory of the position and orientation of the image capturing device 101 is closed, the key frame updating unit 117 may decide not to update the search key frame.

[0118] Also, for example, when a search key frame is disconnected from other key frames in a graph structure showing connections between key frames, the key frame updating unit 117 may decide not to update the disconnected search key frame.

[0119] Also, for example, when the blur strength of the keyframe image of the search keyframe exceeds a predetermined threshold value of the blur strength, the keyframe updating unit 117 may determine that the search keyframe can be updated (is subject to update).

[0120] Also, for example, if the error in the position and orientation of the imaging device 101 when capturing a keyframe image of a search keyframe exceeds a predetermined error threshold, the keyframe update unit 117 may determine that the search keyframe can be updated (is subject to update).

[0121] Also, for example, the key frame update unit 117 may determine whether or not the search key frame is to be updated by combining a plurality of examples of the above-mentioned methods for determining whether or not the key frame is to be updated, or by combining them with a determination based on the similarity score calculated by the matching unit 116. Furthermore, the key frame update unit 117 may adjust the range of the area in which the search key frame image is invalidated, or adjust the weight of the key frame or the similarity score, based on the determination result by each determination method. The information processing device 100 can further improve the accuracy of alignment by combining each determination method according to the environment.

[0122] In addition, in each of the above embodiments, an example in which the matching unit 116 calculates the similarity score by the BoW method has been described, but the similarity score may be calculated by other methods. For example, the similarity score may be calculated based on the results of image feature extraction by machine learning (neural network), object recognition in the image, detection of positional relationships between objects, etc. The information processing device 100 can obtain the effect of differentiating the similarity even in images from which features are difficult to extract, or differentiating the similarity depending on the object and background recognized in the image, by calculating the similarity score using a method according to the surrounding environment.

[0123] Although the present invention has been described in detail above based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. In addition, parts of the above-mentioned embodiments may be appropriately combined.

[0124] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0125] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An acquisition means for acquiring an image captured by an imaging device; A selection means for selecting a key frame image capturing an area similar to the image capturing area of ​​the image from a plurality of key frame images prepared in advance; a control means for controlling a disabling process for disabling at least a part of the key frame image based on a similarity between the image and the key frame image selected by the selection means; 13. An information processing device comprising: (Configuration 2) The control means performs control so as to delete key frame images having a similarity smaller than a first threshold value. 2. The information processing device according to configuration 1. (Configuration 3) The control means performs control so as to invalidate a part of the key frame images having the similarity smaller than a first threshold value. 2. The information processing device according to configuration 1. (Configuration 4) The first threshold is set based on the density of the key frame corresponding to the key frame image selected by the selection means. 4. The information processing device according to configuration 2 or 3. (Configuration 5) The control means performs control so as to invalidate a part of an area in the key frame image selected by the selection means that is not similar to the image. 5. The information processing device according to any one of configurations 1, 3 and 4. (Configuration 6) The control means controls to invalidate a region in the key frame image corresponding to a visual word selected based on a difference in histograms for the visual word obtained by a Bag of Visual Words (BoW) technique for each of the image and the key frame image selected by the selection means. 5. The information processing device according to any one of configurations 1, 3 and 4. (Configuration 7) The control means generates a first image by removing a predetermined area from the image, and controls the invalidation process based on a similarity between the first image and a second image by removing an area corresponding to the predetermined area from the key frame image selected by the selection means. 7. The information processing device according to any one of configurations 1 to 6. (Configuration 8) The predetermined area is an area of ​​a moving object detected in the image. 8. The information processing device according to configuration 7. (Configuration 9) The predetermined region is a region including invalid feature points having an error amount greater than a second threshold value among feature points detected in the image. 8. The information processing device according to configuration 7. (Configuration 10) The control means When the key frame image selected by the selection means is the first generated key frame image among the plurality of key frame images, If the creation date and time of the key frame image selected by the selection means is later than a predetermined date and time, The key frame image selected by the selection means is a key frame image of a key frame that exists at a position where a loop of a trajectory of the position and orientation of the image capture device is closed, or The key frame image selected by the selection means indicates a connection between the key frames. In the graph structure, if the keyframe image is a keyframe that is disconnected from other keyframes, The invalidation process is not performed. 10. The information processing device according to any one of configurations 1 to 9. (Configuration 11) The control means When the key frame image selected by the selection means is a key frame image read from a saved file, When the blur intensity of the key frame image selected by the selection means exceeds a predetermined blur intensity threshold, or When an error in the position and orientation of the imaging device when capturing the key frame image selected by the selection means exceeds a predetermined error threshold value, Control to perform the invalidation process 10. The information processing device according to any one of configurations 1 to 9. (method) An acquisition step of acquiring an image captured by an imaging device; A selection step of selecting a key frame image capturing an image of a range similar to an imaging range of the image from a plurality of key frame images prepared in advance; An information processing method comprising: a control step of controlling an invalidation process for invalidating at least a part of the key frame image based on a similarity between the image and the key frame image selected in the selection step. (program) 12. A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 11. [Explanation of symbols]

[0126] 100: information processing device, 101: imaging device, 105: control unit

Claims

1. An acquisition means for acquiring an image captured by an imaging device; A selection means for selecting a key frame image capturing an area similar to the image capturing area of ​​the image from a plurality of key frame images prepared in advance; a control means for controlling an invalidation process for invalidating at least a part of the key frame image based on a similarity between the image and the key frame image selected by the selection means; 13. An information processing device comprising:

2. The control means performs control so as to delete key frame images having a similarity smaller than a first threshold value.

2. The information processing apparatus according to claim 1,

3. The control means performs control so as to invalidate a part of the key frame images having the similarity smaller than a first threshold value.

2. The information processing apparatus according to claim 1,

4. The first threshold is set based on the density of the key frame corresponding to the key frame image selected by the selection means.

3. The information processing apparatus according to claim 2.

5. The control means performs control so as to invalidate a part of an area in the key frame image selected by the selection means that is not similar to the image.

2. The information processing apparatus according to claim 1,

6. The control means performs control so as to invalidate a region in the key frame image corresponding to a visual word selected based on a difference in histogram for the visual word obtained by a Bag of Visual Words (BoW) technique for each of the image and the key frame image selected by the selection means.

2. The information processing apparatus according to claim 1,

7. The control means generates a first image by removing a predetermined area from the image, and controls the invalidation process based on a similarity between the first image and a second image by removing an area corresponding to the predetermined area from the key frame image selected by the selection means.

2. The information processing apparatus according to claim 1,

8. The predetermined area is an area of ​​a moving object detected in the image.

8. The information processing apparatus according to claim 7,

9. The predetermined region is a region including invalid feature points having an error amount greater than a second threshold value among feature points detected in the image.

8. The information processing apparatus according to claim 7,

10. The control means When the key frame image selected by the selection means is the first generated key frame image among the plurality of key frame images, If the creation date and time of the key frame image selected by the selection means is later than a predetermined date and time, The key frame image selected by the selection means is a key frame image of a key frame that exists at a position where a loop of a trajectory of the position and orientation of the image capture device is closed, and teeth, When the key frame image selected by the selection means is a key frame image of a key frame that is disconnected from other key frames in a graph structure showing connections between key frames, The invalidation process is not performed.

2. The information processing apparatus according to claim 1,

11. The control means When the key frame image selected by the selection means is a key frame image read from a saved file, When the blur intensity of the key frame image selected by the selection means exceeds a predetermined blur intensity threshold, or When an error in the position and orientation of the imaging device when capturing the key frame image selected by the selection means exceeds a predetermined error threshold value, Control to perform the invalidation process 2. The information processing apparatus according to claim 1,

12. An acquisition step of acquiring an image captured by an imaging device; A selection step of selecting a key frame image capturing an image of a range similar to an imaging range of the image from a plurality of key frame images prepared in advance; An information processing method comprising: a control step of controlling an invalidation process for invalidating at least a part of the key frame image based on a similarity between the image and the key frame image selected in the selection step.

13. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 11.

Images