Information processing device, information processing method, and program

Abstract

The present technology pertains to an information processing device, an information processing method, and a program that make it possible to easily confirm a position at which positioning is easy to be successful or a position at which positioning is easy to be failed. An information processing device according to the present technology comprises: an image capture direction calculation unit that calculates an image capture direction of a landmark included in a 3D map generated on the basis of a plurality of captured images of a real space; a viewpoint acquisition unit that acquires a virtual viewpoint of the user to the 3D map; and a drawing unit that draws a first image showing an appearance of the 3D map, and superimposes a second image based on the image capture direction of the landmark and the virtual viewpoint with the first image. This technique can be applied, for example, to an information processing device that visualizes a 3D map used in a VPS technique.

Description

Technical Field

[0001] The present technology relates to an information processing apparatus, an information processing method, and a program, and more particularly, to an information processing apparatus, an information processing method, and a program capable of easily identifying positions where positioning is likely to succeed and positions where positioning is likely to fail. Background Art

[0002] In recent years, visual positioning system (VPS) technology has been developed for estimating (positioning) the position and orientation of a user terminal based on captured images captured by the user terminal using a 3D map. In VPS, the position and orientation of the user terminal can be estimated with higher accuracy than the Global Positioning System (GPS). VPS technology is used in, for example, augmented reality (AR) applications (for example, see Patent Document 1).

[0003] Citation List

[0004] Patent Document

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2022-24169 Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] In practice, positioning cannot be performed at any position in the real space corresponding to the 3D map, but there are positions where positioning is likely to succeed and positions where positioning is likely to fail.

[0008] The 3D map is not in a format that can be understood by humans like a general map, but is stored in the form of a machine-readable database. Therefore, it is difficult for developers of AR applications and the like to determine positions where positioning is likely to succeed and positions where positioning is likely to fail in the real space corresponding to the 3D map.

[0009] The present technology has been proposed in view of such circumstances, and can easily identify positions where positioning is likely to succeed and positions where positioning is likely to fail.

[0010] Solution to the Problem

[0011] An information processing apparatus according to one aspect of the present technology includes: an image capture direction calculation unit that calculates an image capture direction of a landmark included in a 3D map generated based on a plurality of captured images obtained by capturing an image of a real space; a viewpoint acquisition unit that acquires a virtual viewpoint of a user with respect to the 3D map; and a drawing unit that draws a first image showing the appearance of the 3D map and superimposes a second image based on the image capture direction of the landmark and the virtual viewpoint on the first image.

[0012] In an information processing method according to an aspect of the present technology, an information processing apparatus performs the following operations: calculating an image capture direction of a landmark included in a 3D map generated based on a plurality of captured images obtained by capturing an image of the real space; acquiring a virtual viewpoint of a user with respect to the 3D map; and drawing a first image showing an appearance of the 3D map and superimposing a second image based on the image capture direction of the landmark and the virtual viewpoint on the first image.

[0013] A program according to an aspect of the present technology causes a computer to execute: calculating an image capture direction of a landmark included in a 3D map generated based on a plurality of captured images obtained by capturing an image of the real space; acquiring a virtual viewpoint of a user with respect to the 3D map; and drawing a first image showing an appearance of the 3D map and superimposing a second image based on the image capture direction of the landmark and the virtual viewpoint on the first image.

[0014] In an aspect of the present technology, an image capture direction of a landmark included in a 3D map generated based on a plurality of captured images obtained by capturing an image of the real space is calculated; a virtual viewpoint of a user with respect to the 3D map is acquired; and a first image showing an appearance of the 3D map is drawn, and a second image based on the image capture direction of the landmark and the virtual viewpoint is superimposed on the first image. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a diagram showing an application example of the VPS technology.

[0016] Figure 2 is a diagram showing an overview of the VPS technology.

[0017] Figure 3 is a diagram for explaining a method of estimating the KF viewpoint and landmark positions.

[0018] Figure 4 is a diagram for explaining a process of positioning.

[0019] Figure 5 is a diagram for explaining a process of positioning.

[0020] Figure 6 is a diagram for explaining a process of positioning.

[0021] Figure 7 is a diagram showing an example of an environment unsuitable for positioning.

[0022] Figure 8 is a diagram showing an example of positioning failure due to the lack of key frames included in the 3D map.

[0023] Figure 9 is a diagram showing an example of a device for solving failures in positioning.

[0024] Figure 10 A figure showing an example of the image capture direction of a landmark.

[0025] Figure 11 A figure showing an example of the display of a 3D view.

[0026] Figure 12 A block diagram showing an example of the configuration of an information processing apparatus according to a first embodiment of the present technology.

[0027] Figure 13 A flowchart for explaining the processing executed by the information processing apparatus.

[0028] Figure 14 For explaining in Figure 13 A flowchart of the image capture direction calculation process executed in step S3.

[0029] Figure 15 A figure showing an example of the display color of a landmark object.

[0030] Figure 16 A figure showing an example of a top view of a 3D map and a virtual viewpoint image.

[0031] Figure 17 A figure showing an example of a landmark object representing the image capture direction using color.

[0032] Figure 18 A figure showing an example of a landmark object representing the image capture direction using shape.

[0033] Figure 19 A figure showing an example of performing AR display of a landmark object.

[0034] Figure 20 A figure showing an example of a 3D view in which information based on a landmark score is displayed.

[0035] Figure 21 A figure showing an example of a method for generating a heat map.

[0036] Figure 22 A figure showing an example of a UI for inputting an operation for setting an evaluation direction.

[0037] Figure 23 A block diagram showing an example of the configuration of an information processing apparatus according to a second embodiment of the present technology.

[0038] Figure 24 A flowchart for explaining the processing executed by the information processing apparatus.

[0039] Figure 25 A figure showing another example of a UI for inputting an operation for setting an evaluation direction.

[0040] Figure 26 This is a diagram showing an example of multiple evaluation directions set for each grid.

[0041] Figure 27 This is a block diagram showing an example of the configuration of the hardware of a computer. Detailed implementation manners

[0042] Hereinafter, the modes for implementing the present technology will be described. The description will be given in the following order.

[0043] 1. Overview of VPS technology

[0044] 2. First embodiment

[0045] 3. Second embodiment

[0046] <<1. Overview of VPS technology>>

[0047] In recent years, a VPS technology has been developed for estimating the position and orientation of a user terminal based on a captured image captured by the user terminal using a 3D map. Hereinafter, estimating the position and orientation of the user terminal using the 3D map and the captured image is referred to as positioning.

[0048] Similar to VPS, GPS is also a system for estimating the position of a user terminal. In GPS, the estimation accuracy of the position of the user terminal is in meters. On the other hand, in VPS, the estimation accuracy of the position of the user terminal is higher than that in GPS (in the range of dozens of centimeters to several centimeters). In addition, different from GPS, VPS can be used in an indoor environment.

[0049] For example, VPS technology is used in AR applications. Through VPS technology, the position of the application user carrying the user terminal in the real space and the position where the user terminal is oriented can be known. Therefore, for example, in the case where the application user guides the user terminal to a predetermined position where an AR virtual object is virtually arranged in the real space, a VPS technology can be used to implement an AR application in which the AR virtual object is displayed on the display of the user terminal.

[0050] Figure 1 This is a diagram showing an application example of VPS technology.

[0051] For example, as Figure 1 shown, when the application user points the imaging device set on the smartphone as the user terminal in the direction the application user is facing in the town, a virtual object of an arrow indicating the direction of the destination is displayed on the display of the smartphone as superimposed on the captured image captured by the imaging device.

[0052] As described above, the VPS technology is used for navigation, entertainment, etc. using AR virtual objects.

[0053] Figure 2 It is a diagram showing an overview of the VPS technology.

[0054] As Figure 2 shown, the VPS technology includes two technologies: a technology for pre-generating a 3D map and a technology for performing positioning using the 3D map.

[0055] The 3D map is generated based on a set of captured images captured by a camera device at multiple positions and poses in the real space where positioning is to be performed. The 3D map indicates the appearance of the entire real space where imaging has been performed. The 3D map is configured by registering image information about the captured images captured by the camera device, three-dimensional shape information indicating the shape of the real space, etc. in a database.

[0056] One of the technologies for generating a 3D map is Structure from Motion (SfM). SfM is a technology for three-dimensionalizing an object or environment based on a set of captured images obtained by imaging a specific object or environment from various positions and directions. SfM is commonly used in photogrammetry technology that has attracted attention in recent years. Note that in addition to SfM, a 3D map can be generated by methods such as Visual Odometry (VO), Visual Inertial Odometry (VIO), Simultaneous Localization and Mapping (SLAM), or methods that combine images, Light Detection and Ranging (LiDAR), or GPS.

[0057] In the generation of the 3D map, a set of captured images pre-captured at various positions and poses in the real space where positioning is to be performed is used, and image information and three-dimensional shape information are estimated by various methods such as SfM, and these information pieces are stored in a database in a data format that is easy to use for positioning.

[0058] Specifically, the 3D map includes the KF viewpoints (imaging positions and imaging directions) of key frames selected from a set of pre-captured images, the positions of image feature points (keypoints, KPs) in the key frames, the three-dimensional positions (landmark positions) of the image feature points, the feature amounts of the keypoints (image feature amounts), the environmental grid indicating the shape of the real space, etc. Hereinafter, an object appearing at the keypoint part in a key frame is called a landmark. The 3D map also includes correspondence information indicating the correspondence between each keypoint and landmark and the key frame including each keypoint.

[0059] Figure 3 It is a diagram for explaining a method of estimating the KF viewpoint and landmark positions.

[0060] Figure 3The image planes S101 to S103 shown in the figure indicate virtual image planes on which the key frames KF1 to KF3 obtained by imaging the same cube at different positions and poses are projected respectively. In the key frames KF1 to KF3, a vertex (landmark L1) of the cube usually appears. The area (corresponding to the landmark L1) where the landmark L1 appears in the key frame KF1 is set as the key point KP1,1, the area in the key frame KF2 is set as the key point KP1,2, and the area in the key frame KF3 is set as the key point KP1,3. In the two-dimensional coordinate system of the key frame, the position of the key point KP1,1 is indicated by p 1,1 and the position of the key point KP1,2 is indicated by p 1,2 and the position of the key point KP1,3 is indicated by p 1,3

[0061] In various methods such as SfM, based on the positions of the key points KP1,1 to KP1,3 included in the three key frames KF1 to KF3, the landmark position x 1 of the landmark L1 is estimated by triangulation. In addition to estimating the landmark position x 1 , the imaging positions KFP1 to KFP3 and the imaging directions (poses) of the key frames KF1 to KF3 are also estimated based on the positions of the key points KP1,1 to KP1,3.

[0062] Returning to Figure 2 , positioning is performed by querying the captured image (hereinafter referred to as the query image or the real image) captured by the user terminal with respect to the 3D map. The position (position and pose) of the user terminal estimated based on the query image is provided to the user terminal and used for displaying AR virtual objects, etc. Note that the position where positioning can be performed in the real space corresponding to the 3D map is determined by the 3D map.

[0063] The process of positioning will be described with reference to Figures 4 to 6 . Positioning is mainly performed in three steps.

[0064] When positioning starts, as shown on the right side of Figure 4 , the user terminal 1 used by the application user U1 acquires the query image QF1 obtained by imaging the real space. When capturing the query image QF1, first, as indicated by the arrow in Figure 4 , each of the key frames KF1 to KF3 included in the 3D map is compared with the query image QF1, and the image most similar to the query image QF1 is selected from the key frames KF1 to KF3. For example, the key frame KF1 indicated by the thick line in Figure 4 is selected.

[0065] Next, as in Figure 5 ​As indicated by the arrow in, the correspondence of key points is searched for between the selected key frame KF1 and the query image QF1.

[0066] Next, as Figure 6 shown, the viewpoint (imaging position and imaging direction) of the query image QF1 is estimated based on the correspondence of key points between the key frame KF1 and the query image QF1 and the landmark positions corresponding to the key points.

[0067] Figure 6 The image planes S101 and S111 shown in are virtual image planes on which the query image QF1 and the key frame KF1 obtained by imaging the same cube at different positions and poses are projected. The landmark L1 generally appears in the key frame KF and the query image QF1. In the two-dimensional coordinate system of the key frame, the position of the key point KP in the key frame KF1 corresponding to the landmark L1 is indicated by p 1,1 and the position of the key point KP in the query image QF1 is indicated by p 1,2 .

[0068] Since the landmark position x 1 of the landmark L1 is known, the KF viewpoint of the query image QF1 is estimated by performing an optimization calculation based on the landmark position x 1 indicated by arrow #1 and the position of the key point KP on the image plane S111 to obtain the imaging position QFP1 and imaging direction of the query image QF1. In the optimization calculation for obtaining the KF viewpoint of the query image QF1, the positional relationship of the key point KP between the key frame KF1 and the query image QF1 indicated by arrow #2, and the imaging position KFP1 indicated by arrow #3, the position of the key point on the image plane S101, and the positional relationship of the landmark position x 1 are also used.

[0069] Actually, positioning cannot be performed at any position in the real space corresponding to the 3D map, but there are positions where positioning may succeed and positions where positioning may fail.

[0070] The main reasons for positions where positioning may fail are considered to be environments unsuitable for positioning and the lack of key frames included in the 3D map.

[0071] Figure 7 is a diagram showing an example of an environment unsuitable for positioning.

[0072] An environment having a mirror or glass as Figure 7 shown is unsuitable for positioning. In Figure 7 , the object reflected on the mirror or glass is indicated by a dotted line. As Figure 7As indicated by the cross in, an object reflected on a mirror or glass can also be designated as a landmark.

[0073] Since the appearance of an object reflected on a mirror or glass changes according to the imaging position, it is impossible to accurately find the correspondence of key points between the key frame and the query image, and the possibility of positioning failure is high. In addition, positioning may fail in a dark environment where the landmark is not visible in the query image, in an environment without features as landmarks (such as surrounded by a monochromatic wall or floor), in an environment where similar patterns such as a lattice pattern are continuous, etc. Positioning may succeed in an environment without a mirror, etc., that is bright enough and has many unique features.

[0074] Figure 8 It is a diagram showing an example of positioning failure due to the lack of key frames included in the 3D map.

[0075] As Figure 8 shown on the left side of, assume that the 3D map includes three key frames KF1 to KF3. In Figure 8 , the black dots shown in the parts of the building and the trees indicate the landmarks that appear in the key frames KF1 to KF3.

[0076] In the real space, the landmark appears sufficiently in Figure 8 the query image captured by the application user U11 shown on the right side of, and thus the positioning of the position of the application user U11 may succeed.

[0077] The query image captured by the application user U12 does not include sufficient landmarks. In other words, the 3D map does not include sufficient key frames obtained by capturing the landmarks corresponding to the key points in the query image. Since a key frame similar to the query image cannot be selected, the positioning for the position of the application user U12 may fail.

[0078] The query image captured by the application user U13 shows the same object as the object that appears in the key frames KF1 to KF3, but the key frame captured from a direction similar to the query image is not included in the 3D map. In other words, no valid landmark appears in the query image. Therefore, the positioning for the position of the application user U13 may fail.

[0079] As described above, the positioning using a query image captured from a viewpoint similar to the KF viewpoint of the key frames included in the 3D map may succeed to some extent, while the positioning using a query image captured from a viewpoint significantly different from the KF viewpoint of the key frames may fail.

[0080] In the case of developing an AR application using VPS technology, if the locations where positioning may succeed and the locations where positioning may fail are known, the application developer can arrange AR virtual objects in the locations where positioning may succeed. In addition, in the case where the location where it is desired to arrange an AR virtual object is a location where positioning may succeed, the application developer can arrange the AR virtual object at that location.

[0081] In the case where an AR virtual object is arranged at a location where positioning may fail, even if a query image is captured at that location, there is a possibility that the position and orientation of the user terminal cannot be estimated, and the AR virtual object cannot be displayed on the user terminal. Therefore, the application developer can implement measures to not arrange AR virtual objects in locations where positioning may fail.

[0082] In the case where there are locations where positioning may fail due to an environment unsuitable for positioning, the application developer can take measures on the environment side. For example, the application developer can implement measures such as covering the mirror part to make the mirror invisible or pasting posters, stickers, etc. on a featureless wall to create features.

[0083] In addition, in the case where there are locations where positioning may fail due to the lack of key frames included in the 3D map, the application developer can add a set of newly captured key frames near the locations where positioning may fail to the 3D map, as Figure 9 shown.

[0084] In addition to Figure 8 the key frames KF1 to KF3 included in the 3D map of Figure 9 the 3D map of Figure 9 also includes key frames KF11 and KF12. Key frame KF11 is a key frame captured near the position of application user U12 shown on the right side of

[0085] and key frame KF12 is a key frame captured near the position of application user U13.

[0086] Since the 3D map includes key frames KF11 and KF12, the positioning of the positions of application user U12 and application user U13 may succeed. By adding a set of newly imaged key frames to the 3D map, a location where positioning may fail can be set as a location where positioning may succeed.

[0087] Unlike a general map, a 3D map is not in a format that can be understood by humans, but is stored in the form of a machine-readable database. Therefore, in a situation where there are locations where positioning may fail due to the lack of key frames included in the 3D map, it is difficult for application developers (especially those other than the developers of the VPS algorithm) to determine which locations may result in successful positioning and which locations may result in failed positioning.

[0088] By going to the real space corresponding to the 3D map and actually performing positioning, it is possible to confirm whether the location is a location where positioning may succeed or a location where positioning may fail. However, actually going to the real space corresponding to the 3D map requires time and effort.

[0089] There is also the following method: visualizing the information included in the 3D map into a format that can be understood by humans to check the locations where positioning may succeed and the locations where positioning may fail. For example, the point groups indicating the KF viewpoints and landmarks are visualized. In this method, although it is not necessary to actually go to the area where the 3D map is prepared, it is difficult for those who do not understand the VPS technology algorithm to determine the locations where positioning may succeed and the locations where positioning may fail. In addition, in this method, it is only possible to qualitatively determine the locations where positioning may succeed and the locations where positioning may fail.

[0090] <<2. First Embodiment>>

[0091] · Summary of the First Embodiment

[0092] As described above, in a situation where there are locations where positioning may fail due to the lack of key frames included in the 3D map, it is difficult for application developers to determine which locations may result in successful positioning and which locations may result in failed positioning.

[0093] Therefore, an embodiment of the present technology proposes the following technology: it is possible to easily confirm the locations where positioning may succeed and the locations where positioning may fail by calculating the image capture direction of the landmarks included in the 3D map, obtaining the virtual viewpoint of the user relative to the 3D map, drawing a first image showing the appearance of the 3D map, and superimposing a second image based on the image capture direction of the landmarks and the virtual viewpoint on the first image.

[0094] As described with reference to Figure 8 A location where positioning may fail is a location where a query image that does not sufficiently include effective landmarks is captured. In the first embodiment of the present technology, the 3D map is visualized so that application developers can determine whether effective landmarks are sufficiently included in the query image captured at some arbitrary location and pose.

[0095] Specifically, the 3D map is visualized based on the image capture direction, which is the direction of the landmark relative to the imaging position of the key frame where the landmark appears.

[0096] Figure 10 This is a diagram showing an example of the image capture direction of a landmark.

[0097] In Figure 10 the example of, among the three key frames KF1 to KF3 included in the 3D map, the landmark L11 appears in the key frames KF1 and KF3. In Figure 10 , the image capture direction of the landmark L11 in the key frame KF1 is indicated by the arrow A1, and the image capture direction of the landmark L11 in the key frame KF3 is indicated by the arrow A3. The image capture direction of the landmark is calculated based on the landmark position and the KF viewpoints of the key frames in which the landmark appears. When a landmark appears in multiple key frames, the landmark has multiple image capture directions.

[0098] Hereinafter, an environmental grid included in a 3D map is arranged in 3D space and a virtual viewpoint image indicating the appearance (environmental grid) of the 3D map viewed from a virtual viewpoint (position and pose) set by an application developer is called a 3D view.

[0099] Figure 11 This is a diagram showing a display example of a 3D view.

[0100] In the 3D view, as Figure 11 shown on the upper side of, rectangular objects (landmark objects) indicating landmarks are arranged on the environmental grid. Note that the shape of the landmark object is not limited to a rectangle and can be, for example, circular or spherical.

[0101] When there is a key frame in which a landmark appears and is imaged from the same direction as the direction of the virtual viewpoint in the key frame where the landmark appears, the landmark object indicating the landmark is, for example, displayed in green. In other words, the landmark object displayed in green indicates a landmark that is effective when capturing a query image from the viewpoint (real viewpoint) of the real space corresponding to the virtual viewpoint. On the other hand, when there is no key frame in which a landmark appears and is imaged from the direction of the virtual viewpoint in the key frame where the landmark appears, the landmark object indicating the landmark is, for example, displayed in gray.

[0102] In Figure 11 , the effective landmarks in the virtual viewpoint are indicated by white landmark objects, and the ineffective landmarks in the virtual viewpoint are indicated by black landmark objects.

[0103] In Figure 11 the 3D view shown on the upper side of, for example, the landmark object Obj1 is displayed in black (gray), and the landmark object Obj2 is displayed in white (green). When the virtual viewpoint changes, as Figure 11 shown on the lower side of, the landmark object Obj1 is displayed in white (green), and the landmark object Obj2 is displayed in black (gray).

[0104] By viewing a 3D view while changing the virtual viewpoint and confirming the number of green landmark objects, the application developer can determine whether the real viewpoint corresponding to the virtual viewpoint is likely to be successfully located.

[0105] · Configuration of the information processing device

[0106] Figure 12 is a block diagram showing a configuration example of an information processing device 11 according to a first embodiment of the present technology.

[0107] Figure 12 The information processing device 11 in

[0108] is a device that displays a 3D view to confirm whether an effective landmark appears in a query image captured from a real viewpoint corresponding to a virtual viewpoint. For example, the application developer is a user of the information processing device 11. Figure 12 As shown in

[0109] The 3D map storage unit 21 stores a 3D map. The 3D map includes KF viewpoints, landmark positions, correspondence information, environmental meshes, etc. Note that, for example, point cloud data other than environmental meshes can be included in the 3D map as information indicating the shape of the real space.

[0110] The user input unit 22 includes a mouse, a gamepad, a joystick, etc. The user input unit 22 receives an input of an operation for setting a virtual viewpoint in the 3D space. The user input unit 22 provides information indicating the input operation to the control unit 23.

[0111] The control unit 23 includes an image capture direction calculation unit 31, a grid arrangement unit 32, a viewpoint position acquisition unit 33, a display color determination unit 34, an object arrangement unit 35, and a drawing unit 36.

[0112] The image capture direction calculation unit 31 acquires the KF viewpoint, the landmark position, and the correspondence information from the 3D map stored in the 3D map storage unit 21, and calculates the image capture direction of the landmark based on these pieces of information. The image capture direction calculation unit 31 provides the image capture direction of the landmark to the display color determination unit 34. Details of the method for calculating the image capture direction of the landmark will be described later.

[0113] The grid layout unit 32 obtains an environmental grid from the 3D map. The grid layout unit 32 arranges the environmental grid in a 3D space virtually formed on the storage unit 24. When the information indicating the shape of the environment included in the 3D map is point cloud data, the grid layout unit 32 arranges the point cloud indicated by the point cloud data in the 3D space.

[0114] The viewpoint position acquisition unit 33 sets a virtual viewpoint in the 3D space based on the information provided by the user input unit 22, and provides the information indicating the virtual viewpoint to the display color determination unit 34 and the rendering unit 36.

[0115] The display color determination unit 34 determines the color of the landmark object based on the image capture direction of the landmark calculated by the image capture direction calculation unit 31 and the virtual viewpoint set by the viewpoint position acquisition unit 33, and provides the information indicating the color of the landmark object to the object layout unit 35. A method for determining the color of the landmark object will be described later.

[0116] The object layout unit 35 obtains the landmark position from the 3D map, and arranges the landmark object with the color determined by the display color determination unit 34 at the landmark position on the environmental grid in the 3D space.

[0117] The rendering unit 36 renders a virtual viewpoint image indicating the appearance of the 3D map viewed from the virtual viewpoint determined by the viewpoint position acquisition unit 33, and provides the virtual viewpoint image to the display unit 25. The rendering unit 36 also serves as a rendering control unit that presents the virtual viewpoint image to the application developer.

[0118] The storage unit 24 is provided, for example, in a partial storage area of a random access memory (RAM). In the storage unit 24, a 3D space in which the environmental grid and the landmark objects are arranged is virtually formed.

[0119] The display unit 25 includes a display provided in a PC, a tablet terminal, a smart phone, etc., a monitor connected to these devices, etc. The display unit 25 displays the virtual viewpoint image provided by the rendering unit 36.

[0120] Note that the 3D map storage unit 21 may be provided in a cloud server connected to the information processing device 11. In this case, the control unit 23 obtains the information included in the 3D map from the cloud server.

[0121] · Operation of the information processing device

[0122] Next, the processing executed by the information processing device 11 having the above configuration will be described with reference to Figure 13 the flowchart.

[0123] In step S1, the control unit 23 loads the 3D map stored in the 3D map storage unit 21.

[0124] In step S2, the grid layout unit 32 arranges environmental grids in the 3D space.

[0125] In step S3, the image capture direction calculation unit 31 performs an image capture direction calculation process. Through the image capture direction calculation process, the image capture direction of each landmark included in the 3D map is calculated. Subsequently, Figure 14 The details of the image capture direction calculation process will be described. Note that the image capture direction of each landmark calculated when generating the 3D map can be included in the 3D map. In this case, the image capture direction calculation unit 31 obtains the image capture direction of each landmark from the 3D map.

[0126] In step S4, the object placement unit 35 places landmark objects at the landmark positions on the environmental map in the 3D space.

[0127] In step S5, the user input unit 22 receives an input of an operation related to the virtual viewpoint.

[0128] In step S6, the viewpoint position acquisition unit 33 sets a virtual viewpoint based on the operation received by the user input unit 22, and controls the position and attitude of the virtual camera device for drawing the virtual viewpoint image.

[0129] In step S7, the display color determination unit 34 determines the display color of the landmark object based on the virtual viewpoint and the image capture direction of the landmark.

[0130] In step S8, the object placement unit 35 updates the display color of the landmark object.

[0131] In step S9, the drawing unit 36 draws a virtual viewpoint image. The virtual viewpoint image drawn by the drawing unit 36 is displayed on the display unit 25. After that, the processes of steps S5 to S9 are repeatedly executed.

[0132] Next, with reference to Figure 14 the flowchart of Figure 13 the image capture direction calculation process executed in step S3 will be described.

[0133] In step S21, the image capture direction calculation unit 31 obtains the KF viewpoint of the key frame in which landmark [i] appears.

[0134] In step S22, the image capture direction calculation unit 31 calculates the vector from the landmark position of landmark [i] to the position of the KF viewpoint of key frame [j] as the image capture direction of landmark [i]. Assuming x iThe landmark position of landmark [i] and p j The KF viewpoint of key frame [j], the image capture direction v i Is represented by the following expression (1).

[0135] [Mathematical formula 1]

[0136]

[0137] In step S23, the image capture direction calculation unit 31 determines whether the image capture directions of all key frames in which landmark [i] appears have been calculated.

[0138] In the case where it is determined in step S23 that the image capture directions of all key frames in which landmark [i] appears have not been calculated, the image capture direction calculation unit 31 increments j (j = j + 1) in step S24. Thereafter, the process returns to step S22, and the process of step S22 is repeatedly executed until the image capture directions of all key frames in which landmark [i] appears are calculated.

[0139] On the other hand, in the case where it is determined in step S23 that the image capture directions of all key frames in which landmark [i] appears have been calculated, in step S25, the image capture direction calculation unit 31 determines whether the image capture directions of all landmarks have been calculated.

[0140] In the case where it is determined in step S25 that the image capture directions of all landmarks have not been calculated, the image capture direction calculation unit 31 increments i (i = i + 1) in step S26. Thereafter, the process returns to step S21, and the processes of steps S21 to S23 are repeatedly executed until the image capture directions of all landmarks are calculated. On the other hand, in the case where it is determined in step S25 that the image capture directions of all landmarks have been calculated, the process returns to Figure 13 Step S3 in, and subsequent processing is executed.

[0141] As described above, in the information processing apparatus 11, a virtual viewpoint image (first image) indicating the appearance of the 3D map viewed from the virtual viewpoint is presented to the application developer, and a second image including a landmark object drawn in a color according to the image capture direction is superimposed on the virtual viewpoint image. The landmark object is drawn in a color based on the image capture direction of the landmark (for example, green or gray). By viewing the 3D view while changing the virtual viewpoint and confirming the number of green landmark objects, the application developer can easily determine whether the positioning of the virtual viewpoint is likely to succeed.

[0142] · Method for determining the display color of the landmark object

[0143] When the image capture direction of the landmark is oriented towards the position of the virtual viewpoint, it is considered that the landmark appears in the key frame captured from the KF viewpoint similar to the virtual viewpoint, and it can be said that the landmark is valid for the virtual viewpoint.

[0144] In other words, it can be said that the smaller the angle formed by the image capture direction of the landmark and the direction of the virtual viewpoint, the more effective the landmark. Assume that the vector of the image capture direction of landmark [i] is v i and the vector of the direction of the virtual viewpoint is c. The angle θ formed by the image capture direction (the opposite direction) of landmark [i] and the direction of the virtual viewpoint is represented by the following expression (2).

[0145] [Mathematical formula 2]

[0146] v i = p i - x i …(1)

[0147]

[0148] Figure 15 is a diagram showing an example of the display color of the landmark object.

[0149] On the left side of A in Figure 15 , the arrow A11 shows an example where the image capture direction of the landmark indicated by the landmark object Obj11 is opposite to the direction of the imaging device C1 for drawing the virtual viewpoint image in which the landmark object Obj11 appears.

[0150] As shown on the left side of A in Figure 15 , when the angle formed by the image capture direction (the opposite direction) of the landmark indicated by the landmark object Obj11 and the direction of the virtual viewpoint is greater than the threshold value, the landmark is invalid for the virtual viewpoint. Therefore, as shown on the right side of A in Figure 15 , the gray landmark object Obj11 is displayed in the 3D view.

[0151] On the left side of B in Figure 15 , the arrow A12 shows an example where the image capture direction of the landmark indicated by the landmark object Obj11 is towards the direction near the imaging device C1.

[0152] As shown on the left side of B in Figure 15 , when the angle formed by the image capture direction (the opposite direction) of the landmark indicated by the landmark object Obj11 and the direction of the virtual viewpoint is less than the threshold value, the landmark is valid for the virtual viewpoint. Therefore, as shown on the right side of B in Figure 15 , the green (indicated by white in Figure 15 ) landmark object Obj11 is displayed in the 3D view.

[0153] As described above, the landmark object is drawn in a color corresponding to the angle formed by the image capture direction of the landmark and the direction of the virtual viewpoint. How small the angle formed by the image capture direction of the landmark and the direction of the virtual viewpoint is such that the landmark can be used for the virtual viewpoint depends on the positioning algorithm. Therefore, the threshold for determining the display color of the landmark object is appropriately set by the positioning algorithm. Note that the color of the landmark object can gradually change according to the angle formed by the image capture direction of the landmark and the direction of the virtual viewpoint.

[0154] · Modification

[0155] <Example considering being blocked by buildings etc.>

[0156] Landmarks that are far enough from the position of the virtual viewpoint and invisible (blocked) landmarks that are blocked by an object such as a building from the virtual viewpoint are not used for positioning. Therefore, the landmark objects indicating such landmarks may not be displayed in the 3D view.

[0157] Figure 16 is a diagram showing an example of a top view of a 3D map and an image of a virtual viewpoint.

[0158] In Figure 16 In the 3D map shown on the upper side, landmarks exist in the part surrounded by the ellipse, but even when viewed from the virtual viewpoint CP1, the landmark objects indicating the landmarks cannot be seen because the landmarks are blocked by the building existing therebetween. In the case where the shape of the real space is indicated by point cloud data in the 3D map, when viewed from the virtual viewpoint CP1, the landmark objects indicating the landmarks may be visible through between the point clouds.

[0159] Therefore, the information processing device 11 arranges a grid at the position of the building existing between the landmark and the virtual viewpoint CP1. By arranging the grid, as Figure 16 shown on the lower side, in the 3D view, the landmark object Obj21 that is not blocked by a building etc. is displayed, but the landmark object blocked by the building is not displayed.

[0160] In addition, the information processing device 11 calculates the distance between the position of the virtual viewpoint and the landmark position, and does not display the landmark object when the distance is equal to or greater than the threshold.

[0161] As described above, by preventing landmarks (landmark objects) that are not used for positioning from being displayed in the 3D view, for example, it is possible to prevent application developers from misidentifying that there are many effective landmarks when viewing landmarks that are not used for positioning.

[0162] <Example representing the image capture direction with the color of the landmark object>

[0163] Figure 17It is a diagram showing an example of a landmark object that represents the image capture direction in color.

[0164] As Figure 17 shown in A of, the shape of the landmark object Obj51 is spherical, and in the spherical surface, the part facing the image capture direction indicated by the arrow is drawn in a light color, and the part not facing the image capture direction is drawn in a dark color. In practice, for example, the part of the spherical surface facing the image capture direction (the part where the normal direction is consistent with the image capture direction) is drawn in green, and as the normal direction of the spherical surface moves away from the image capture direction, the color gradually changes to red.

[0165] As Figure 17 shown in B of, when viewing the building from the front side in the 3D view, since the entire light-colored part is visible on the spherical surface of the landmark object Obj51, the position where the image capture direction faces the virtual viewpoint can be seen.

[0166] As Figure 17 shown in C of, when viewing the building from the side surface side in the 3D view, since a part of the light color is seen on the left side of the spherical surface of the landmark object Obj51, it can be seen that the image capture direction points to the left when viewed from the virtual viewpoint.

[0167] As described above, the part of the landmark object where the normal direction of the landmark object is consistent with the image capture direction of the landmark can be drawn in a color indicating the image capture direction of the landmark. By representing the image capture direction with the color of the landmark object, the image capture direction of the landmark can be confirmed while viewing the 3D view. When the image capture direction is represented by the color of the landmark object, the virtual viewpoint is not used to determine the color of the landmark object. Note that the shape of the landmark object can be a shape other than the spherical shape (for example, the shape of a polyhedron). In the case where the shape of the landmark object is a polyhedron, for example, the surface of the polyhedron where the normal direction of the polyhedron is consistent with the image capture direction of the landmark is drawn in a color indicating the image capture direction of the landmark.

[0168] <Examples of Representing the Image Capture Direction with the Shape of a Landmark Object>

[0169] Figure 18 It is a diagram showing an example of a landmark object that represents the image capture direction in shape.

[0170] As Figure 18 shown in A of, the shape of the landmark object Obj52 is a spherical shape in which a spherical part facing the image capture direction indicated by the arrow protrudes in a protruding shape.

[0171] As Figure 18As shown in B, when viewing the building from the front side in the 3D view, the shadow of the landmark object Obj52 can be seen protruding toward the position side of the virtual viewpoint, so that the image capture direction toward the position of the virtual viewpoint can be seen.

[0172] As Figure 18 shown in C, when viewing the building from the side surface in the 3D view, since it can be seen that the landmark object Obj52 protrudes to the left when viewed from the virtual viewpoint, the image capture direction can be seen to be to the left when viewed from the virtual viewpoint.

[0173] As described above, the landmark object can be drawn in a shape indicating the image capture direction of the landmark. By representing the image capture direction with the shape of the landmark object, the image capture direction of the landmark can be confirmed while viewing the 3D view. In the case where the image capture direction is represented by the shape of the landmark object, the virtual viewpoint is not used to determine the shape of the landmark object.

[0174] <Example of AR display of landmark object>

[0175] Figure 19 is a diagram showing an example of performing AR display of a landmark object.

[0176] Assume that when application developer D1 actually goes to the area where the 3D map is prepared, the tablet terminal 11A, which is the information processing device 11, captures a captured image facing the surrounding environment. In this case, as Figure 19 shown in the bubble chart in, the landmark object Obj displayed in the virtual viewpoint image with the imaging position and imaging direction of the captured image as the virtual viewpoint can be superimposed on the captured image and displayed on the display of the tablet terminal 11A.

[0177] Note that the imaging position and imaging direction of the captured image can be obtained by sensors provided in the tablet terminal 11A, or can be estimated using VPS technology.

[0178] <Example of calculating positioning score>

[0179] A score indicating the ease of positioning (positioning score) can be calculated, and information according to the positioning score can be displayed in the 3D view.

[0180] In the VPS technology, localization often succeeds as many effective landmarks appear in the query image. Therefore, a localization score is calculated based on the number of landmarks appearing in the virtual viewpoint image, the angles formed by the image capture directions of each landmark and the direction of the virtual viewpoint, the distances from the position of the virtual viewpoint to the positions of each landmark, the image feature amounts of the key points corresponding to the landmarks, and so on. For example, the value obtained by adding the angles formed by the image capture directions of each landmark appearing in the virtual viewpoint image and the direction of the virtual viewpoint is set as the landmark score.

[0181] Figure 20 FIG. is an example of a 3D view showing information based on the landmark score.

[0182] For example, when the landmark score is equal to or lower than a threshold, as Figure 20 shown in A of, in the 3D view, the text T1 of "difficult to localize" is displayed superimposed on the virtual viewpoint image.

[0183] In addition, for example, when the landmark score is equal to or less than a threshold, the overall color of the virtual viewpoint image is changed and displayed as Figure 20 the hatched lines in B of. Note that when the landmark score is equal to or less than a threshold, the color of a part of the screen of the 3D view can be changed.

[0184] The overall color of the virtual viewpoint image or the color of a part of the screen of the 3D view can be changed according to the landmark score. For example, as the landmark score decreases, a part of the screen of the 3D view becomes yellow or red. The landmark score can be directly displayed on the screen of the 3D view.

[0185] <<3. Second Embodiment>>

[0186] · Outline of the Second Embodiment

[0187] In the second embodiment of the present technology, a localization score is calculated for each grid obtained by dividing the entire 3D map, and a heat map corresponding to the localization score of each grid is displayed.

[0188] Figure 21 FIG. is an example of a diagram showing a method of generating a heat map.

[0189] As Figure 21 shown on the upper side of, in the information processing apparatus 11, the 3D map viewed from a certain viewpoint (for example, a top-down viewpoint whose field of view includes the entire 3D map) is divided into a plurality of grids, and the application developer sets the direction of the virtual viewpoint (evaluation direction) for each grid. Note that the application developer can set one direction as the evaluation direction in all grids. In Figure 21In the example, the dashed triangle in each grid indicates that the direction from the center of the grid towards the upper right part of the grid is the evaluation direction.

[0190] Based on the evaluation direction set by the application developer, the positioning score of each grid is calculated, and as Figure 21 shown on the lower side of, a heat map is generated that draws the grids in colors corresponding to the positioning scores. For example, grids with high positioning scores are drawn in green, grids with medium positioning scores are drawn in yellow, and grids with low positioning scores are drawn in red.

[0191] The heat map is displayed as superimposed on the top-down image, which shows the appearance (environmental grid) of the 3D map viewed from the top-down viewpoint when the grid is divided. Hereinafter, displaying the heat map corresponding to the top-down image to be superimposed on the top-down image is referred to as the heat map view.

[0192] Figure 22 is a diagram showing an example of a UI for inputting an operation for setting the evaluation direction.

[0193] As Figure 22 shown, for example, an arrow user interface (UI) 101 for inputting an operation in which all the evaluation directions to be set for each grid point in the same direction is superimposed and displayed on the upper right side of the heat map. The application developer can change the evaluation direction by changing the direction of the arrow UI 101 using a mouse operation or a touch operation. For example, the direction of the arrow UI 101 is actually the evaluation direction. The arrow UI 101 can change its direction not only in the horizontal direction but also in the vertical direction.

[0194] By viewing the colors of the grids in the heat map view while operating the direction of the arrow UI 101, the application developer can confirm where and from which direction to capture the query image so that positioning may succeed or positioning may fail.

[0195] · Configuration of the information processing device

[0196] Figure 23 is a block diagram showing a configuration example of an information processing device 11 according to the second embodiment of the present technology. In Figure 23 it, components the same as those in Figure 12 are denoted by the same reference numerals. Redundant descriptions will be appropriately omitted.

[0197] Figure 23 The information processing device 11 in is different from the information processing device 11 in Figure 12 in that a viewpoint position acquisition unit 33, a display color determination unit 34, and a drawing unit 36 are not provided, but an off-screen drawing unit 151, a score calculation unit 152, and a heat map drawing unit 153 are provided.

[0198] Figure 23 The information processing device 11 in Figure 23 is a device that displays a heat map view for checking the ease of positioning of each grid obtained by dividing the entire 3D map.

[0199] The user input unit 22 receives an input of an operation for setting the width of the grid and the evaluation direction. The user input unit 22 provides setting data indicating the width of the grid and the evaluation direction set by the application developer to the control unit 23.

[0200] The image capture direction calculation unit 31 provides the image capture direction of each landmark to the storage unit 24 and stores the image capture direction.

[0201] The off-screen drawing unit 151 divides the 3D map viewed from a certain top-down viewpoint into a plurality of grids with a grid width set by the application developer. The off-screen drawing unit 151 determines the virtual viewpoint of each grid and draws a virtual viewpoint image indicating the appearance of the 3D map (environmental grid) viewed from the virtual viewpoint for each grid. Note that the virtual viewpoint image is drawn off-screen.

[0202] The position of the virtual viewpoint of each grid is, for example, the center of the grid and is a position at a predetermined height from the ground in the environmental grid. The center of the grid is determined based on the grid width set by the application developer. The direction of the virtual viewpoint of each grid is the evaluation direction determined by the application developer.

[0203] The off-screen drawing unit 151 provides the off-screen drawing result of each grid to the storage unit 24 and stores the result.

[0204] The score calculation unit 152 obtains the off-screen drawing result of each grid from the storage unit 24 and calculates the positioning score of each grid based on the off-screen drawing result. For example, the score calculation unit 152 detects landmark objects that appear in the virtual viewpoint image as a result of off-screen drawing, and calculates the positioning score based on the number of detected landmark objects, the image capture direction of the landmarks indicated by the landmark objects, and so on.

[0205] The format of the landmark objects arranged in the 3D space can be any format as long as the score calculation unit 152 can detect the landmark objects. As metadata of the landmark objects, information corresponding to the landmarks (correspondence information indicating the correspondence with key points, image capture directions, etc.) can be saved, or information corresponding to the landmarks can be saved in other formats.

[0206] The score calculation unit 152 provides the positioning score calculated for each grid to the heat map drawing unit 153.

[0207] The heat map drawing unit 153 draws a heat map based on the positioning scores of each grid calculated by the score calculation unit 152. The heat map drawing unit 153 draws a top-down image showing the appearance of the 3D map viewed from the top-down viewpoint when the grid is divided, superimposes the heat map on the top-down image, and provides the superimposed heat map to the display unit 25. The drawing unit 36 also serves as a rendering control unit that presents the top-down image with the heat map superimposed thereon to the application developer.

[0208] The display unit 25 displays the image provided by the heat map drawing unit 153. For example, under the control of the heat map drawing unit 153, the display unit 25 also presents a UI for inputting an operation for setting the evaluation direction, such as an arrow UI.

[0209] · Operations of the information processing device

[0210] Next, the processing performed by the information processing device 11 having the above configuration will be described with reference to Figure 24 the flowchart of.

[0211] The processing of steps S51 to S54 is similar to Figure 13 the processing of steps S1 to S4 in.

[0212] In step S55, the control unit 23 determines whether the setting data has been changed and waits until the setting data is changed. For example, in the case where the application developer changes the grid width and the evaluation direction by operating the user input unit 22, it is determined that the setting data has been changed. In the case of setting the grid width and the evaluation direction for the first time, the processing continues in a similar manner as when the setting data is changed.

[0213] In the case where it is determined in step S55 that the setting data has been changed, in step S56, the off-screen drawing unit 151 performs off-screen drawing on the grid [i].

[0214] In step S57, the score calculation unit 152 detects the landmarks (landmark objects) that appear in the result of the off-screen drawing.

[0215] In step S58, the score calculation unit 152 calculates the positioning score of the grid [i] based on the number of landmarks that appear in the off-screen drawing result, etc.

[0216] In step S59, the score calculation unit 152 determines whether the positioning scores of all grids have been calculated.

[0217] In the case where it is determined in step S59 that the positioning scores of all grids have not been calculated, the score calculation unit 152 increments i (i = i + 1) in step S60. Thereafter, the process returns to step S58, and the process of step S58 is repeatedly executed until the positioning scores of all grids are calculated.

[0218] On the other hand, in the case where it is determined in step S59 that the positioning scores of all the grids have been calculated, in step S61, the heat map drawing unit 153 draws a top view image showing the appearance of the 3D map viewed from the top view point when the grid is divided.

[0219] In step S62, the heat map drawing unit 153 draws the grids on the top view image with colors corresponding to the positioning scores.

[0220] In step S63, the display unit 25 displays the drawing result of the heat map drawing unit 153. After that, the processing of steps S56 to S63 is repeatedly executed each time the setting data is changed.

[0221] As described above, in the information processing apparatus 11, a top view image (first image) is presented to the application developer, and on this top view image (first image), a heat map (second image) indicating the ease of positioning by color is superimposed on each of the grids into which the 3D map viewed from the top view point is divided. By viewing the colors of the grids in the heat map view while changing the evaluation direction, the application developer can confirm where and from which direction it is easy to succeed in positioning or easy to fail in positioning when capturing the query image.

[0222] · Modification

[0223] <Example of UI for inputting an operation for setting the evaluation direction>

[0224] Figure 25 is a diagram showing another example of the UI for inputting an operation for setting the evaluation direction.

[0225] As Figure 25 shown, the target object of interest 201 can be arranged and displayed on the heat map (grid) as a UI that allows the application developer to change the position. The evaluation direction of each grid is set, for example, to the direction from the center of each grid toward the center of the target object of interest (a point in the top view image).

[0226] <Example of setting multiple evaluation directions>

[0227] Multiple evaluation directions can be set for each grid. In this case, the application developer does not need to set the evaluation direction.

[0228] Figure 26 is a diagram showing an example of the multiple evaluation directions set for each grid.

[0229] As Figure 26As indicated by the four dashed triangles in A, for example, four evaluation directions, i.e., up, down, left, and right, are set for a grid. In this case, off-screen rendering is performed on a grid by setting each of the four evaluation directions as the direction of a virtual viewpoint, and four positioning scores are calculated.

[0230] In the case of calculating four positioning scores for each grid, as Figure 26 shown in B, a grid is divided into four regions A101 to A104 on the upper side, lower side, left side, and right side, and the regions A101 to 104 corresponding to the four evaluation directions on the upper side, lower side, left side, and right side are drawn in colors corresponding to the positioning scores.

[0231] <Example of calculating positioning score without performing off-screen rendering>

[0232] In the case of not performing off-screen rendering, only the metadata of the ID of the landmark that appears in the virtual viewpoint image viewed from the virtual viewpoint in grid [i] and the uv coordinates on the virtual viewpoint image can be stored in the storage unit 24, and the positioning score can be calculated based on the metadata of the ID of the landmark and the uv coordinates. For example, the image feature amount and the image capture direction associated with the landmark ID are obtained and used for calculating the positioning score.

[0233] <<Computer>>

[0234] The above series of processing steps can be executed by hardware and can also be executed by software. In the case of executing the series of processing steps by software, the program included in the software is installed on a computer, a general-purpose personal computer, etc. included in dedicated hardware from a program recording medium.

[0235] Figure 27 is a block diagram showing a configuration example of the hardware of a computer that executes the above series of processing by a program.

[0236] The central processing unit (CPU) 501, read-only memory (ROM) 502, and random access memory (RAM) 503 are connected to each other via a bus 504.

[0237] The input / output interface 505 is also connected to the bus 504. An input unit 506 including a keyboard, a mouse, etc. and an output unit 507 including a display, a speaker, etc. are connected to the input / output interface 505. In addition, a storage unit 508 including a hard disk, a non-volatile memory, etc., a communication unit 509 including a network interface, etc., and a drive 510 for driving a removable medium 511 are connected to the input / output interface 505.

[0238] In a computer configured as described above, for example, the CPU 501 loads a program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504, and executes the program to perform the above-described series of processes.

[0239] For example, a program executed by the CPU 501 is recorded in a removable medium 511, or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and then installed in the storage unit 508.

[0240] A program executed by a computer may be a program that performs processes in a time series in the order described in this specification, or may be a program that performs processes in parallel, or may be a program that performs processes at a necessary time such as when a call is made.

[0241] Note that the effects described in this specification are merely examples and not restrictive, and other effects may be provided.

[0242] Embodiments of the present technology are not limited to the above embodiments, and various modifications can be made without departing from the scope of the present technology.

[0243] For example, the present technology may be configured as cloud computing, in which functions are shared by multiple devices via a network to perform processing together.

[0244] In addition, each step described in the above flowchart may be executed by one device, or may be shared and executed by multiple devices.

[0245] In addition, in the case where a step includes multiple processes, the multiple processes included in the one step may be shared and executed by multiple devices in addition to being executed by one device.

[0246] <<Combined Example of Configuration>>

[0247] The present technology may also be configured as follows.

[0248] (1) An information processing apparatus, comprising:

[0249] An image capture direction calculation unit that calculates an image capture direction of a landmark included in a 3D map generated based on a plurality of captured images obtained by capturing an image of the real space;

[0250] A viewpoint acquisition unit that acquires a virtual viewpoint of the user with respect to the 3D map; and

[0251] A drawing unit that draws a first image showing the appearance of the 3D map, and superimposes a second image based on the image capture direction of the landmark and the virtual viewpoint on the first image.

[0252] (2) The information processing apparatus according to (1), wherein,

[0253] The second image is an image that uses a real image captured from a real viewpoint and the 3D map to indicate the ease of estimation of the real viewpoint, and the real viewpoint is a viewpoint in the real space corresponding to the virtual viewpoint.

[0254] (3) The information processing apparatus according to (2), wherein,

[0255] The first image is a virtual viewpoint image showing the appearance of the 3D map viewed from the virtual viewpoint, and

[0256] The second image includes an object indicating the landmark.

[0257] (4) The information processing apparatus according to (3), wherein,

[0258] The object is drawn in a color based on the image capture direction of the landmark.

[0259] (5) The information processing apparatus according to (4), wherein,

[0260] The object is drawn in a color corresponding to the angle formed by the image capture direction of the landmark and the direction of the virtual viewpoint.

[0261] (6) The information processing apparatus according to claim (4), wherein,

[0262] The portion of the object whose normal direction is consistent with the image capture direction of the landmark is drawn in a color indicating the image capture direction of the landmark.

[0263] (7) The information processing apparatus according to (3), wherein,

[0264] The object is drawn in a shape indicating the image capture direction of the landmark.

[0265] (8) The information processing apparatus according to any one of (3) to (7), wherein,

[0266] The drawing unit superimposes the object on the real image.

[0267] (9) The information processing apparatus according to any one of (3) to (8) further includes:

[0268] A presentation control unit that presents to the user information on a score indicating the ease of estimation of the real viewpoint and the virtual viewpoint image superimposed with the object.

[0269] (10) The information processing apparatus according to (2), wherein,

[0270] the first image is a top - down image showing the appearance of the entire area of the 3D map viewed from a top - down viewpoint, and

[0271] the second image is a heat map that indicates the ease of estimating the true viewpoint in color for each grid obtained by dividing the top - down image.

[0272] (11) The information processing apparatus according to (10) further includes:

[0273] a score calculation unit that calculates, for each grid, a score indicating the ease of estimating the true viewpoint based at least on the image capture direction of the landmark and the virtual viewpoint, wherein,

[0274] in the heat map, the grid is drawn in a color corresponding to the score.

[0275] (12) The information processing apparatus according to (11), wherein,

[0276] the score calculation unit calculates scores corresponding to each direction of a plurality of virtual viewpoints set for each grid based on the directions of the plurality of virtual viewpoints, and

[0277] in the heat map, the area of the grid is divided according to the directions of the plurality of virtual viewpoints and drawn in a color corresponding to the corresponding score.

[0278] (13) The information processing apparatus according to any one of (10) to (12) further includes:

[0279] a presentation control unit that presents to the user a UI for inputting an operation to direct all directions of the virtual viewpoints set for each grid to the same direction and a top - down image overlaid with the heat map.

[0280] (14) The information processing apparatus according to any one of (10) to (13) further includes:

[0281] a presentation control unit that presents to the user a UI for inputting an operation to direct the direction of the virtual viewpoints set for each grid to a point in the top - down image and a top - down image overlaid with the heat map.

[0282] (15) An information processing method executed by an information processing apparatus, including:

[0283] calculating an image capture direction of a landmark included in a 3D map generated based on a plurality of captured images obtained by capturing an image of a real space;

[0284] Obtain a virtual viewpoint of the user with respect to the 3D map; and

[0285] Draw a first image showing the appearance of the 3D map, and superimpose a second image based on the image capture direction of the landmark and the virtual viewpoint on the first image.

[0286] (16) A program for causing a computer to execute the following processes:

[0287] Calculate the image capture direction of a landmark included in a 3D map generated based on a plurality of captured images obtained by capturing an image of the real space;

[0288] Obtain a virtual viewpoint of the user with respect to the 3D map; and

[0289] Draw a first image showing the appearance of the 3D map, and superimpose a second image based on the image capture direction of the landmark and the virtual viewpoint on the first image.

[0290] List of reference numerals

[0291] 11 Information processing device

[0292] 21 3D map storage unit

[0293] 22 User input unit

[0294] 23 Control unit

[0295] 24 Storage unit

[0296] 25 Display unit

[0297] 31 Image capture direction calculation unit

[0298] 32 Grid arrangement unit

[0299] 33 Viewpoint position acquisition unit

[0300] 34 Display color determination unit

[0301] 35 Object arrangement unit

[0302] 36 Drawing unit

[0303] 151 Off-screen drawing unit

[0304] 152 Score calculation unit

[0305] 153 Heat map drawing unit

Claims

1. An information processing apparatus, comprising: an image capture direction calculation unit that calculates an image capture direction of a landmark included in a 3D map generated based on a plurality of captured images obtained by capturing an image of a real space; a viewpoint acquisition unit that acquires a virtual viewpoint of the user with respect to the 3D map; and a rendering unit that renders a first image showing an appearance of the 3D map and superimposes a second image based on the image capture direction of the landmark and the virtual viewpoint on the first image.

2. The information processing apparatus according to claim 1, wherein the second image is an image indicating an ease of estimation of the real viewpoint using a real image captured from the real viewpoint and the 3D map, the real viewpoint being a viewpoint in the real space corresponding to the virtual viewpoint.

3. The information processing apparatus according to claim 2, wherein the first image is a virtual viewpoint image showing an appearance of the 3D map viewed from the virtual viewpoint, and the second image includes an object indicating the landmark.

4. The information processing apparatus according to claim 3, wherein the object is drawn in a color based on the image capture direction of the landmark.

5. The information processing apparatus according to claim 4, wherein the object is drawn in a color corresponding to an angle formed by the image capture direction of the landmark and the direction of the virtual viewpoint.

6. The information processing apparatus according to claim 4, wherein a portion of the object whose normal direction is consistent with the image capture direction of the landmark is drawn in a color indicating the image capture direction of the landmark.

7. The information processing apparatus according to claim 3, wherein the object is drawn in a shape indicating the image capture direction of the landmark.

8. The information processing apparatus according to claim 3, wherein the rendering unit superimposes the object on the real image.

9. The information processing apparatus according to claim 3, further comprising: a presentation control unit that presents to the user information on a score indicating an ease of estimation of the real viewpoint and a virtual viewpoint image on which the object is superimposed.

10. The information processing apparatus according to claim 2, wherein the first image is a top view image showing an appearance of the entire area of the 3D map viewed from a top view point, and the second image is a heat map indicating an ease of estimation of the real viewpoint in color for each grid obtained by dividing the top view image.

11. The information processing apparatus according to claim 10, further comprising: a score calculation unit that calculates a score indicating an ease of estimation of the real viewpoint for each grid based at least on the image capture direction of the landmark and the virtual viewpoint, wherein in the heat map, the grid is drawn in a color corresponding to the score.

12. The information processing apparatus according to claim 11, wherein the score calculation unit calculates a score corresponding to each direction of a plurality of virtual viewpoints based on directions of a plurality of virtual viewpoints set for each grid, and In the heat map, regions of a grid are divided according to the directions of the plurality of virtual viewpoints and are drawn in colors corresponding to the corresponding scores.

13. The information processing apparatus according to claim 10, further comprising: a presentation control unit that presents to a user a UI for inputting an operation in which directions of virtual viewpoints to be set for each grid point to the same direction, and a top view image overlaid with the heat map.

14. The information processing apparatus according to claim 10, further comprising: a presentation control unit that presents to a user a UI for inputting an operation in which directions of virtual viewpoints to be set for each grid point to a point in the top view image, and a top view image overlaid with the heat map.

15. An information processing method executed by an information processing apparatus, comprising: calculating an image capture direction of a landmark included in a 3D map generated based on a plurality of captured images obtained by capturing an image of a real space; acquiring a virtual viewpoint of the user with respect to the 3D map; and and drawing a first image showing an appearance of the 3D map, and superimposing a second image based on the image capture direction of the landmark and the virtual viewpoint on the first image.

16. A program for causing a computer to execute the following processing: calculating an image capture direction of a landmark included in a 3D map generated based on a plurality of captured images obtained by capturing an image of a real space; acquiring a virtual viewpoint of the user with respect to the 3D map; and drawing a first image showing an appearance of the 3D map, and superimposing a second image based on the image capture direction of the landmark and the virtual viewpoint on the first image.

Images