Information processing device, information processing method, and program

Abstract

The present technology pertains to an information processing device, an information processing method, and a program that make it possible to provide a successfully estimated image capture position and image capture direction and an unsuccessfully estimated image capture position and image capture direction when estimating the image capture position and image capture direction. The receiving unit acquires an image for evaluation and image capture information for evaluation. An estimation unit estimates an image capture position and an image capture direction for evaluation by matching feature points of a 3D map with feature points of an image for evaluation. An evaluation map creation unit creates an evaluation map that registers success / failure-related information on whether or not the estimation is successful in association with a capture position and a capture direction of an image for evaluation. If the estimation succeeds, the evaluation map updating unit updates success / failure related information associated with the estimated image capture position and image capture direction, and if the estimation fails, the evaluation map updating unit updates success / failure related information associated with the estimated image capture position and image capture direction. If so, an evaluation map updating unit updates success / failure related information associated with the image capture position and the image capture direction represented by the image capture information for evaluation. This technology can be applied, for example, to a server or the like.

Description

Technical Field

[0001] The present technology relates to an information processing device, an information processing method, and a program, and more specifically, to an information processing device, an information processing method, and a program that are capable of providing an imaging position and an imaging direction in a case where the estimation is successful and a case where the estimation fails, and in a case where the imaging position and the imaging direction are estimated. Background Art

[0002] In recent years, technologies such as visual positioning systems (VPS) have been developed that use 3D maps to estimate (locate) imaging positions and imaging directions from captured images. Generation of 3D maps can be achieved, for example, by executing an open source called COLMAP (see, for example, non-patent document 1).

[0003] Reference List

[0004] Non-patent literature

[0005] Non-patent literature 1: Johannes L. Schonberger, Jan-Michael Frahm, "Structure-from-Motion Revisited", 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, p.4104-4113 Summary of the invention

[0006] Problems to be solved by the present invention

[0007] However, it is not considered that multiple evaluation observations of the VPS are integrated and presented to the evaluator in an easily understandable manner. Therefore, it is difficult for the evaluator to determine whether a 3D map for the VPS is accurately created.

[0008] For example, in the case where the evaluator evaluates the VPS, it is necessary to identify the imaging position and imaging direction where the evaluation is successful and the imaging position and imaging direction where the evaluation fails. Therefore, in the case where the imaging position and imaging direction are estimated by the VPS or the like, a method of providing the imaging position and imaging direction in the case where the estimation is successful and in the case where the estimation fails is required, but this demand cannot be fully met.

[0009] The present technology has been made in view of such circumstances, and an object thereof is to provide an imaging position and an imaging direction in a case where the estimation is successful and in a case where the imaging position and the imaging direction are estimated.

[0010] Solution to the problem

[0011] An information processing device or program according to one aspect of the present technology is an information processing device, including: an acquisition unit that acquires an evaluation image captured by an imaging device and imaging information for evaluation, the imaging information for evaluation being imaging information indicating an imaging position and an imaging direction of the evaluation image; an estimation unit that estimates the imaging position and the imaging direction of the evaluation image by performing matching between feature points of a 3D map and feature points of the evaluation image acquired by the acquisition unit; a creation unit that creates an evaluation map in which success / failure related information regarding whether the estimation of the estimation unit is successful is registered in association with the imaging position and the imaging direction of the evaluation image; and an evaluation map updating unit that, in a case where the estimation performed by the estimation unit is successful, updates the success / failure related information corresponding to the imaging position and the imaging direction estimated by the estimation unit, and in a case where the estimation performed by the estimation unit fails, updates the success / failure related information corresponding to the imaging position and the imaging direction indicated by the imaging information for evaluation, or updates a program for causing a computer to function as an information processing device.

[0012] An information processing method according to one aspect of the present technology, wherein the information processing device includes: an acquisition step of acquiring an evaluation image captured by an imaging device and imaging information for evaluation, the imaging information for evaluation being imaging information representing an imaging position and an imaging direction of the evaluation image; an estimation step of estimating the imaging position and the imaging direction of the evaluation image by performing matching between feature points of a 3D map and feature points of the evaluation image acquired by the acquisition step; a creation step of creating an evaluation map in which success / failure related information regarding whether the evaluation of the estimation step is successful is registered in association with the imaging position and the imaging direction of the evaluation image; and an evaluation map updating step of updating, in a case where the estimation performed by the estimation step is successful, the success / failure related information corresponding to the imaging position and the imaging direction estimated by the estimation step, and in a case where the estimation performed by the estimation step fails, updating the success / failure related information corresponding to the imaging position and the imaging direction represented by the imaging information for evaluation.

[0013] In one aspect of the present technology, an evaluation image captured by an imaging device and imaging information for evaluation are acquired, the imaging information for evaluation being imaging information indicating an imaging position and an imaging direction of the evaluation image, the imaging position and the imaging direction of the evaluation image are estimated by performing matching between feature points of a 3D map and feature points of the evaluation image, success / failure related information regarding whether the estimation is successful is registered in association with the imaging position and the imaging direction of the evaluation image, success / failure related information corresponding to the imaging position and the imaging direction is updated if the estimation is successful, and success / failure related information corresponding to the imaging position and the imaging direction indicated by the imaging information for evaluation is updated if the estimation fails.

[0014] It should be noted that the information processing device according to one aspect of the present technology may be an independent device or may be a module incorporated in another device.

[0015] In order to realize the information processing apparatus according to one aspect of the present technology, a program executed by a computer may be provided by being transmitted via a transmission medium or by being recorded on a recording medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a block diagram showing a configuration example of a first embodiment of an information processing system including an information processing device to which the present technology is applied.

[0017] Figure 2 It is shown Figure 1 A block diagram of a configuration example of an imaging device in FIG.

[0018] Figure 3 It is shown Figure 1 A block diagram of an example configuration of a server in FIG.

[0019] Figure 4 It is shown Figure 1 A block diagram of a configuration example of a mobile terminal in FIG.

[0020] Figure 5 is a diagram showing a first configuration example of the evaluation screen.

[0021] Figure 6 is a diagram showing the effect of the evaluation screen.

[0022] Figure 7 is a flowchart showing the creation process.

[0023] Figure 8 is a flowchart illustrating an evaluation map update process.

[0024] Fig. 9 : is a block diagram showing a configuration example of a server in a second embodiment of an information processing system including an information processing device to which the present technology is applied.

[0025] Fig.10 is a diagram illustrating a second configuration example of the evaluation screen.

[0026] Fig.11 It is shown Fig.10 An illustration of a detailed configuration example of a circle in FIG.

[0027] Fig.12 : is a block diagram showing a configuration example of a server in a third embodiment of an information processing system including an information processing device to which the present technology is applied.

[0028] Fig.13is a block diagram showing a configuration example of a mobile terminal in a third embodiment of an information processing system including an information processing device to which the present technology is applied.

[0029] Fig.14 is a diagram illustrating a third configuration example of the evaluation screen.

[0030] Fig.15 is a flowchart showing a 3D map update process.

[0031] Fig.16 is a diagram showing an example of the shape of a mesh.

[0032] Fig.17 is a diagram showing an example of distribution of meshes.

[0033] Fig.18 is a diagram showing another example of distribution of meshes.

[0034] Fig.19 is a diagram showing another example of the shape of a mesh.

[0035] Fig. 20 is a block diagram showing a configuration example of hardware of a computer. DETAILED DESCRIPTION

[0036] Hereinafter, a mode for carrying out the present technology (hereinafter, referred to as an embodiment) will be described. Note that the description is given in the following order.

[0037] 1. First Embodiment (Information Processing System for Evaluating Estimation of Each Imaging Position and Imaging Direction)

[0038] 2. Second Embodiment (Information Processing System for Evaluating Estimation of Each Imaging Position in a Predetermined Range and Imaging Direction in a Predetermined Range)

[0039] 3. Third Embodiment (Information Processing System for Updating 3D Model)

[0040] 4. Grid

[0041] 5. Computer

[0042] <1. First Embodiment>

[0043] Figure 1 : is a block diagram showing a configuration example of a first embodiment of an information processing system including a server as an information processing device to which the present technology is applied.

[0044] Figure 1The information processing system 10 in the embodiment includes an imaging device 11, a server 12, and a mobile terminal 13. The information processing system 10 creates a 3D map used in a VPS, and displays an evaluation screen for evaluating the VPS using the 3D map.

[0045] Specifically, for example, when a provider who provides a VPS creates a 3D map, the imaging device 11 is used. The imaging device 11 images a predetermined area at a plurality of imaging positions and a plurality of imaging directions determined by the provider. The imaging device 11 transmits a result image group including a plurality of captured images having different imaging positions and imaging directions and meta information indicating the imaging position of each captured image of the image group to the server 12.

[0046] The server 12 is a device that executes VPS. The server 12 receives an image group and meta information transmitted from the imaging device 11. The server 12 creates a 3D map (3D model) of a predetermined area using the image group and meta information. For example, by executing an open source called COLMAP described in non-patent document 1, generation of a 3D map can be achieved. COLMAP is a pipeline of structure-from-motion (SfM) and multi-view stereo (MVS).

[0047] The server 12 also receives an evaluation image and imaging information for evaluation, the evaluation image being an image for evaluating the VPS transmitted from the mobile terminal 13, and imaging information for evaluation being imaging information indicating the imaging position and imaging direction of the evaluation image. The server 12 estimates the imaging position and imaging direction of the evaluation image (the posture of the mobile terminal 13 at the time of imaging) using the 3D map. The server 12 generates display control information for displaying the evaluation screen based on the estimation result, and transmits the display control information to the mobile terminal 13.

[0048] For example, when a VPS is evaluated by a provider who provides the VPS, the mobile terminal 13 is used. The mobile terminal 13 (imaging device) captures an evaluation image and transmits the evaluation image and imaging information used for evaluation to the server 12. The mobile terminal 13 receives display control information transmitted from the server 12, and displays an evaluation screen based on the display control information.

[0049] <Configuration Example of Imaging Device>

[0050] Figure 2 It is shown Figure 1 A block diagram of a configuration example of the imaging device 11 in FIG.

[0051] Figure 2 The imaging device 11 in the embodiment includes an imaging unit 31 , a detection unit 32 , a storage unit 33 , a transmission unit 34 , a touch panel 35 , and a control unit 36 ​​.

[0052] The imaging unit 31 images a predetermined area with the current position of the imaging device 11 as the imaging position and the current direction of the imaging device 11 as the imaging direction. The imaging unit 31 provides the generated captured image to the touch panel 35 so that the touch panel 35 displays the captured image, and provides the captured image to the storage unit 33 so that the storage unit 33 stores the captured image.

[0053] The detection unit 32 includes a global positioning system (GPS) sensor, a gravity sensor, a self-position estimation unit that estimates a self-position using a simultaneous localization and mapping (SLAM) technology, an acquisition unit that acquires setting information and semantic information input by a provider, a depth sensor, etc. The detection unit 32 acquires, for each captured image, meta information including an imaging position detected by the GPS sensor, gravity information indicating a detection result of the gravity sensor, a self-position estimation result of the self-position estimation unit, setting information, semantic information, depth information indicating a detection result of the depth sensor, etc. The detection unit 32 provides the meta information to the storage unit 33 so that the storage unit 33 stores the meta information associated with the captured image.

[0054] The storage unit 33 stores the captured image supplied from the imaging unit 31 and the meta information of the captured image supplied from the detection unit 32 in association with each other.

[0055] The transmission unit 34 reads the image group and the meta information of each captured image of the image group from the storage unit 33 , and transmits the image group and the meta information to the server 12 .

[0056] The touch panel 35 includes an input unit 41 and a display unit 42. The input unit 41 receives a touch operation performed by the provider and provides an operation signal indicating the touch operation to the control unit 36. The display unit 42 displays a captured image or the like provided from the imaging unit 31. For example, the provider selects an image for creating a 3D map by performing a touch operation while viewing the captured image displayed on the display unit 42. For example, the provider moves the imaging device 11 to a desired imaging position and performs a touch operation in a desired imaging direction, thereby issuing an imaging command. These touch operations are received by the input unit 41.

[0057] The control unit 36 ​​controls each unit of the imaging device 11. For example, the control unit 36 ​​controls the transmission unit 34 based on the operation signal supplied from the input unit 41, and causes the server 12 to transmit an image group including a captured image selected by the provider as an image for creating a 3D map, and meta information of the image group. For example, the control unit 36 ​​causes the imaging unit 31 to perform imaging at a timing desired by the provider based on the operation signal supplied from the input unit 41.

[0058] <Server Configuration Example>

[0059] Figure 3 It is shown Figure 1 A block diagram of a configuration example of the server 12 in FIG.

[0060] Figure 3 The server 12 includes a receiving unit 51, a 3D map creating unit 52, a 3D map storage unit 53, a detection unit 54, an evaluation map creating unit 55, an evaluation map storage unit 56, a receiving unit 57, an estimating unit 58, an evaluation map updating unit 59, a generating unit 60 and a transmitting unit 61.

[0061] The receiving unit 51 receives the image group and meta information transmitted from the transmission unit 34 of the imaging device 11 , and supplies the image group and meta information to the 3D map creating unit 52 .

[0062] The 3D map creating unit 52 generates a 3D map using the image group and the meta information supplied from the receiving unit 51. The 3D map creating unit 52 supplies the 3D map to the 3D map storing unit 53.

[0063] The 3D map storage unit 53 stores the 3D map and the like supplied from the 3D map creation unit 52 .

[0064] The detection unit 54 reads the 3D map from the 3D map storage unit 53. The detection unit 54 detects the ground area from the 3D map. For example, there is a method using segmentation and principal component analysis (PCA) as a method of detecting the ground area.

[0065] Note that the detection unit 54 may also read meta information from the 3D map storage unit 53 and detect the ground area from the 3D map using the meta information. In this case, the detection unit 54 detects the ground area from the meta information using, for example, gravity information, setting information (such as body information of the provider), semantic information, depth information, etc. Alternatively, for example, the detection unit 54 detects a surface formed by connecting the self-position estimation results on the 3D map as the ground area based on the trajectory of the self-position estimation results included in the meta information.

[0066] In the case where an inertial measurement unit (IMU) is included in the detection unit 32 of the imaging device 11 and the angular velocity and acceleration of the imaging device 11 detected by the IMU are included in the meta information, the detection unit 54 may also detect the ground area using the angular velocity and acceleration and the PCA. The detection unit 54 may also detect the ground area using a 3D model of an object included in the 3D map.

[0067] The detection unit 54 determines the posture (direction) of the 3D map relative to the real world coordinate system based on the ground area. The detection unit 54 updates the posture of the 3D map stored in the 3D map storage unit 53 based on the posture. The detection unit 54 provides the ground area information indicating the position of the ground area on the 3D map to the evaluation map creation unit 55.

[0068] The evaluation map creation unit 55 reads the 3D map from the 3D map storage unit 53. The evaluation map creation unit 55 creates an evaluation map in the ground area on the 3D map based on the ground area information provided from the detection unit 54. The evaluation map is a map for registering success-or-failure related information about whether the estimation of the evaluation image by the estimation unit 58 is successful or not, in association with the imaging position and imaging direction of the evaluation image. In the first embodiment, the success-or-failure related information is success-or-failure information indicating whether the evaluation is successful, that is, whether the evaluation is successful or failed. The evaluation map creation unit 55 provides the evaluation map to the evaluation map storage unit 56.

[0069] The evaluation map storage unit 56 stores the evaluation map supplied from the evaluation map creation unit 55 .

[0070] The receiving unit 57 (acquisition unit) receives (acquires) the evaluation image and the imaging information for evaluation transmitted from the mobile terminal 13 , and supplies the evaluation image and the imaging information for evaluation to the estimating unit 58 .

[0071] The estimation unit 58 reads the 3D map from the 3D map storage unit 53. The estimation unit 58 estimates the imaging position and imaging direction of the evaluation image by performing matching between the feature points of the 3D map and the feature points of the evaluation image provided from the receiving unit 57.<URL:https: / / www.graphics.rwth-aachen.de / software / image-localization / > The software provided in can realize this estimation.

[0072] In the case of successful estimation, the estimation unit 58 supplies the estimated imaging information indicating the estimated imaging position and imaging direction to the evaluation map updating unit 59 and the generating unit 60. In the case of failed estimation, the estimation unit 58 supplies the imaging information for evaluation supplied from the receiving unit 57 to the evaluation map updating unit 59 and the generating unit 60.

[0073] The evaluation map updating unit 59 reads the evaluation map from the evaluation map storage unit 56. The evaluation map updating unit 59 acquires the estimated imaging information or the imaging information for evaluation provided from the estimation unit 58. The evaluation map updating unit 59 updates the evaluation map by updating the success / failure information corresponding to the imaging position and the imaging direction indicated by the estimated imaging information or the imaging information for evaluating the evaluation map.

[0074] Specifically, the evaluation map updating unit 59 updates the success or failure information corresponding to the imaging position and the imaging direction represented by the estimated imaging information of the evaluation map to success or failure information indicating that the estimation was successful. The evaluation map updating unit 59 updates the success or failure information corresponding to the imaging position and the imaging direction represented by the imaging information used for the evaluation of the evaluation map to success or failure information indicating that the estimation failed. The evaluation map updating unit 59 provides the updated evaluation map to the evaluation map storage unit 56, and updates the evaluation map stored in the evaluation map storage unit 56.

[0075] The generation unit 60 reads the evaluation map stored in the evaluation map storage unit 56. The generation unit 60 acquires the estimated imaging information or the imaging information for evaluation provided from the estimation unit 58. The generation unit 60 generates display control information based on the evaluation map and the estimated imaging information or the imaging information for evaluation. The display control information is information for displaying direction information indicating each imaging direction and success / failure information registered in the evaluation map in association with the imaging position at each imaging position registered in the evaluation map on the evaluation image. The generation unit 60 supplies the display control information to the transmission unit 61.

[0076] The transmission unit 61 transmits the display control information supplied from the generation unit 60 to the mobile terminal 13 .

[0077] <Configuration example of mobile terminal>

[0078] Figure 4 It is shown Figure 1 13 is a block diagram of a configuration example of the mobile terminal 13 in FIG.

[0079] Figure 4 The mobile terminal 13 in the embodiment includes an imaging unit 71 , a detecting unit 72 , a transmitting unit 73 , a receiving unit 74 , a generating unit 75 , a touch panel 76 and a control unit 77 .

[0080] The imaging unit 71 images a predetermined area with the current position of the mobile terminal 13 as the imaging position and the current direction of the mobile terminal 13 as the imaging direction. The imaging unit 71 provides the captured image captured in real time as the camera view image to the generation unit 75. The imaging unit 71 provides the captured image at a predetermined timing to the transmission unit 73 as an evaluation image.

[0081] The detection unit 72 includes a GPS sensor, a gravity sensor, a self-position estimation unit, and the like. For example, there is a method of estimating the current position by a movement amount, a movement direction, and the like according to past positions and time constraints as a method of estimating the current position by the self-position estimation unit. The detection unit 72 detects the imaging position and the imaging direction of each captured image. The detection unit 72 supplies imaging information for evaluation to the transmission unit 73 based on the detection result, and supplies camera view imaging information as imaging information of the camera view image to the generation unit 75.

[0082] The transmission unit 73 transmits the evaluation image supplied from the imaging unit 71 and the imaging information for evaluation supplied from the detection unit 72 to the server 12 .

[0083] The receiving unit 74 receives the display control information transmitted from the server 12 and supplies the display control information to the generating unit 75. The generating unit 75 generates an evaluation screen in which the direction information and the success / failure information are superimposed on the imaging position of each evaluation image of the camera view image supplied from the imaging unit 71, based on the camera view imaging information supplied from the detecting unit 72 and the display control information supplied from the receiving unit 74. The generating unit 75 supplies the evaluation screen to the touch panel 76.

[0084] The touch panel 76 includes an input unit 81 and a display unit 82. The input unit 81 receives a touch operation of the provider and supplies an operation signal indicating the touch operation to the control unit 77. The display unit 82 displays an evaluation screen or the like supplied from the generation unit 75. For example, the provider performs a touch operation while viewing a camera view image displayed by the display unit 82, thereby selecting the camera view image displayed at that time as an evaluation image. The touch operation is received by the input unit 81.

[0085] The control unit 77 controls each unit of the mobile terminal 13. For example, the control unit 77 controls the imaging unit 71 and the detection unit 72 based on the operation signal supplied from the input unit 81, and supplies the evaluation image selected by the provider and the imaging information for evaluating the evaluation image to the transmission unit 73.

[0086] <Configuration example of evaluation screen>

[0087] Figure 5 is a diagram showing a configuration example of an evaluation screen.

[0088] exist Figure 5 On the evaluation screen 90, a camera view image 91 and an evaluation image button 92 are displayed. At the imaging position of each evaluation image of the camera view image 91, one of two types of cones (view cones) 91a, 91b is displayed as direction information and success information.

[0089] Specifically, in each of the view cones 91a and 91b, one vertex is set at the imaging position, and the direction of the vertical line from the vertex to the bottom surface represents the imaging direction. The type of the view cones 91a and 91b represents the success or failure of the estimation by the estimation unit 58. Figure 5 In the example of , the types of view cones 91a and 91b are line types. The view cone 91a with a thick solid line indicates that the estimation by the estimation unit 58 succeeds, and the view cone 91b with a thin solid line indicates that the estimation by the estimation unit 58 fails.

[0090] exist Figure 5 In the example of , the imaging position of the mobile terminal 13 moves during the estimation by the estimation unit 58, and based on the imaging information for evaluating the evaluation image corresponding to the estimation, the view cone 91c is also displayed on the camera view image 91. This view cone 91c is different from the view cones 91a and 91b in that the line type is a dotted line, and is configured similarly to the view cones 91a and 91b in other respects. After the estimation ends, the view cone 91c is changed to the view cone 91a in the case of successful estimation, and the view cone 91c is changed to the view cone 91b in the case of failed estimation.

[0091] Note that the type of the cones 91 a to 91 c may be, for example, a color type other than a line.

[0092] In the case where the camera view image 91 is selected as the evaluation image, the provider touches and operates the evaluation image button 92. When the provider performs a touch operation on the evaluation image button 92, the touch operation is received by the input unit 81, and an operation signal indicating the touch operation is provided to the control unit 77. As a result, under the control of the control unit 77, the camera view image 91 displayed when the provider performs the touch operation is transmitted to the server 12 as the evaluation image. In addition, under the control of the control unit 77, the camera view imaging information of the camera view image 91 is transmitted to the server 12 as imaging information for evaluation.

[0093] <Description of the effect>

[0094] Figure 6 is a diagram showing the effect of the evaluation screen 90 .

[0095] like Figure 6 As shown, on the evaluation screen 100 , in which, in the case of a successful evaluation, only the viewing cone 91 a is superimposed on the camera view image 91 , the provider cannot recognize the imaging position and imaging direction where the evaluation failed.

[0096] On the other hand, on the evaluation screen 90, not only the view cone 91a in the case of successful estimation but also the view cone 91b in the case of failed estimation is superimposed on the camera view image 91. Therefore, the provider can recognize the imaging position and the imaging direction in both the case of successful estimation and the case of failed estimation.

[0097] <Description of Server Processing>

[0098] Figure 7 : is a flowchart showing a creation process of creating a 3D map by the server 12. This creation process is started when an image group and meta information are transmitted from the imaging device 11, for example.

[0099] exist Figure 7 In step S11, the receiving unit 51 receives the image group and meta information transmitted from the transmission unit 34 and provides the image group and meta information to the 3D map creation unit 52. In step S12, the 3D map creation unit 52 creates a 3D map using the image group and meta information received by the process in step S11. In step S13, the 3D map creation unit 52 provides the 3D map generated by the process in step S12 to the 3D map storage unit 53 so that the 3D map storage unit 53 stores the 3D map.

[0100] In step S14, the detection unit 54 reads the 3D map stored by the process in step S13, and detects the ground area from the 3D map. The detection unit 54 updates the posture of the 3D map stored in the 3D map storage unit 53 based on the ground area information. The detection unit 54 provides the ground area information to the evaluation map creation unit 55.

[0101] In step S15, the evaluation map creation unit 55 reads the 3D map from the 3D map storage unit 53, and creates an evaluation map in the ground area on the 3D map based on the ground area information supplied from the detection unit 54. In step S16, the evaluation map creation unit 55 supplies the evaluation map generated by the process in step S15 to the evaluation map storage unit 56 so that the evaluation map storage unit 56 stores the evaluation map. Then, the process ends.

[0102] Figure 8 : is a flowchart showing an evaluation map updating process in which the server 12 updates the evaluation map.

[0103] exist Figure 8 In step S30, the receiving unit 57 determines whether the evaluation image and the imaging information for evaluation have been transmitted from the mobile terminal 13. In the case where it is determined in step S30 that the evaluation image and the imaging information for evaluation have not been transmitted, the process waits until the evaluation image and the imaging information for evaluation are transmitted.

[0104] On the other hand, in the case where it is determined in step S30 that the evaluation image and the imaging information for evaluation have been transmitted, the process proceeds to step S31. In step S31, the receiving unit 57 receives the evaluation image and the imaging information for evaluation transmitted from the mobile terminal 13, and provides the evaluation image and the imaging information for evaluation to the estimation unit 58.

[0105] In step S32 , the estimation unit 58 estimates the imaging position and imaging direction of the evaluation image by performing matching between the feature points of the 3D map stored in the 3D map storage unit 53 and the feature points of the evaluation image received through the process in step S31 .

[0106] In step S33, the estimation unit 58 determines whether the estimation is successful. Specifically, for example, the estimation unit 58 calculates the likelihood of the estimation result. When the likelihood is greater than a predetermined value, the estimation unit 58 determines that the estimation is successful, and when the likelihood is less than a predetermined value, the estimation unit 58 determines that the estimation is unsuccessful.

[0107] In the event that determination is made in step S33 that the estimation is successful, the estimation unit 58 supplies the estimated imaging information to the evaluation map updating unit 59 and the generation unit 60 , and the process proceeds to step S34 .

[0108] In step S34, the evaluation map updating unit 59 reads the evaluation map from the evaluation map storage unit 56 and updates the evaluation map based on the estimated imaging information. The evaluation map updating unit 59 supplies the updated evaluation map to the evaluation map storage unit 56 and updates the evaluation map stored in the evaluation map storage unit 56.

[0109] In step S35, the generation unit 60 generates display control information based on the evaluation map updated in step S34 and the estimated imaging information supplied from the estimation unit 58. The generation unit 60 supplies the display control information to the transmission unit 61, and the process proceeds to step S38.

[0110] On the other hand, in a case where it is determined in step S33 that the estimation is unsuccessful, that is, in a case where it is determined that the estimation has failed, the estimation unit 58 provides the imaging information for evaluation received through the processing in step S31 to the evaluation map updating unit 59 and the generating unit 60, and the processing proceeds to step S36.

[0111] In step S36, the evaluation map updating unit 59 reads the evaluation map from the evaluation map storage unit 56 and updates the evaluation map based on the imaging information for evaluation. The evaluation map updating unit 59 supplies the updated evaluation map to the evaluation map storage unit 56 and updates the evaluation map stored in the evaluation map storage unit 56.

[0112] In step S37, the generation unit 60 generates display control information based on the evaluation map updated in step S36 and the imaging information for evaluation supplied from the estimation unit 58. The generation unit 60 supplies the display control information to the transmission unit 61, and the process proceeds to step S38.

[0113] In step S38 , the transmission unit 61 transmits the display control information generated in step S35 or S37 to the mobile terminal 13 .

[0114] In step S39, the server 12 determines whether to end the evaluation map update process. For example, in the case where the provider gives a command to end the evaluation by performing a touch operation on the touch panel 76, and as a result, under the control of the control unit 77, the command to end the evaluation is transmitted from the transmission unit 73 to the server 12, the server 12 determines to end the evaluation map update process. It should be noted that the server 12 may determine to end the evaluation map update process in the case where the number of pieces of success / failure information registered on the evaluation map is a predetermined number or more.

[0115] In a case where it is determined in step S39 that the evaluation map update process has not ended, the process returns to step S30 and the subsequent processes are repeated.

[0116] On the other hand, in a case where it is determined in step S39 that the evaluation map update process ends, the process ends.

[0117] As described above, the server 12 acquires an evaluation image and imaging information for evaluation, estimates an imaging position and an imaging direction of the evaluation image, creates an evaluation map in which success / failure related information is registered in association with the imaging position and the imaging direction of the evaluation image, updates the success / failure related information corresponding to the estimated imaging information in the case of successful estimation, and updates the success / failure related information corresponding to the imaging information for evaluation in the case of failed estimation. Thus, in the case of estimating the imaging position and the imaging direction, the imaging position and the imaging direction can be provided in the case of successful estimation and in the case of failed estimation.

[0118] <2. Second Embodiment>

[0119] <Server Configuration Example>

[0120] The configuration of the second embodiment of the information processing system including a server as an information processing device to which the present technology is applied, in addition to the server and the evaluation screen, is the same as that of the second embodiment of the information processing system. Figure 1 Therefore, the following description focuses on the server and the evaluation screen in the second embodiment.

[0121] Fig. 9is a block diagram showing a configuration example of a server in the second embodiment.

[0122] exist Fig. 9 In the server 110 in Figure 3 Units corresponding to the units of the server 12 in FIG. 1 are denoted by the same reference numerals. Therefore, descriptions of the units will be omitted as appropriate, and description will be given focusing on units different from the server 12. The server 110 is different from the server 12 in that the server 110 includes an evaluation map creation unit 111, an evaluation map update unit 112, and a generation unit 113, an evaluation map update unit 59, and a generation unit 60 instead of the evaluation map creation unit 55, and the other units are configured similarly to the server 12. The server 110 registers success / failure related information for each imaging position within a predetermined range and each imaging direction within a predetermined range in the evaluation map.

[0123] Specifically, the evaluation map creation unit 111 reads the 3D map from the 3D map storage unit 53. The evaluation map creation unit 111 creates an evaluation map by forming a plurality of grids in the ground area on the 3D map based on the ground area information provided from the detection unit 54. In the evaluation map, direction range information and information related to success or failure are registered in association with each grid, and the direction range information indicates the range of imaging directions of the evaluation image where the imaging position exists in the area corresponding to each grid. That is, in the evaluation map, information related to success or failure is registered for each imaging position within a predetermined range and each imaging direction within a predetermined range. In the second embodiment, the information related to success or failure is success rate information indicating the estimated success rate. The evaluation map creation unit 111 provides the evaluation map to the evaluation map storage unit 56.

[0124] The evaluation map updating unit 112 reads the evaluation map from the evaluation map storage unit 56. The evaluation map updating unit 112 acquires the estimated imaging information or the imaging information for evaluation provided from the estimation unit 58. The evaluation map updating unit 112 updates the evaluation map by updating the success rate information corresponding to the grid or the imaging information for evaluating the evaluation map and the direction range information of the imaging direction, the grid corresponding to the imaging position indicated by the estimated imaging information. The evaluation map updating unit 112 provides the updated evaluation map to the evaluation map storage unit 56, and updates the evaluation map stored in the evaluation map storage unit 56.

[0125] The generation unit 113 reads the evaluation map stored in the evaluation map storage unit 56. The generation unit 113 acquires the estimated imaging information or the imaging information for evaluation provided from the estimation unit 58. The generation unit 113 generates display control information based on the evaluation map and the estimated imaging information or the imaging information for evaluation. The display control information is information for displaying the direction range information and the success rate information registered in association with each grid of the evaluation map at a position corresponding to each grid on the evaluation image. The generation unit 113 supplies the display control information to the transmission unit 61.

[0126] <Configuration example of evaluation screen>

[0127] Fig.10 is a diagram showing a configuration example of an evaluation screen displayed on the mobile terminal 13 based on display control information generated by the server 110 .

[0128] exist Fig.10 In the evaluation screen 120, corresponding to Figure 5 The parts of the evaluation screen 90 are denoted by the same reference symbols. Therefore, the description of these parts will be appropriately omitted, and the following description will be provided focusing on the parts that are different from the evaluation screen 90. The evaluation screen 120 is different from the evaluation screen 90 in that the circle 121 is provided instead of the cones 91a and 91b, and the cone 91c is not provided, and the other parts are configured similarly to the evaluation screen 90.

[0129] On the evaluation screen 120, with the position corresponding to each grid on the camera view image 91 as the center 121a, a circle 121 as the direction range information and the success rate information corresponding to the grid is superimposed. Fig.11 The detailed configuration of the circle 121 is described.

[0130] <Detailed configuration example of circle>

[0131] Fig.11 It is shown Fig.10 1 is a diagram showing a detailed configuration example of the circle 121 in FIG.

[0132] exist Fig.11 In the example of , the imaging direction is classified into five ranges of 72 degrees each. Therefore, the circle 121 is divided into five fan-shaped areas 131-1 to 131-5 with a central angle of 72 degrees. It should be noted that in the following, in the case where it is not necessary to distinguish the areas 131-1 to 131-5 separately, the areas are collectively referred to as areas 131. Each area 131 is directional range information indicating that the imaging direction is a range, and when the center 121a of the circle 121 is set as the starting point, the end point of the range is included in the area 131. The density of each area 131 is success rate information.

[0133] exist Fig.11 In the example of , the success rate information is information indicating the success rate for each of the four ranges of the number of times the estimation is attempted by the estimation unit 58. Therefore, each area 131 is further divided into four areas 141 to 144. The area 141 closest to the center 121a has a fan shape, and the other areas 142 to 144 each have a ring shape.

[0134] The density of region 141 indicates, for example, the success rate in the case where the number of attempts for estimation is less than 5 times, and the density of region 142 indicates, for example, the success rate in the case where the number of attempts for estimation is 5 times or more and less than 10 times. The density of region 143 indicates, for example, the success rate in the case where the number of attempts for estimation is 10 times or more and less than 15 times, and the density of region 144 indicates, for example, the success rate in the case where the number of attempts for estimation is 15 times or more. Note that, in the case where estimation is not performed as many times as the number of attempts corresponding to regions 141 to 144, regions 141 to 144 are represented with a polka dot pattern as information indicating that data does not exist.

[0135] exist Fig.11 In the example of , for the evaluation images corresponding to the range of imaging positions corresponding to the circle 121 and the range of imaging directions corresponding to the area 131-1, the number of attempts for estimation is less than 15. Therefore, the areas 141 to 143 in the area 131-1 are each represented by a density indicating the corresponding success rate, and the area 144 is represented by a polka dot pattern.

[0136] exist Fig.11 In the example of , for the evaluation images corresponding to the range of imaging positions corresponding to the circle 121 and the range of imaging directions corresponding to the area 131-2, the number of attempts for estimation is less than 10. Therefore, the areas 141 and 142 in the area 131-2 are each represented by a density indicating the corresponding success rate, and the areas 143 and 144 are represented by a polka dot pattern.

[0137] exist Fig.11 In the example of , estimation of the evaluation image corresponding to the range of the imaging position corresponding to the circle 121 and the range of the imaging direction corresponding to the area 131-3 has not been performed. Therefore, the areas 141 to 144 in the area 131-3 are represented by a polka dot pattern.

[0138] exist Fig.11In the example of , for the evaluation images corresponding to the range of imaging positions corresponding to the circle 121 and the range of imaging directions corresponding to the area 131-4, the number of attempts for estimation is less than 5. Therefore, the area 141 in the area 131-4 is represented as a density indicating the corresponding success rate, and the areas 142 to 144 are represented as polka dot patterns.

[0139] exist Fig.11 In the example of FIG. 1 , for the evaluation images corresponding to the range of imaging positions corresponding to the circle 121 and the range of imaging directions corresponding to the area 131-5, the number of attempts for estimation is more than 15 times. Therefore, the areas 141 to 144 in the area 131-5 are each represented by a density representing the corresponding success rate.

[0140] It should be noted that the success rate may be represented by a character, color, pattern, etc. other than density. The information indicating the absence of data may be a predetermined pattern other than the polka dot pattern, a predetermined character, a predetermined color, a predetermined density, etc. The density of the regions 141 to 144 may become lower over time, i.e., approach a density representing zero.

[0141] The success rate information may be information indicating the success rate at each time from the time when the estimation is successful. In this case, the success rate indicated by the density of regions 141 to 144 does not differ in the number of estimations, but differs in the time after the estimation is successful. The success rate information may be information indicating the success rate at each height or pitch of the mobile terminal 13 at the time of imaging. In this case, the success rate indicated by the density of regions 141 to 144 differs in the number of estimations, but differs in the height or pitch of the mobile terminal 13 at the time of imaging.

[0142] As described above, the server 110 quantifies the imaging position and the imaging direction, and registers the success rate information as the success / failure related information corresponding to the imaging position within the predetermined range and the imaging direction within the predetermined range in the evaluation map. Therefore, it is possible to suppress the increase of the direction information and the success / failure related information superimposed on the camera view image 91 due to the increase in the number of attempts for estimation. As a result, even in the case where the number of attempts for estimation increases, the provider can easily recognize the success / failure related information for each of the imaging position within the predetermined range and the imaging direction within the predetermined range.

[0143] Note that the flowcharts of the creation process and evaluation map update process of the server 110 are similar to Figure 7 The creation process and Figure 8 This is a flowchart of the evaluation map update process in , and therefore, its description is omitted.

[0144] As described above, similar to the server 12, the server 110 acquires the evaluation image and the imaging information used for evaluation, estimates the imaging position and the imaging direction of the evaluation image, creates the evaluation map, updates the success / failure related information corresponding to the estimated imaging information of the evaluation map in the case of successful estimation, and updates the success / failure related information corresponding to the imaging information used for evaluation of the evaluation map in the case of failed estimation. Therefore, in the case of estimating the imaging position and the imaging direction, the imaging position and the imaging direction can be provided in the case of successful estimation and in the case of failed estimation.

[0145] In addition, the evaluation map created by the server 110 is an evaluation map in which the direction range information of the evaluation image of the imaging position in the area corresponding to each grid and the information related to success or failure are registered in association with each of the plurality of grids formed on the 3D map. Therefore, even in the case where the number of attempts for estimation increases, the range of the imaging position and the range of the imaging direction can be simply provided in the case where the estimation succeeds and the case where the estimation fails.

[0146] <3. Third Embodiment>

[0147] <Server Configuration Example>

[0148] The configuration of the third embodiment of the information processing system including a server as an information processing device to which the present technology is applied is similar to that of the information processing system of the second embodiment except for the server and the mobile terminal. Therefore, the following description focuses on the server and the mobile terminal in the third embodiment.

[0149] Fig.12 is a block diagram showing a configuration example of a server in the third embodiment.

[0150] exist Fig.12 In the server 202, corresponding to Fig. 9 The units of server 110 in FIG. 1 are denoted by the same reference numerals. Therefore, the description of the units will be appropriately omitted, and the description will be given focusing on the units that are different from server 110. Server 202 is different from server 110 in that server 202 includes a receiving unit 221, an estimating unit 222, and a 3D map storage unit 224 instead of receiving unit 57, estimating unit 58, and 3D map storage unit 53, and server 202 newly includes a 3D map updating unit 223. The other units are configured similarly to server 110. Server 202 also updates the 3D map using an update image (key frame), which is an image for updating the 3D map transmitted from a mobile terminal 203 described later.

[0151] Specifically, the receiving unit 221 of the server 202 receives the evaluation image and the imaging information for evaluation transmitted from the mobile terminal 203, and provides the evaluation image and the imaging information for evaluation to the estimation unit 222. The receiving unit 221 (acquisition unit) also receives (acquires) the update image and the imaging information for update (i.e., the imaging information of the update image) transmitted from the mobile terminal 203, and provides the update image and the imaging information for update to the estimation unit 222.

[0152] The estimation unit 222 reads the 3D map from the 3D map storage unit 53. Similar to the estimation unit 58, the estimation unit 222 estimates the imaging position and imaging direction of the evaluation image using the 3D map and the evaluation image. In the case of successful estimation, the estimation unit 222 provides the estimated imaging information to the evaluation map update unit 112 and the generation unit 113. In the case of failed estimation, the estimation unit 222 provides the imaging information for evaluation provided from the receiving unit 221 to the evaluation map update unit 112 and the generation unit 113.

[0153] Similar to the case of the evaluation image, the estimation unit 222 also estimates the imaging position and imaging direction of the update image using the 3D map and the update image. In the case of successful estimation, the estimation unit 222 provides the estimated imaging information and the update image to the 3D map update unit 223. In the case of failed estimation, the estimation unit 222 provides the imaging information and the update image for updating provided from the receiving unit 221 to the 3D map update unit 223.

[0154] The 3D map updating unit 223 reads the 3D map from the 3D map storage unit 224. The 3D map updating unit 223 updates the 3D map using the update image and the estimated imaging information or imaging information for updating provided from the estimation unit 222. The 3D map updating unit 223 provides the updated 3D map to the 3D map storage unit 224.

[0155] The 3D map storage unit 224 stores the 3D map supplied from the 3D map creation unit 52. The 3D map storage unit 224 updates the stored 3D map with the updated 3D map supplied from the 3D map update unit 223.

[0156] <Configuration example of mobile terminal>

[0157] Fig.13 is a block diagram showing a configuration example of a mobile terminal in the third embodiment.

[0158] exist Fig.13 In the mobile terminal 203, Figure 4Units corresponding to the units of the mobile terminal 13 are denoted by the same reference numerals. Therefore, the description of the units will be appropriately omitted, and the description will be given focusing on the units different from the mobile terminal 13. The mobile terminal 13 is different from the mobile terminal 13 in that the mobile terminal 13 includes an imaging unit 271, a detection unit 272, a transmission unit 273, and a control unit 277 instead of the imaging unit 71, the detection unit 72, the transmission unit 73, and the control unit 77, and the other units are configured similarly to the mobile terminal 13. The mobile terminal 203 transmits the update image selected by the provider to the server 202.

[0159] Specifically, the imaging unit 271 images a predetermined area with the current position of the mobile terminal 203 as the imaging position and the current direction of the mobile terminal 203 as the imaging direction. The imaging unit 271 provides the captured image captured in real time as the camera view image to the generation unit 75. The imaging unit 271 provides the captured image at a predetermined time as the evaluation image to the transmission unit 273. The imaging unit 271 provides the captured image at a predetermined time as the update image to the transmission unit 273.

[0160] The detection unit 272 is configured similarly to the detection unit 72, and detects the imaging position and imaging direction of each captured image. The detection unit 272 provides the imaging information for evaluation and the imaging information for update to the transmission unit 273 based on the detection result, and provides the camera view imaging information to the generation unit 75.

[0161] The transmission unit 273 transmits the evaluation image provided from the imaging unit 271 and the imaging information for evaluation provided from the detection unit 272 to the server 202. The transmission unit 273 transmits the update image provided from the imaging unit 271 and the imaging information for update provided from the detection unit 272 to the server 202.

[0162] The control unit 277 controls each unit of the mobile terminal 203. For example, similar to Figure 4 The control unit 277 controls the imaging unit 71 and the detection unit 72 to provide the evaluation image and the imaging information for evaluation to the transmission unit 273. The control unit 277 controls the imaging unit 271 and the detection unit 272 based on the operation signal provided from the input unit 81 to provide the update image selected by the provider and the imaging information for evaluation of the update image to the transmission unit 273. Similar to the selection of the evaluation image, the selection of the update image is performed by the provider performing a touch operation on the touch panel 76.

[0163] <Configuration example of evaluation screen>

[0164] Fig.14is a diagram showing a configuration example of an evaluation screen displayed on the mobile terminal 203 .

[0165] exist Fig.14 In the evaluation screen 300, corresponding to Fig.10 The parts of the evaluation screen 120 are denoted by the same reference symbols. Therefore, the description of these parts will be appropriately omitted, and the following description will be provided focusing on the parts that are different from the evaluation screen 120. The evaluation screen 300 is different from the evaluation screen 120 in that it has an update image button 301, an overhead view 302, a log notification unit 303, and a character 304, and the other components are the same as the evaluation screen 120.

[0166] In the case where the camera view image 91 is selected as the update image, the provider touches and operates the update image button 301. For example, while viewing the circle 121 displayed on the evaluation screen 300, the provider selects the camera view image 91 as the update image while moving the mobile terminal 203 until the success rate of each circle 121 exceeds a predetermined rate. When the provider performs a touch operation on the update image button 301, the touch operation is received by the input unit 81, and an operation signal indicating the touch operation is provided to the control unit 277. As a result, under the control of the control unit 277, the camera view image 91 displayed when the provider performs the touch operation is transmitted to the server 12 as the update image. In addition, under the control of the control unit 277, the camera view imaging information of the camera view image 91 is transmitted to the server 202 as the imaging information for evaluation.

[0167] The bird's-eye view image 302 is a bird's-eye view image of a grid of the evaluation map. On the bird's-eye view image 302, an imaging area 302a representing an area corresponding to the camera view image 91 is displayed.

[0168] The log notification unit 303 is an area for notifying the log of the evaluation map. Figure 3 In the example of , the log notification unit 303 displays the number of completions / number of evaluation areas as information indicating the completion rate of the evaluation map. Here, the number of evaluation areas is the sum of the areas 141 to 144 of all circles 121. The number of completions is the number of areas 141 to 144 in which data exists, that is, the number of areas 141 to 144 estimated by the corresponding number of attempts.

[0169] The character 304 is a character used in, for example, a video see-through augmented reality (AR) application in which the provider provides a VPS. By displaying the character 304, the provider can visually recognize the screen actually displayed in the video see-through AR application that provides the VSP.

[0170] The bird's-eye view image 302, the log notification means 303, and the character 304 are displayed, for example, according to the display control information. That is, the display control information in the third embodiment is information for also displaying the bird's-eye view image 302, the log notification means 303, and the character 304.

[0171] The bird's-eye view 302, log notification unit 303 and role 304 may also be included in Fig.10 In the evaluation screen 120 in FIG. 300 , the bird's-eye view 302 , the log notification unit 303 , and the character 304 may not be displayed.

[0172] <Description of Server Processing>

[0173] Fig.15 3D map update processing performed by the server 202 to update the 3D map is a flowchart showing the 3D map update processing.

[0174] exist Fig.15 In step S50, the receiving unit 221 determines whether the update image and the imaging information for update have been transmitted from the mobile terminal 203. In the case where it is determined in step S50 that the update image and the imaging information for update have not been transmitted, the process waits until the update image and the imaging information for update are transmitted.

[0175] On the other hand, in the case where it is determined in step S50 that the update image and the imaging information for update have been transmitted, the process proceeds to step S51. In step S51, the receiving unit 221 receives the update image and the imaging information for update transmitted from the mobile terminal 203, and provides the update image and the imaging information for update to the estimation unit 222.

[0176] In step S52 , the estimation unit 222 estimates the imaging position and imaging direction of the update image by performing matching between the feature points of the 3D map stored in the 3D map storage unit 53 and the feature points of the update image received through the process in step S51 .

[0177] In step S53, similar to Figure 8 In the process of step S33 in , the estimation unit 222 determines whether the estimation is successful. In the case where it is determined in step S53 that the estimation is successful, the estimation unit 222 supplies the estimated imaging information and the update image to the 3D map updating unit 223, and the process proceeds to step S54.

[0178] In step S54, the 3D map updating unit 223 reads the 3D map from the 3D map storage unit 224, and updates the 3D map using the estimated imaging information and the update image. Then, the process proceeds to step S56.

[0179] On the other hand, in the case where it is determined in step S53 that the estimation is unsuccessful, that is, in the case where it is determined that the estimation has failed, the estimation unit 222 provides the imaging information for updating and the updated image received through the processing in step S51 to the 3D map update unit 223, and the processing proceeds to step S55.

[0180] In step S55, the 3D map updating unit 223 reads the 3D map from the 3D map storage unit 224, and updates the 3D map using the imaging information for updating and the update image. Then, the process proceeds to step S56.

[0181] In step S56, the 3D map updating unit 223 supplies the 3D map updated in step S54 or step S55 to the 3D map storage unit 224 so that the 3D map storage unit 224 updates the stored 3D map. Then, the process ends.

[0182] Note that the evaluation image may be automatically used to update the 3D map as the update image. In addition, the update image may be transmitted from the imaging device 11 instead of from the mobile terminal 203.

[0183] As described above, similar to the server 12, the server 202 acquires the evaluation image and the imaging information for evaluation, estimates the imaging position and the imaging direction of the evaluation image, creates the evaluation map, updates the success / failure related information corresponding to the estimated imaging information of the evaluation map in the case of successful estimation, and updates the success / failure related information corresponding to the imaging information for evaluation of the evaluation map in the case of failed estimation. Therefore, in the case of estimating the imaging position and the imaging direction, the imaging position and the imaging direction can be provided in the case of successful estimation and in the case of failed estimation.

[0184] Furthermore, the server 202 creates an evaluation map similar to the server 110. Therefore, even in the case where the number of attempts for estimation increases, the range of imaging positions and the range of imaging directions can be simply provided in the case where the estimation succeeds and in the case where the estimation fails.

[0185] Furthermore, the server 202 updates the 3D map using the updated image. Therefore, the success rate of estimating the imaging position and imaging direction of the evaluation image can be improved.

[0186] In the first to third embodiments, the mobile terminal 13 (203) generates the evaluation screen, but the server 12 (202) may generate the evaluation screen. In this case, the mobile terminal 13 (203) transmits the camera view image and the camera view imaging information to the mobile terminal 13 (203).

[0187] The success / failure related information may be information indicating a failure rate of estimation, likelihood information indicating a likelihood of an estimation result, matching number information indicating the number of matches (number of matching inner points) as the number of feature points matched by the estimation unit 58 (222), etc. The success / failure related information may be information indicating a time average of the likelihood and the matching number, rather than the likelihood and the matching number themselves.

[0188] When the estimation unit 58 (222) determines the estimation result based on the likelihood of each imaging position and each imaging direction, there is a possibility that a plurality of estimation results having the same likelihood can be obtained. In this case, the evaluation map can be updated based on the plurality of estimation results.

[0189] In the third embodiment, in the case of estimation failure immediately after updating of the 3D map, regions 141 to 144 corresponding to the evaluation images for estimation may be given a color, pattern, etc. indicating estimation failure even immediately after updating of the 3D map.

[0190] The 3D mesh in the 3D model corresponding to the camera view image 91 may be superimposed on the camera view image 91. In the evaluation screen 90 (120, 300), success or failure of the estimation of the current evaluation image may be notified by voice.

[0191] Although the direction range information and the success rate information are circles 121 with the position corresponding to each grid as the center 121a, the direction range information and the success rate information may be two-dimensional figures such as hexagons other than circles as long as the range of positions and directions can be represented.

[0192] <4. Grid>

[0193] As the shape of the mesh formed as the evaluation map in the second and third embodiments, for example, a polygonal mesh such as Fig.16 The regular hexagonal grid 341 is shown. Fig.17 As shown, a grid 351 may be distributed around the location 352 of each captured image (key frame) used to generate the 3D map. Fig.18 As shown, grid 361 may be distributed around location 362 where character 304 is arranged.

[0194] The grid may correspond not only to the imaging position within a predetermined range, but also to the imaging direction within a predetermined range. Specifically, for example, in the case where the imaging direction is classified into six ranges of 60 degrees each, as Fig.19As shown in FIG. 3 , six equilateral triangle meshes 371 to 376 having the same vertex 370 correspond to imaging positions within the same range, and each of the meshes 371 to 376 may correspond to a corresponding range of six imaging directions. In this case, success / failure related information is registered in association with the mesh corresponding to the estimated imaging information or the imaging information for evaluation.

[0195] <5. Computer>

[0196] The above-mentioned series of processing of the server 12 (110, 202) may be performed by hardware or may be performed by software. In the case where the series of processing is performed by software, the program included in the software is installed on a computer. Here, examples of the computer include a computer incorporated in dedicated hardware and a general-purpose personal computer capable of performing various functions by installing various programs.

[0197] Fig. 20 : is a block diagram showing a configuration example of hardware of a computer that executes the above-described series of processes by a program.

[0198] In the computer, a central processing unit (CPU) 901 , a read only memory (ROM) 902 , and a random access memory (RAM) 903 are interconnected with a bus 904 .

[0199] An input / output interface 905 is further connected to the bus 904 . An input unit 906 , an output unit 907 , a storage unit 908 , a communication unit 909 , and a drive 910 are connected to the input / output interface 905 .

[0200] The input unit 906 includes a keyboard, a mouse, a microphone, etc. The output unit 907 includes a display, a speaker, etc. The storage unit 908 includes a hard disk, a nonvolatile memory, etc. The communication unit 909 includes a network interface, etc. The drive 910 drives a removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0201] In the computer configured as described above, for example, the CPU 901 loads a program stored in the storage unit 908 into the RAM 903 via the input / output interface 905 and the bus 904 and executes the program, thereby performing the above-described series of processing.

[0202] For example, the program executed by the computer (CPU 901 ) can be provided by being recorded in the removable medium 911 as a package medium, etc. Furthermore, the program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0203] In the computer, the program is installed in the storage unit 908 via the input / output interface 905 by mounting the removable medium 911 on the drive 910. In addition, the program can be received by the communication unit 909 via a wired or wireless transmission medium to be installed on the storage unit 908. In addition, the program can be installed on the ROM 902 and the storage unit 908 in advance.

[0204] It should be noted that the program executed by the computer may be a program that executes processing in time series in the order described in this specification, or may be a program that executes processing in parallel or at necessary timing such as when a call is made.

[0205] In this specification, a system means a group of multiple components (devices, modules (components), etc.), and it does not matter whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected to each other via a network and a device in which multiple modules are housed in one housing are both systems.

[0206] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

[0207] For example, a mode obtained by combining all or some of the above-mentioned multiple embodiments may be adopted. Specifically, for example, similar to the second and third embodiments, the success / failure related information in the first embodiment may be success rate information, etc. The evaluation map in the third embodiment may be configured similarly to the evaluation map in the first embodiment. The success / failure related information in the third embodiment may be success / failure information similar to the success / failure related information in the first embodiment.

[0208] For example, the present technology may be configured as cloud computing in which functions are shared by a plurality of devices via a network to be processed together.

[0209] Furthermore, each step described in the above flowchart may be performed by one device or may be shared and performed by a plurality of devices.

[0210] Furthermore, in the case where a plurality of processes are included in one step, the plurality of processes included in one step may be executed by one device or may be shared and executed by a plurality of devices.

[0211] The effects described in this specification are merely examples and are not limiting, and other effects may exist.

[0212] The present technology can have the following configurations.

[0213] (1) An information processing device comprising:

[0214] an acquisition unit that acquires an evaluation image captured by an imaging device and imaging information for evaluation, the imaging information for evaluation being imaging information indicating an imaging position and an imaging direction of the evaluation image;

[0215] an estimating unit that estimates an imaging position and an imaging direction of the evaluation image by performing matching between feature points of the 3D map and feature points of the evaluation image acquired by the acquiring unit;

[0216] a creating unit that creates an evaluation map in which success / failure related information regarding whether the estimation performed by the estimation unit is successful is registered in association with an imaging position and an imaging direction of the evaluation image; and

[0217] An evaluation map updating unit updates, when the estimation performed by the estimation unit is successful, information related to success or failure corresponding to the imaging position and the imaging direction estimated by the estimation unit, and when the estimation performed by the estimation unit fails, information related to success or failure corresponding to the imaging position and the imaging direction represented by the imaging information used for evaluation is updated.

[0218] (2) The information processing device according to (1), further comprising:

[0219] a generating unit that generates, based on the evaluation map, display control information for displaying direction information indicating an imaging direction registered in the evaluation map in association with the imaging position and success / failure related information at an imaging position registered in the evaluation map on the evaluation image acquired by the acquiring unit; and

[0220] The transmission unit transmits the display control information to the imaging device.

[0221] (3) The information processing device according to (2),

[0222] The display control information is to also display information indicating the achievement rate of the evaluation map.

[0223] (4) The information processing device according to any one of (1) to (3),

[0224] The success / failure related information is success / failure information indicating whether the estimation performed by the estimation unit is successful.

[0225] (5) The information processing device according to any one of (1) to (3),

[0226] The success / failure related information is information indicating the success rate or failure rate of the estimation performed by the estimation unit.

[0227] (6) The information processing device according to (5),

[0228] The success / failure related information is information indicating a success rate or a failure rate for each of the number of times the estimation unit attempts the estimation or for each estimation that succeeds.

[0229] (7) The information processing device according to (5),

[0230] The success / failure related information is information indicating the success rate or failure rate for each height or pitch of the imaging device.

[0231] (8) The information processing device according to any one of (1) to (3),

[0232] The success / failure related information is likelihood information indicating the likelihood of the result of the estimation performed by the estimation unit.

[0233] (9) The information processing device according to any one of (1) to (3),

[0234] The success or failure related information is matching number information indicating the matching number, where the matching number is the number of feature points matched by the estimation unit.

[0235] (10) The information processing device according to any one of (1) to (9),

[0236] Among them, the creation unit is configured to create an evaluation map by forming multiple grids on a 3D map, in which the direction range information and information related to success or failure are registered in association with each grid, and the direction range information represents the range of the imaging direction of the evaluation image, and in the range, the imaging position exists in the area corresponding to each grid.

[0237] (11) The information processing device according to (10),

[0238] The creation unit is configured to create a plurality of grids in a ground area of ​​the 3D map.

[0239] (12) The information processing device according to (11), further comprising:

[0240] Detection unit, detects ground area from 3D map,

[0241] The creation unit is configured to create a plurality of grids in a ground area detected by the detection unit.

[0242] (13) The information processing device according to any one of (1) to (12), further comprising:

[0243] a 3D map updating unit that updates the 3D map using an update image as an image for updating the 3D map,

[0244] The acquisition unit is configured to acquire the updated image and updated imaging information indicating the imaging position and imaging direction of the updated image.

[0245] The estimation unit is configured to estimate an imaging position and an imaging direction of the update image by performing matching between feature points of the 3D map and feature points of the update image, and

[0246] The 3D map updating unit is configured to update the 3D map using the update image, and the imaging position and imaging direction estimated by the estimation unit or the imaging information for updating.

[0247] (14) An information processing method executed by an information processing device, the information processing method comprising:

[0248] an acquisition step of acquiring an evaluation image captured by an imaging device and imaging information for evaluation, wherein the imaging information for evaluation is imaging information indicating an imaging position and an imaging direction of the evaluation image;

[0249] an estimating step of estimating an imaging position and an imaging direction of the evaluation image by performing matching between feature points of the 3D map and feature points of the evaluation image acquired by the acquiring step;

[0250] a creating step of creating an evaluation map in which success / failure related information regarding whether the evaluation in the estimating step is successful is registered in association with an imaging position and an imaging direction of the evaluation image; and

[0251] An evaluation map updating step, in the case where the estimation performed by the estimation step is successful, updates the success / failure related information corresponding to the imaging position and the imaging direction estimated by the estimation step, and in the case where the estimation performed by the estimation step fails, updates the success / failure related information corresponding to the imaging position and the imaging direction represented by the imaging information used for evaluation.

[0252] (15) A program for causing a computer to function as an information processing device, the information processing device comprising:

[0253] an acquisition unit that acquires an evaluation image captured by an imaging device and imaging information for evaluation, the imaging information for evaluation being imaging information indicating an imaging position and an imaging direction of the evaluation image;

[0254] an estimating unit that estimates an imaging position and an imaging direction of the evaluation image by performing matching between feature points of the 3D map and feature points of the evaluation image acquired by the acquiring unit;

[0255] a creating unit that creates an evaluation map in which success / failure related information regarding whether the estimation performed by the estimation unit is successful is registered in association with an imaging position and an imaging direction of the evaluation image; and

[0256] An evaluation map updating unit updates, when the estimation performed by the estimation unit is successful, information related to success or failure corresponding to the imaging position and the imaging direction estimated by the estimation unit, and when the estimation performed by the estimation unit fails, information related to success or failure corresponding to the imaging position and the imaging direction represented by the imaging information used for evaluation is updated.

[0257] Reference Symbols List

[0258] 12 Server

[0259] 13 Mobile Terminals

[0260] 54 Detection Unit

[0261] 55 Evaluation Map Creation Unit

[0262] 57 Receiving Unit

[0263] 58 Estimation Units

[0264] 59 Evaluation Map Update Unit

[0265] 60 Generation Units

[0266] 61 Transmission Unit

[0267] 91a to 91c cones

[0268] 110 Server

[0269] 111 Evaluation Map Creation Unit

[0270] 112 Evaluation Map Update Unit

[0271] 113 Generation Unit

[0272] 121 Circle

[0273] 202 Server

[0274] 203 Mobile Terminal

[0275] 221 Receiving Unit

[0276] 222 Estimation Unit

[0277] 303 Log notification unit.

Claims

1. An information processing device, comprising: an acquisition unit that acquires an evaluation image captured by an imaging device and imaging information for evaluation, wherein the imaging information for evaluation is imaging information indicating an imaging position and an imaging direction of the evaluation image; an estimating unit that estimates the imaging position and the imaging direction of the evaluation image by performing matching between feature points of a 3D map and feature points of the evaluation image acquired by the acquiring unit; a creating unit that creates an evaluation map in which success / failure related information regarding whether the estimation performed by the estimation unit is successful is registered in association with the imaging position and the imaging direction of the evaluation image; as well as An evaluation map updating unit updates the success / failure related information corresponding to the imaging position and the imaging direction estimated by the estimation unit, if the estimation performed by the estimation unit is successful, and updates the success / failure related information corresponding to the imaging position and the imaging direction represented by the imaging information used for evaluation, if the estimation performed by the estimation unit fails.

2. The information processing device according to claim 1, further comprising: a generating unit that generates, based on the evaluation map, display control information for displaying direction information indicating the imaging direction registered in the evaluation map in association with the imaging position and the success / failure related information at the imaging position registered in the evaluation map on the evaluation image acquired by the acquiring unit; as well as A transmission unit transmits the display control information to the imaging device.

3. The information processing device according to claim 2, in, The display control information is to further display information indicating the achievement rate of the evaluation map.

4. The information processing device according to claim 1, in, The success / failure related information is success / failure information indicating whether the estimation performed by the estimation unit is successful.

5. The information processing device according to claim 1, in, The success / failure related information is information indicating a success rate or a failure rate of estimation performed by the estimation unit.

6. The information processing device according to claim 5, in, The success / failure related information is information indicating the success rate or the failure rate for each of the number of times the estimation is attempted by the estimation unit or for each success of the estimation.

7. The information processing device according to claim 5, in, The success / failure related information is information indicating the success rate or the failure rate for each height or pitch of the imaging device.

8. The information processing device according to claim 1, in, The success / failure related information is likelihood information indicating the likelihood of a result of estimation performed by the estimation unit.

9. The information processing device according to claim 1, in, The success / failure related information is matching number information indicating the matching number, where the matching number is the number of feature points matched by the estimation unit.

10. The information processing device according to claim 1, in, The creation unit is configured to create the evaluation map by forming a plurality of grids on the 3D map, in which direction range information and the success / failure related information are registered in association with each grid, the direction range information indicating a range of the imaging direction of the evaluation image, within which the imaging position exists in an area corresponding to each grid.

11. The information processing device according to claim 10, in, The creation unit is configured to create the plurality of grids in a ground area of ​​the 3D map.

12. The information processing device according to claim 11, further comprising: a detection unit, detecting the ground area from the 3D map, Wherein, the creation unit is configured to create the plurality of grids in the ground area detected by the detection unit.

13. The information processing device according to claim 1, further comprising: a 3D map updating unit that updates the 3D map using an update image as an image for updating the 3D map, The acquisition unit is configured to acquire the updated image and updated imaging information indicating an imaging position and an imaging direction of the updated image. The estimation unit is configured to estimate the imaging position and the imaging direction of the update image by performing matching between feature points of the 3D map and feature points of the update image, and The 3D map updating unit is configured to update the 3D map using the update image, and the imaging position and the imaging direction estimated by the estimation unit or the imaging information for updating.

14. An information processing method performed by an information processing device, the information processing method comprising: An acquisition step of acquiring an evaluation image captured by an imaging device and imaging information for evaluation, wherein the imaging information for evaluation is imaging information indicating an imaging position and an imaging direction of the evaluation image; an estimating step of estimating the imaging position and the imaging direction of the evaluation image by performing matching between feature points of a 3D map and feature points of the evaluation image acquired by the acquiring step; a creating step of creating an evaluation map in which success / failure related information regarding whether the evaluation performed in the estimating step is successful is registered in association with the imaging position and the imaging direction of the evaluation image; as well as An evaluation map updating step, in the case where the estimation performed by the estimation step is successful, updating the success / failure related information corresponding to the imaging position and the imaging direction estimated by the estimation step, and in the case where the estimation performed by the estimation step fails, updating the success / failure related information corresponding to the imaging position and the imaging direction represented by the imaging information used for evaluation.

15. A program for causing a computer to function as an information processing device, the information processing device comprising: an acquisition unit that acquires an evaluation image captured by an imaging device and imaging information for evaluation, wherein the imaging information for evaluation is imaging information indicating an imaging position and an imaging direction of the evaluation image; an estimating unit that estimates an imaging position and an imaging direction of the evaluation image by performing matching between feature points of the 3D map and feature points of the evaluation image acquired by the acquiring unit; a creating unit that creates an evaluation map in which success / failure related information regarding whether the estimation performed by the estimation unit is successful is registered in association with an imaging position and an imaging direction of the evaluation image; as well as An evaluation map updating unit updates the success / failure related information corresponding to the imaging position and the imaging direction estimated by the estimation unit, if the estimation performed by the estimation unit is successful, and updates the success / failure related information corresponding to the imaging position and the imaging direction represented by the imaging information used for evaluation, if the estimation performed by the estimation unit fails.

Images