Information processing apparatus, mobile device, system, control method, and program

Abstract

To detect a mobile object in a map used for positioning of a mobile device in order to achieve a stable and highly accurate position and posture estimation process.SOLUTION: An information processing apparatus includes a first map information holding unit that holds a first map including at least one image acquired by a mobile device and feature information of the image, an image input unit that inputs an image of a fixed camera, a mobile object detection unit that detects a mobile object from the image of the fixed camera, a map correspondence detection unit that detects a correspondence between the feature information of the image of the fixed camera and the feature information included in the first map, a map update unit that generates a second map in which information indicating whether it is the mobile object is added to the feature information included in the first map using detection results of the mobile object detection unit and the map correspondence detection unit, and a second map information holding unit that holds the second map.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to techniques for detecting moving objects from maps used for positioning of moving devices. [Background technology]

[0002] In recent years, there has been a demand for shortening the time and reducing costs required for evaluation using prototypes in the fields of design and manufacturing. Mixed reality (MR) systems have been introduced to evaluate ease of assembly and maintainability using design (shape and design) data created with CAD (Computer-Aided Design) systems.

[0003] Image display devices for MR systems are classified into video see-through and optical see-through types. Video see-through image display devices display a composite image in which an image of a virtual space (such as a virtual object or text information drawn by computer graphics) generated according to the position and orientation of an imaging device is superimposed on an image of real space captured by an imaging device. Optical see-through image display devices display an image of a virtual space generated according to the position and orientation of the observer's viewpoint on an optical see-through display worn on the observer's head.

[0004] One of the important issues in mixed reality technology is how to accurately align the real and virtual worlds. To achieve this, it is necessary to estimate the position and orientation of the camera in the real world accurately and in real time.

[0005] One method for achieving this is to construct a map of the real space using images captured by a camera, and then calculate the position and orientation of the camera based on that map. There is a technique called Simultaneous Localization and Mapping (SLAM) (see, for example, Non-Patent Document 1). This technology is used not only in MR systems, but also in self-localization estimation of robots and automated guided vehicles.

[0006] Visual SLAM is a method that obtains the position and orientation of a camera in real space from an image captured by the camera, while at the same time extracting keyframe information from the image as necessary and building a map that accumulates that information. Keyframe information includes the image, the position and orientation of the camera that captured the image, and 3D information of feature points extracted from the image. In the Visual SLAM method, if a new map is created during execution, position and orientation estimation is performed based on a different reference coordinate system each time it is executed. Therefore, in the same environment, it is desirable to perform processing using a map created in the past and perform position and orientation estimation based on a consistent reference coordinate system.

[0007] In addition, in VisualSLAM, if moving objects are included in the images captured by the camera, they will generate motion vectors in a direction different from the camera movement, which will cause errors in the estimation of the camera position and orientation. Therefore, in order to estimate the camera position and orientation more reliably, it is desirable to remove the influence of moving objects.

[0008] In Patent Document 1, this is achieved by determining, from among corresponding points on the map and the image, points with different motion vectors as points on a moving object and removing them from the map. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] JP 2012-221042 A [Non-patent literature]

[0010] [Non-Patent Document 1] G. Klein and D. Murray: “Parallel Tracking and Mapping for Small AR Workspaces”, Proceedings of Sixth IEEE and ACM International Symposium on Mixed and Augmented Reality, 2007. [Non-Patent Document 2] B. Elvira, JDTardos and JMMMontiel: “ORBSLAM-Atlas: A Robust And Accurate Multi-map System”, IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), 2019. [Non-Patent Document 3] DGLowe: “Distinctive Image Features from Scale-Invariant Keypoints”, International Journal of Computer Vision, 2004. Summary of the Invention [Problem to be solved by the invention]

[0011] However, in the conventional method, when the proportion of corresponding points on moving objects is large among all corresponding points, there is a possibility that points on stationary objects will be removed. Also, even if a corresponding point is on a moving object, if the difference in motion vector is less than a threshold, it is determined to be on a stationary object and will not be removed, which may result in a decrease in the accuracy of camera position and orientation estimation.

[0012] Therefore, an object of the present invention is to detect a moving object from a map used for positioning a moving device in order to realize stable and highly accurate position and orientation estimation processing. [Means for solving the problem]

[0013] The first aspect of the present invention is a first map information storage unit that stores a first map including at least one image captured by a mobile device and feature information of the image; an image input unit for inputting an image from a fixed camera; a moving object detection unit that detects a moving object from an image captured by the fixed camera; a map correspondence detection unit that detects correspondence between feature information of an image of the fixed camera and feature information included in the first map; a map update unit that uses detection results of the moving object detection unit and the map correspondence detection unit to generate a second map in which information indicating whether or not the object is a moving object is added to the feature information included in the first map; and a second map information storage unit for storing the second map; The information processing device is characterized by comprising:

[0014] A second aspect of the present invention is a first map generating unit that generates a first map including an image captured by a camera and feature information of the image; a first map transmission unit configured to transmit the first map to another device; a second map receiving unit that receives, from the other device, a second map in which information indicating whether or not the object is a moving object is added to the feature information included in the first map; a self-location estimation unit that estimates a location of the moving device based on feature information of an object that is not the moving object among feature information included in the second map; A mobile device comprising:

[0015] A third aspect of the present invention is A system including a mobile device and an information processing device, The mobile device comprises: a first map generating unit that generates a first map including an image captured by a camera associated with the mobile device and feature information of the image from the image; A self-location estimation unit that estimates a location of the moving device; Equipped with The information processing device includes: an image input unit for inputting an image from a fixed camera; a moving object detection unit that detects a moving object from an image captured by the fixed camera; Equipped with a map correspondence detection unit that detects correspondence between feature information of an image of the fixed camera and feature information included in the first map; a map update unit that uses detection results of the moving object detection unit and the map correspondence detection unit to generate a second map in which information indicating whether or not the object is a moving object is added to the feature information included in the first map; and a second map information storage unit for storing the second map; are provided in at least one of the mobile device and the information processing device, The self-location estimation unit estimates a location of the moving device based on feature information of an object other than the moving object, among feature information included in the second map. The system is characterized by the above.

[0016] A fourth aspect of the present invention is A method for controlling an information processing device, comprising: a first map information storage step of storing a first map including at least one image captured by the mobile device and feature information of the image; A moving object detection step of detecting a moving object from an image of a fixed camera; a map correspondence detection step of detecting a correspondence between feature information of an image of the fixed camera and feature information included in the first map; a map updating step of generating a second map in which information indicating whether or not the object is a moving object is added to the feature information included in the first map, using detection results of the moving object detection step and the map correspondence detection step; a second map information storage step for storing the second map; The control method includes:

[0017] A fifth aspect of the present invention is 1. A method for controlling a mobile device, comprising: a first map generating step of generating a first map including an image captured by the camera and feature information of the image; a first map transmission step of transmitting the first map to another device; a second map receiving step of receiving a second map from the other device in which information indicating whether or not the object is a moving object is added to the feature information included in the first map; a self-location estimation step of estimating a location of the moving device based on feature information of an object other than the moving object among feature information included in the second map; The control method includes:

[0018] A sixth aspect of the present invention is A method for controlling a system including a mobile device and an information processing device, comprising: an image input step in which the information processing device inputs an image from a fixed camera; a moving object detection step in which the information processing device detects a moving object from an image of the fixed camera; a first map generation step in which the mobile device generates a first map from an image captured by a camera associated with the mobile device, the first map including the image and feature information of the image; a map correspondence detection step in which at least one of the mobile device and the information processing device detects correspondence between feature information of the image of the fixed camera and feature information included in the first map; a map update step in which at least one of the moving device and the information processing device generates a second map by adding information indicating whether or not the object is a moving object to the feature information included in the first map, using detection results of the moving object detection step and the map correspondence detection step; and a second map information holding step in which at least one of the mobile device and the information processing device holds the second map; The mobile device includes a self-location estimation unit that estimates a location of the mobile device; The control method includes: Effect of the Invention

[0019] According to the present invention, a moving object can be detected from a map used for positioning a moving device, and a stable and highly accurate estimation of the position and orientation can be achieved without being affected by the moving object. [Brief description of the drawings]

[0020] [Figure 1] 1 is a block diagram showing a configuration of a system according to a first embodiment. [Diagram 2] 1 is a schematic diagram showing the arrangement of a camera 100 and a mobile device 2 in real space. [Diagram 3] FIG. 2 is a block diagram showing a hardware configuration of an information processing device. [Figure 4] 11A and 11B are schematic diagrams illustrating a map-corresponding detection process; [Diagram 5] 4 is a flowchart showing a process performed by the information processing device according to the first embodiment. [Figure 6] FIG. 11 is a block diagram showing the configuration of a system according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the configurations described in the following embodiments are representative examples, and the scope of the present invention is not necessarily limited to those specific configurations.

[0022] <Embodiment 1> 1 is a block diagram showing an example of the configuration of a system according to this embodiment. As shown in FIG 1, the system according to this embodiment has a configuration in which an information processing device 1 is connected to a camera 100 and a mobile device 2.

[0023] The information processing device 1 includes an image input unit 101, a first map receiving unit 102, a first map information holding unit 108, a map correspondence detection unit 103, a moving object detection unit 104, a map update unit 105, a second map information holding unit 106, and a second map transmission unit 107.

[0024] The mobile device 2 is a device that can move in real space and can estimate its own position and orientation based on a map of real space constructed using an image captured by a camera 200 associated with the mobile device 2. The mobile device 2 may have a built-in camera 200 or may be connected to the camera 200. The mobile device 2 may be, for example, a smartphone with a camera or an HMD (Head Mounted Display). In order to present an MR image to a user, the mobile device 2 may estimate its own position and orientation based on a map of the real space. The mobile device 2 may be an autonomous mobile robot, and may estimate its own position and orientation for navigation. In this embodiment, processing for one mobile device 2 is described, but multiple mobile devices 2 may be connected to the information processing device 1 in the present invention, and processing for the multiple mobile devices 2 may be performed simultaneously. The mobile device 2 and the information processing device 1 may be configured with the same hardware.

[0025] 2 is a schematic diagram showing the state in which the camera 100, the information processing device 1, and the mobile device 2 in this embodiment are arranged in real space. In this embodiment, the mobile device 2 is described as a video see-through type HMD that can be worn on the head of the user. A desk 301 and a chair 302 are objects in real space, and the desk 301 is described as an example of a stationary object, and the chair 302 is described as an example of a moving object.

[0026] In this embodiment, the camera 100 connected to the information processing device 1 is described as a monocular fixed camera fixed in real space, but may be a multi-lens fixed camera. Also, the camera 100 is described as a camera that acquires a luminance image, but may be a camera that can acquire a color image, a depth image, a non-visible light image, etc. In the present invention, the camera parameters and the position and orientation of the camera 100 do not need to be calibrated in advance, but may be calibrated in advance.

[0027] The camera 200 connected to the mobile device 2 will be described as a stereo camera configuration having imaging means for the left and right eyes, but the camera 200 may be a monocular camera, or two or more cameras may be connected to the mobile device 2. The map of real space constructed in the mobile device 2 includes at least images captured by the camera 200. However, other sensors such as an acceleration sensor, an angular velocity sensor, an ultrasonic sensor, a magnetic sensor, a depth image sensor, and a laser distance sensor, or information obtained from markers placed in the real space may also be used to construct the map of real space.

[0028] The hardware configuration of the information processing device 1 will be described with reference to FIG. 3. In the figure, 31 is a CPU that controls the entire device. 32 is a RAM that is used as a working area when the CPU 31 performs processing while controlling each unit. 33 is a ROM that stores a control program, various application programs, data, and the like. The CPU 31 deploys the control program stored in the ROM 33 in the RAM 32 and executes it, thereby realizing each functional unit shown in FIG. 1. That is, the CPU 31 corresponds to the image input unit 101, the first map receiving unit 102, the first map information holding unit 108, the map correspondence detection unit 103, the moving object detection unit 104, the map update unit 105, the second map information holding unit 106, and the second map transmission unit 107. The input I / F 34 inputs input signals from the camera 100 and the first map receiving unit 102 in a format that can be processed by the information processing device 1. The output I / F 35 outputs an output signal from the second map transmission unit 107 to an external device in a format that can be processed by the external device.

[0029] As described above, the functions of each unit of the information processing device 1 shown in Fig. 1 can be realized by the CPU 31 executing a program. However, at least some of the units shown in Fig. 1 may be executed by dedicated hardware or a GPU (not shown). In this case, the dedicated hardware or the GPU operates under the control of the CPU 31.

[0030] The mobile device 2 also has a hardware configuration similar to that of the information processing device 1 shown in Figure 3, and the functions of a first map generation unit 201, a first map transmission unit 202, a second map receiving unit 203, and a self-position and orientation estimation unit 204 are realized by the CPU executing a program.

[0031] An information processing device 1 and a mobile device 2 according to this embodiment will be described.

[0032] 1 acquires an image of real space captured by a camera 100, and sends it to a map correspondence detection unit 103 and a moving object detection unit 104. The camera 100 is a device that records moving images, and the image input unit 101 inputs an image for each frame of the moving image.

[0033] The moving device 2 acquires an image of the real space captured by the camera 200, and the first map generating unit 201 constructs three-dimensional information of the real space including information of the image as a first map. The self-position and orientation estimating unit 204 estimates the self-position and orientation based on the first map. Since the estimation of the self-position and orientation can be realized using existing technology such as SLAM, detailed explanation is omitted. Note that the moving device 2 may have a self-position estimating unit that estimates only the position of the moving device 2 instead of the self-position and orientation estimating unit 204. The first map transmitting unit 202 transmits information of the first map to the first map receiving unit 102 of the information processing device 1 at a predetermined transmission timing. The second map receiving unit 203 receives the second map transmitted from the second map transmitting unit 107 of the information processing device 1. The self-position and orientation estimating unit 204 estimates the self-position and orientation based on the information of the second map. At this time, the self-position and orientation estimating unit 204 estimates the self-position and orientation without using the feature points to which a flag indicating that it is a moving object is added in the second map. The self-position and orientation estimation unit 204 can estimate the self-position and orientation based on either the first map or the second map, but using the second map enables highly accurate estimation that excludes the influence of moving objects. Note that the mobile device 2 in the present invention does not need to have the configuration described above, and may be any device that can transmit the first map and receive the second map to and from the information processing device 1.

[0034] The first map in this embodiment includes images of multiple key frames, feature information detected from the images, and the camera position and orientation when the images were captured. The feature information in this embodiment is the three-dimensional coordinates of feature points in space and the image coordinates of the key frames corresponding to the feature points. However, the feature information in the present invention is not limited to feature points, and may include, for example, information on characteristic edges and characteristic areas. The first map can be constructed, for example, by a method disclosed in Non-Patent Document 1. The camera position and orientation when the images were captured can be obtained by self-position and orientation estimation using the images of the multiple key frames. The first map may also include information other than the above, such as information from an orientation sensor.

[0035] The timing for the mobile device 2 to transmit the first map information to the first map receiving unit 102 may be, for example, when it is determined that the mobile device 2 is in the vicinity of the camera 100 (within a predetermined distance). The fact that the mobile device 2 is in the vicinity of the camera 100 can be determined, for example, by using a wireless signal. The timing for transmitting the first map information may also be when the mobile device 2 obtains new key frame information from the camera 200. The timing for transmitting the first map information may also be when new information obtained from the camera 200 is added to the received second map. In this case, the mobile device 2 sends the map obtained by adding the new information obtained from the camera 200 to the received second map again to the first map receiving unit 102 of the information processing device 1 as the first map information.

[0036] The first map receiving unit 102 receives information about the first map from the mobile device 2, stores it in the first map information storage unit 108, and notifies the map correspondence detection unit 103. At this time, the current location of the mobile device 2 may be received at the same time and stored in the first map information storage unit 108.

[0037] Map correspondence detection unit 103 performs a correspondence detection process between the first map stored in first map information storage unit 108 and the image input from image input unit 101. A specific example of the correspondence detection process will be described with reference to Figs. 4(a) and 4(b).

[0038] FIG. 4(a) is a schematic diagram showing an image of a key frame included in the information of the first map. Here, the explanation will be given assuming that a desk 301 and a chair 302 are observed as real objects. The first map includes information on the coordinates in the image (image coordinates) of the feature points (401 to 406 in the drawing) of the desk 301 and the chair 302 detected from the image of the key frame, and the coordinates in three-dimensional space (three-dimensional coordinates) of each feature point.

[0039] 4(b) is a schematic diagram showing an image input from the image input unit 101. It is assumed that a desk 301 and a chair 302 are observed as real objects common to the key frame in the first map.

[0040] In the correspondence detection process, first, feature points are detected from the image of the camera 100 input from the image input unit 101 (411 to 416 in the figure). Next, the detected feature points 411 to 416 are compared with the feature points 401 to 406 of the key frame included in the first map to obtain pairs of corresponding feature points. Here, it is assumed that the feature points 411 to 416 are obtained as corresponding points of the feature points 401 to 406, respectively. The process of detecting corresponding points between two images can be performed using, for example, SIFT features (see Non-Patent Document 3). However, the process of the map correspondence detection unit 103 in the present invention is not limited to this. For example, a segmentation process that divides the image into multiple object regions may be performed on each of the image input from the image input unit 101 and the image of the key frame, and whether or not each region is a region of the same object may be determined to obtain the corresponding region. In addition, during the process, information of an artificial index arranged in real space may be used, or a known three-dimensional model of a real object may be used.

[0041] The correspondence detection process may be performed between the image of camera 100 input from image input unit 101 and all of the key frames included in the first map, or may be performed between the input image of camera 100 and some of the key frames. Alternatively, the correspondence detection process may be performed in advance between key frames included in the first map, and the correspondence with all of the key frames may be detected by combining the results of the correspondence detection process performed between the image input from image input unit 101 and some of the key frames.

[0042] The result of the correspondence detection process obtained using the above method is sent to a map update unit 105, which will be described later.

[0043] The moving object detection unit 104 performs a process of detecting a moving object from an image input from the image input unit 101, and sends the result to the map update unit 105. The process of detecting a moving object can be realized by using a plurality of frames of images input from the image input unit 101. As an example, a background subtraction method can be adopted. The background subtraction method compares an image captured by the camera 100 so that a moving object is not captured (i.e., a background image) with an image input from the image input unit 101, and detects an image area in which the pixel brightness difference is equal to or greater than a threshold value. Since it can be considered that a moving object is present in the detected image area and that a non-moving object is present in the other image areas, in this embodiment, information on the detected image area is sent to the map update unit 105 as a detection result of a moving object. However, the method of detecting a moving object in the present invention is not limited to this. For example, a moving object may be detected by detecting a feature point from an image and tracking the feature point between frames of a moving image input from the image input unit 101. In order to detect a moving object, an artificial index placed in a real space may be used, or a three-dimensional model of a real object may be used.

[0044] The map update unit 105 identifies information corresponding to the moving object from among the information included in the first map, using the result of the correspondence detection process sent from the map correspondence detection unit 103 and the detection result of the moving object sent from the moving object detection unit 104. The map update unit 105 further generates a second map using the identified information, and stores it in the second map information storage unit 106.

[0045] A specific method of generating the second map will be described by taking as an example a case where the moving object detection unit 104 detects the physical object 302 as a moving object. Since the feature points 414 to 416 detected by the map correspondence detection unit 103 are in the image area of ​​the moving object (302), the map update unit 105 determines that these feature points 414 to 416 belong to the moving object. Furthermore, from the result of the processing by the map correspondence detection unit 103, it is found that the feature points in the first map corresponding to the feature points 414 to 416 are the feature points 404 to 406, respectively. Therefore, the map update unit 105 can specify that the feature points 404 to 406 are points of a moving object. Using this specification result, the map update unit 105 generates a second map in which a flag indicating whether or not the feature points are moving objects is added as correspondence information to each feature point in the first map, and stores the second map information storage unit 106. It is sufficient that the second map is configured so that it is possible to determine whether each feature point is a moving object or a non-moving object. Therefore, a flag may be added to all feature points in the second map indicating whether they are moving or non-moving objects, or a flag may be added only to feature points of moving objects, or a flag may be added only to feature points of non-moving objects.

[0046] By using the second map generated in this manner, the mobile device 2 is able to estimate the camera position and orientation using only the feature points of stationary objects, which is expected to improve robustness to moving objects and estimation accuracy.

[0047] Furthermore, when generating the second map, the map update unit 105 may integrate the newly generated second map with information of the second map that was generated in the past and is stored in the second map information storage unit 106. The integration of the maps includes a process of extracting corresponding points between key frames included in the two maps, finding the correspondence between the three-dimensional points included in each map, and integrating the corresponding points into one point. The integration of the maps also includes a process of estimating the relative positions and orientations between the key frames based on the information, in other words, a process of aligning the origins (reference coordinate systems) of the two maps. By integrating the information of the past maps, it is possible to obtain information on key frames from more viewpoints, and therefore it is expected that the camera position and orientation in the mobile device 2 can be stabilized. In addition, there is also an effect that the camera position and orientation can be estimated based on the same reference coordinate system as when the camera position and orientation processing was performed in the past. As a result, for example, when generating an MR image, it becomes possible to generate an image in which a CG model previously placed in real space continues to exist in the same position thereafter. A specific method for integrating two maps is to obtain the correspondence between key frames between the maps and calculate their relative positions and orientations, as disclosed in, for example, Non-Patent Document 2. However, the map integration method in the present invention is not limited to this.

[0048] The second map transmission unit 107 transmits information of the second map stored in the second map information storage unit 106 to the mobile device 2. At that time, the second map transmission unit 107 may transmit only map information of the second map that is close to (within a predetermined distance from) the current location of the mobile device 2, instead of transmitting the entire second map. The current location of the mobile device 2 may be estimated from the camera position and orientation of a key frame included in the first map, or, if the first map information storage unit 108 stores the current location of the mobile device 2 together with the first map, the information may be used to estimate the current location.

[0049] An example of processing by the information processing device 1 will be described with reference to the flowchart shown in FIG.

[0050] Step S501 is a process of importing an image captured by the camera 100 into the information processing device 1, and is processed by the image input unit 101.

[0051] Step S502 is a process of detecting a moving object region from the image input in step S501 by a background subtraction method. are processed.

[0052] Step S503 is a process for determining whether or not a first map has been received from mobile device 2, and is processed by first map receiving unit 102. If the first map has not been received, the process returns to step S501 and waits for the next frame. If the first map has been received, first map receiving unit 102 stores the information in first map information storage unit 108, and proceeds to the process of step S504.

[0053] Step S504 is a process of acquiring information on feature points corresponding to the image captured in step S501 for key frames in the first map stored in the first map information storage unit 108. Step S504 is processed by the map correspondence detection unit 103.

[0054] Step S505 is a process of identifying points corresponding to the moving object among the feature points in the first map, using the area of ​​the moving object detected in step S502 and the information of the corresponding feature points acquired in step S504. Step S505 is processed by the map update unit 105.

[0055] Step S506 is a process of generating a second map by removing information of the point corresponding to the moving object identified in step S505 from the first map. The generated information of the second map is held in the second map information holding unit 106. Step S506 is processed by the map update unit 105.

[0056] Step S507 is a process of transmitting the information of the second map stored in the second map information storage unit 106 to the mobile device 2. Step S507 is processed by the second map transmission unit 107.

[0057] Step S508 is a process for judging whether or not there is a request to end the process to the information processing device 1. If there is no request to end the process, the process returns to step S401 and waits for the input of the next frame. If there is a request to end the process, the process ends.

[0058] <Variation 1> In the first embodiment, the map correspondence detection unit 103 determines the correspondence between the image input from the image input unit 101 and the image of the key frame included in the information of the first map. In this case, a difference occurs between the time when the image of the key frame was captured and the time when the image input from the image input unit 101 was captured. As a result, for example, if an object that was captured by both the camera 100 and the camera 200 when the key frame was captured was not captured by the camera 100 at the time when the image was input from the image input unit 101, the moving object in the image of the key frame cannot be detected.

[0059] In this modified example, to address the above-mentioned problem, images from camera 100 are associated with the shooting time and periodically recorded in an image storage unit (not shown), and image input unit 101 inputs images from the image storage unit for use as needed.

[0060] In this modification, first map receiving unit 102 receives the shooting time of the key frame image in addition to the information described in the first embodiment, and stores it in first map information storage unit 108. Each of map correspondence detection unit 103, moving object detection unit 104, and map update unit 105 selects an image with a time closest to the shooting time of the key frame image from the image storage unit, inputs it via image input unit 101, and processes it. This makes it possible to generate a second map in which a flag indicating whether or not a feature point in the first map is a moving object is more accurately assigned to each feature point in the first map.

[0061] Note that it is not necessary to select an image taken at the closest time to the key frame image. In another example of the shooting time corresponding to the shooting time of the image of the key frame, a shooting time whose difference with the shooting time of the image of the key frame is within a threshold value may be used.

[0062] <Variation 2> A plurality of mobile devices may be connected to the information processing device 1 of the present invention, and in that case, the processing described in the first embodiment is performed for each of the mobile devices. In this case, the information stored in the second map information storage unit 106 may be shared among the processing for the plurality of mobile devices. For example, by using the method of integrating the information of the past second map stored in the second map information storage unit 106, as described in the map update unit 105 of the first embodiment, it becomes possible to determine the camera position and orientation using information obtained from other mobile devices.

[0063] However, using map information created by mobile devices with different characteristics may degrade the performance of the camera position and orientation. Therefore, in this modification, multiple maps are stored in the second map information storage unit 106 according to the characteristics of the mobile devices, and map information of mobile devices with the same characteristics can be used.

[0064] In this modification, when the first map receiving unit 102 receives the first map, in addition to the information described in the first embodiment, it simultaneously receives model information indicating the characteristics of the mobile device 2 and stores the model information in association with the first map. The model information indicating the characteristics of the mobile device 2 may include, for example, at least one of camera parameters such as the resolution, focal length, and distortion coefficient of the camera 200, and an identifier indicating the model of the mobile device 2. The second map information storage unit 106 is capable of storing multiple maps for each piece of model information.

[0065] When integrating the information of the new second map with the information of the second map stored in the second map information storage unit 106, the map update unit 105 selects a map corresponding to the received model information from the second map information storage unit 106 and integrates it. The map corresponding to the received model information is, for example, a second map associated with model information having the same or similar characteristics as the received model information. The integrated second map is stored in the second map information storage unit 106 in association with the model information.

[0066] As a result, for example, in a situation where the same model and other models are simultaneously connected to the information processing device 1, a second map can be generated in which only maps with the same model are integrated.

[0067] <Variation 3> In the first embodiment, the moving object detection unit 104 detects moving objects and associates each feature point in the first map with whether or not the object is a moving object. In this modification, each moving object is associated with each feature point in the first map.

[0068] In this modification, the moving object detection unit 104 provides identification information (such as a unique identifier) ​​for each of a plurality of moving objects detected in an image. The moving object detection unit 104 then tracks the objects using a means such as matching feature points between frames, provides the same identification information to image areas of the same object, and sends them to the map update unit 105. In this way, the second map generated by the map update unit 105 can provide each feature point in the first map with not only a flag indicating whether it is a moving object or not, but also the identification information of the moving object.

[0069] By obtaining a second map to which the identification information of the moving object is added, it becomes possible to obtain a group of feature points belonging to the same object (for example, it becomes clear that the feature points 404 to 406 in FIG. 4(a) belong to the same object). By tracking these feature points using the moving device 2, it becomes possible to not only estimate the position and orientation of the camera, but also track the position and orientation of a moving object.

[0070] <Embodiment 2> The system configuration shown in the first embodiment is one example of the system according to the present invention, and the division of processing roles between the information processing device 1 and the mobile device 2 can be changed as appropriate.

[0071] Fig. 6 is a block diagram showing a configuration example of a system according to this embodiment. As shown in Fig. 6, in the system according to this embodiment, the mobile device 2 includes a map correspondence detection unit 103, a map update unit 105, a first map information storage unit 108, and a second map information storage unit 106. Each of these functional units has been described in the first embodiment, so detailed description will be omitted.

[0072] In this embodiment, the information processing device 1 transmits to the moving device an image of the camera 100 acquired via the image input unit 101 and a detection result of a moving object by the moving object detection unit 104. In addition, the first map generation unit 201 of the moving device 2 generates a first map from an image acquired from the camera 200 and stores it in the first map information storage unit 108. The map correspondence detection unit 103 performs a correspondence detection process between the first map stored in the first map information storage unit 108 and the image of the camera 100 received from the information processing device 1, and sends the detection result to the map update unit 105. The map update unit 105 uses the result of the correspondence detection process sent from the map correspondence detection unit 103 and the detection result of the moving object sent from the moving object detection unit 104 to identify information corresponding to the moving object among the information included in the first map. The map update unit 105 generates a second map by the same process as in the first embodiment, and stores it in the second map information storage unit 106.

[0073] According to this embodiment, the second map obtained from information of other moving devices cannot be used, but otherwise the same effects as those of the first embodiment can be obtained. Also, in this embodiment, although the calculation cost of the moving device 2 increases, it is not necessary to send and receive maps. Since maps are large in data volume, the amount of communication data can be reduced.

[0074] Furthermore, the processes may be shared between the information processing device 1 and the mobile device 2 in a manner different from that of this embodiment, and some of the processes may be executed by still other devices.

[0075] <Other Examples> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0076] <Additional Notes> The present disclosure includes the following configurations and methods. [Configuration 1] a first map information storage unit that stores a first map including at least one image captured by a mobile device and feature information of the image; an image input unit for inputting an image from a fixed camera; a moving object detection unit that detects a moving object from an image captured by the fixed camera; a map correspondence detection unit that detects correspondence between feature information of an image of the fixed camera and feature information included in the first map; a map update unit that uses detection results of the moving object detection unit and the map correspondence detection unit to generate a second map in which information indicating whether or not the object is a moving object is added to the feature information included in the first map; and a second map information storage unit for storing the second map; An information processing device comprising: [Configuration 2] a map receiving unit that receives the first map stored in the first map information storage unit from another device; a map transmission unit that transmits at least a portion of the second map stored in the second map information storage unit to the mobile device; 2. The information processing device according to configuration 1, comprising: [Configuration 3] the map transmission unit selects map information of a position corresponding to a current location of the mobile device from the second map stored in the second map information storage unit and transmits the map information to the mobile device; 3. The information processing device according to configuration 2. [Configuration 4] the map update unit integrates the generated second map with a second map that was generated in the past and was stored in the second map information storage unit, and stores the second map generated by the integration in the second map information storage unit. 4. The information processing device according to any one of configurations 1 to 3. [Configuration 5] An image storage unit that records the image of the fixed camera together with the image capture time, The map correspondence detection unit detects correspondence by selecting an image captured by a fixed camera having a shooting time corresponding to a shooting time of an image included in the first map; the moving object detection unit and the map update unit perform processing using the image of the selected fixed camera. 5. The information processing device according to any one of configurations 1 to 4. [Configuration 6] the first map information storage unit stores, in addition to the first map, model information indicating a property of a device that captured the image included in the first map; the second map information storage unit stores a plurality of second maps according to the model information, the map update unit selects a second map corresponding to the model information when integrating the generated second map with a second map that was generated in the past and stored in the second map information storage unit. 6. The information processing device according to configuration 4 or 5. [Configuration 7] the moving object detection unit further generates identification information for each detected moving object, and tracks the moving object between frames of the image input from the image input unit; the map update unit generates a second map by adding identification information of the moving object to feature information in the space included in the first map. 7. The information processing device according to any one of configurations 1 to 6. [Configuration 8] a first map generating unit that generates a first map including an image captured by a camera and feature information of the image; a first map transmission unit configured to transmit the first map to another device; a second map receiving unit that receives, from the other device, a second map in which information indicating whether or not the object is a moving object is added to the feature information included in the first map; a self-location estimation unit that estimates a location of the moving device based on feature information of an object that is not the moving object among feature information included in the second map; A mobile device comprising: [Configuration 9] The timing at which the first map transmission unit transmits the first map to the other device is when it is determined that the mobile device is within a predetermined distance of a fixed camera associated with the other device; When the mobile device receives a new key frame from the camera, and when the mobile device adds new information from the camera to the second map received by the mobile device, At least one of the following: 9. The mobile device of embodiment 8. [Configuration 10] The mobile device is an HMD. 10. The mobile device of claim 8 or 9. [Configuration 11] An information processing device according to any one of configurations 1 to 7; A mobile device according to any one of configurations 8 to 10; A system including: [Configuration 12] A system including a mobile device and an information processing device, The mobile device comprises: a first map generating unit that generates a first map including an image captured by a camera associated with the mobile device and feature information of the image from the image; A self-location estimation unit that estimates a location of the moving device; Equipped with The information processing device includes: an image input unit for inputting an image from a fixed camera; a moving object detection unit that detects a moving object from an image captured by the fixed camera; Equipped with At least one of the mobile device and the information processing device a map correspondence detection unit that detects correspondence between feature information of an image of the fixed camera and feature information included in the first map; a map update unit that uses detection results of the moving object detection unit and the map correspondence detection unit to generate a second map in which information indicating whether or not the object is a moving object is added to the feature information included in the first map; and a second map information storage unit for storing the second map; Equipped with The self-location estimation unit estimates a location of the moving device based on feature information of an object other than the moving object, among feature information included in the second map. A system characterized by: [Configuration 13] A method for controlling an information processing device, comprising: a first map information storage step of storing a first map including at least one image captured by the mobile device and feature information of the image; A moving object detection step of detecting a moving object from an image of a fixed camera; a map correspondence detection step of detecting a correspondence between feature information of an image of the fixed camera and feature information included in the first map; a map updating step of generating a second map in which information indicating whether or not the object is a moving object is added to the feature information included in the first map, using detection results of the moving object detection step and the map correspondence detection step; a second map information storage step for storing the second map; A control method comprising: [Configuration 14] 1. A method for controlling a mobile device, comprising: a first map generating step of generating a first map including an image captured by the camera and feature information of the image; a first map transmission step of transmitting the first map to another device; a second map receiving step of receiving a second map from the other device in which information indicating whether or not the object is a moving object is added to the feature information included in the first map; a self-location estimation step of estimating a location of the moving device based on feature information of an object other than the moving object among feature information included in the second map; A control method comprising: [Configuration 15] A method for controlling a system including a mobile device and an information processing device, comprising: an image input step in which the information processing device inputs an image from a fixed camera; a moving object detection step in which the information processing device detects a moving object from an image of the fixed camera; a first map generation step in which the mobile device generates a first map from an image captured by a camera associated with the mobile device, the first map including the image and feature information of the image; a map correspondence detection step in which at least one of the mobile device and the information processing device detects correspondence between feature information of the image of the fixed camera and feature information included in the first map; a map update step in which at least one of the moving device and the information processing device generates a second map by adding information indicating whether or not the object is a moving object to the feature information included in the first map, using detection results of the moving object detection step and the map correspondence detection step; and a second map information holding step in which at least one of the mobile device and the information processing device holds the second map; The mobile device includes a self-location estimation unit that estimates a location of the mobile device; A control method comprising: [Configuration 16] 16. A program for causing a computer to execute each step of the control method according to any one of configurations 13 to 15. [Explanation of symbols]

[0077] 1: information processing device, 101: image input unit, 103: map correspondence detection unit 104: moving object detection unit, 105: map update unit 106: second map information storage unit, 108: first map information storage unit

Claims

1. a first map information storage unit that stores a first map including at least one image captured by a mobile device and feature information of the image; an image input unit that inputs an image from a fixed camera; a moving object detection unit that detects a moving object from an image captured by the fixed camera; a map correspondence detection unit that detects correspondence between feature information of the image of the fixed camera and feature information included in the first map; a map update unit that generates a second map in which information indicating whether or not the feature information included in the first map is the moving object is added using detection results from the moving object detection unit and the map correspondence detection unit; and a second map information storage unit that stores the second map; An information processing device comprising:

2. a map receiving unit that receives the first map stored in the first map information storing unit from another device; a map transmission unit that transmits at least a portion of the second map stored in the second map information storage unit to the mobile device; 2. The information processing apparatus according to claim 1, further comprising:

3. the map transmission unit selects map information of a position corresponding to a current location of the mobile device from the second map stored in the second map information storage unit and transmits the selected map information to the mobile device; 3. The information processing apparatus according to claim 2, wherein:

4. the map update unit generates a new second map by integrating a second map generated using the first map with a second map generated in the past and stored in the second map information storage unit, and stores the new second map in the second map information storage unit.

2. The information processing apparatus according to claim 1, wherein:

5. An image storage unit that stores images from the fixed camera in association with the time of shooting, The map correspondence detection unit detects a time of a photograph corresponding to the time of a photograph of an image included in the first map. Selecting images from a fixed camera having a time stamp to detect correspondence; the moving object detection unit and the map update unit perform processing using the image from the selected fixed camera.

2. The information processing apparatus according to claim 1, wherein:

6. the first map information storage unit stores, in association with the first map, model information indicating the characteristics of a device that captured the image included in the first map; the second map information storage unit stores a plurality of second maps according to the model information, the map update unit selects a second map corresponding to the model information when integrating a second map generated using the first map with a second map generated in the past and stored in the second map information storage unit.

5. The information processing apparatus according to claim 4,

7. the moving object detection unit further generates identification information for each detected moving object, and tracks the moving object between frames of images input from the image input unit; the map update unit generates a second map by adding identification information of the moving object to feature information in the space included in the first map.

2. The information processing apparatus according to claim 1, wherein:

8. A mobile device, comprising: a first map generating unit that generates a first map including an image captured by a camera and feature information of the image; a first map transmission unit that transmits the first map to another device; a second map receiving unit that receives, from the other device, a second map in which information indicating whether or not the object is a moving object is added to the feature information included in the first map; a self-location estimation unit that estimates a location of the moving device based on feature information of an object that is not the moving object, among feature information included in the second map; A mobile device comprising:

9. The timing at which the first map transmitter transmits the first map to the other device is determined as follows: when it is determined that the mobile device is within a predetermined distance of a fixed camera associated with the other device; When the mobile device receives a new keyframe from the camera, and When the mobile device adds new information from the camera to the second map received by the mobile device, At least one of the following is true: The mobile device of claim 8 .

10. The mobile device is an HMD. The mobile device of claim 8 .

11. An information processing device according to any one of claims 1 to 7; A mobile device according to any one of claims 8 to 10; A system including:

12. A system including a mobile device and an information processing device, The mobile device a first map generation unit that generates a first map including an image captured by a camera associated with the mobile device and feature information of the image from the image; a self-location estimation unit that estimates a location of the mobile device; Equipped with The information processing device includes: an image input unit that inputs an image from a fixed camera; a moving object detection unit that detects a moving object from an image captured by the fixed camera; Equipped with a map correspondence detection unit that detects correspondence between feature information of the image of the fixed camera and feature information included in the first map; a map update unit that generates a second map by adding information indicating whether the feature information included in the first map is the moving object or not, using the detection results of the moving object detection unit and the map correspondence detection unit; and a second map information storage unit that stores the second map; are provided in at least one of the mobile device and the information processing device, the self-location estimation unit estimates the location of the moving device based on feature information of an object that is not the moving object, among feature information included in the second map; A system characterized by:

13. A control method for an information processing device, comprising: a first map information holding step of holding a first map including at least one image captured by the mobile device and feature information of the image; a moving object detection step of detecting a moving object from an image of a fixed camera; a map correspondence detection step of detecting correspondence between feature information of the image of the fixed camera and feature information included in the first map; a map updating step of generating a second map in which information indicating whether or not the feature information included in the first map is the moving object is added using detection results of the moving object detection step and the map correspondence detection step; and a second map information holding step for holding the second map; A control method comprising:

14. 1. A method for controlling a mobile device, comprising: a first map generation step of generating a first map including an image captured by a camera and feature information of the image; a first map transmitting step of transmitting the first map to another device; a second map receiving step of receiving, from the other device, a second map in which information indicating whether or not the object is a moving object is added to the feature information included in the first map; a self-location estimation step of estimating a location of the moving device based on feature information of an object that is not the moving object, among feature information included in the second map; A control method comprising:

15. A method for controlling a system including a mobile device and an information processing device, comprising: an image input step in which the information processing device inputs an image from a fixed camera; a moving object detection step in which the information processing device detects a moving object from an image captured by the fixed camera; a first map generation step in which the mobile device generates a first map from an image captured by a camera associated with the mobile device, the first map including the image and feature information of the image; a map correspondence detection step in which at least one of the mobile device and the information processing device detects correspondence between feature information of the image of the fixed camera and feature information included in the first map; At least one of the mobile device and the information processing device is a map updating step of generating a second map by adding information indicating whether the object is a moving object to the feature information included in the first map using the map and the detection result of the map correspondence detection step; a second map information holding step in which at least one of the mobile device and the information processing device holds the second map; a self-location estimation step in which the mobile device estimates a position of the mobile device; A control method comprising:

16. A program for causing a computer to execute each step of the control method according to any one of claims 13 to 15.

Images