Information processing device, terminal device, information processing method, and program

Abstract

To reduce the processing load when a server map, which is a combination of a plurality of maps, is updated in response to changes in the configuration of the plurality of maps.SOLUTION: An information processing device comprises: acquisition means for acquiring a local map generated using map information including captured images and information on the position and orientation of an imaging unit when the images were captured for each predetermined source associated with any of a plurality of data spaces; first generation means for generating a server map from one or more local maps; acceptance means for accepting changes to the data space corresponding to the predetermined source; and update means for updating the server map by updating the local map of the predetermined source in which the data space has been changed in the server map.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, a terminal device, an information processing method, and a program.

Background Art

[0002] As technologies for fusing the real space and the virtual space in real time, Mixed Reality (MR) and Augmented Reality (AR) are known. These technologies are technologies for seamlessly fusing the real space and the virtual space created by a computer. The technologies of MR and AR are not limited to entertainment such as games, and are used in various applications such as map navigation apps. In addition, the technologies of MR and AR are widely used in industrial applications such as superimposing computer graphics (CG) on a physical object to confirm it without prototyping.

[0003] In an application using the technologies of MR and AR, when the user looks around through a display, the CG is aligned to a desired position and orientation so as to appear fixed in the real space. Conventionally, a technique for realizing alignment by defining a coordinate system of a space using a marker having a black-and-white characteristic pattern and detecting the marker whose position observed from a camera is known by image processing has been common.

[0004] Furthermore, in recent years, as a technique for achieving alignment without using markers with known positions, for example, a method called SLAM (Simultaneous Localization And Mapping) has been established. The mechanism of SLAM is described in Non-Patent Document 1. In SLAM, one or more stationary features are detected in the input camera image, and the relative position and orientation with respect to the detected features are obtained, thereby updating the position and orientation of the camera. The features detected from the camera image are points or edges where the luminance change in the image is relatively prominent and form corners. For example, when a cube of a color different from that of the table is placed on a plain table, each vertex of the cube is a feature point, and each contour line of the cube is an edge. Various applications can use the information on the position and orientation of the camera and the detected features to provide an image with CG placed at a specific position in the real space.

[0005] As the name indicates, SLAM is a technique that performs self-position and orientation estimation and mapping to create a surrounding environmental map from camera images. The map created by SLAM is different from a so-called geometrically accurate map. When alignment fails (is lost) in the map created by SLAM, it is possible to search for a similar image from the map created in the past and continue alignment using the found image. Also, the map by SLAM can optimize the map using images of the same object viewed from various angles and continuously achieve more accurate alignment over a wider range.

[0006] The map is composed of keyframe information including images captured at predetermined time intervals or for predetermined moving distances (referred to as keyframe images) and information on the position and orientation of the camera when the keyframe images were captured. The keyframe information further includes information on feature points in the keyframe image and information regarding the connection with other keyframe information.

[0007] Feature points are points where the change in luminance with the surroundings is greater than a predetermined amount, like corners in an image. In the estimation of the self-position and orientation in SLAM, the position and orientation of the camera can be estimated as the relative position and orientation with respect to feature points by extracting and tracking a plurality of feature points from the captured image. Also, by giving the position and orientation of known reference points, the absolute position and orientation of the self (camera) in space can be obtained. This is possible. The map is composed of a collection of keyframe information (hereinafter also referred to as keyframes), and the keyframe information is stored in memory or a file.

[0008] Since feature points in the real space are fixed to a certain extent, in a space where multiple devices are arranged, the work of each device performing mapping individually every time is redundant. Therefore, instead of performing mapping with a single device, a method has been proposed in which multiple devices generate and share a map by SLAM. Non-Patent Document 2 discloses a method of constructing a global map by combining local maps constructed by multiple robots.

Prior Art Documents

Patent Documents

[0009]

Patent Document 1

Non-Patent Documents

[0010]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0011] In business applications and the like, there may be cases where it is desired to avoid maintaining and managing maps generated by multiple devices as a single piece of data. Keyframe images are camera images observed at various positions and postures, and may include, for example, images of products under development and images of employees' faces. From the perspective of information security, it is desirable to manage maps separately for each use, such as for in-house development use and off-site demonstration use. Patent Document 1 discloses a configuration in which IoT sensor data is transmitted over a network separated for each operator when uploading it to the cloud.

[0012] However, when map data is separated and stored based on a configuration according to the roles of users, objects, rooms, etc., the configuration for separating the map data can often change in actual operation. When changes occur in the roles of people, objects, rooms, and personnel changes, etc., from the perspective of security, it is desirable that the map data be separated according to the changed configuration. When there is a change in the configuration, the combined server map on the server side is once separated and reconstructed according to the changed configuration. For example, when the group to which a certain client terminal belongs is changed, the server map in which maps from multiple terminals are combined is deleted once. The server can reconstruct the server map by having each terminal resend a map based on the new configuration. However, every time the configuration is changed, if the server map is reconstructed by acquiring maps from each terminal, the communication volume and processing load increase.

[0013] An object of the present invention is to provide an information processing apparatus that reduces the processing load when a server map obtained by combining a plurality of maps is updated in response to a change in the configuration of the plurality of maps.

Means for Solving the Problems

[0014] A first aspect of the present invention includes an acquisition unit that acquires a local map generated using map information including a captured image and information on the position and orientation of an imaging unit when the captured image was captured, for each predetermined origin associated with any one of a plurality of data spaces, a first generation unit that generates a server map from one or more of the local maps, a reception unit that receives a change in the data space corresponding to the predetermined origin, and an update unit that updates the server map by updating the local map of the predetermined origin in which the data space has changed, among the server maps. The information processing apparatus is characterized by having these units. A second aspect of the present invention includes a step of acquiring a local map generated using map information including a captured image and information on the position and orientation of an imaging unit when the captured image was captured, for each predetermined origin associated with any one of a plurality of data spaces, a step of generating a server map from one or more of the local maps, a step of receiving a change in the data space corresponding to the predetermined origin, and a step of updating the server map by updating the local map of the predetermined origin in which the data space has changed, among the server maps. The information processing method is characterized by having these steps.

[0015] A third aspect of the present invention is a program for causing a computer to function as each unit of the information processing apparatus described above.

[0016] According to the present invention, it is possible to reduce the processing load when a server map obtained by combining a plurality of maps is updated in response to a change in the configuration of the plurality of maps.

Effects of the Invention

[0017] According to the present invention, it is possible to reduce the processing load when a server map obtained by combining a plurality of maps is updated in response to a change in the configuration of the plurality of maps.

Brief Description of the Drawings

[0018]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Embodiments for Carrying Out the Invention

[0019] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0020] <Embodiment 1> [Hardware Configuration] FIG. 1 is a diagram illustrating the configuration of an information processing system 100. The information processing system 100 includes a client device 110, which is an example of a terminal device, and a server device 120, which is an example of an information processing device. The client device 110 includes an imaging device 111, an input unit 112, a storage unit 113, a display unit 114, a control unit 115, and a bus 116. The server device 120 includes an input unit 122, a storage unit 123, a control unit 125, and a bus 126.

[0021] FIG. 1 illustrates the configuration of an information processing system 100 in which the client device 110 and the server device 120 each have roles as a client and a server. The configuration of the information processing system 100 is not limited to this, and a configuration in which a plurality of client devices 110 communicate with each other in a Peer to Peer communication method (P2P communication method) may also be used. In this case, each device included in the information processing system 100 is not distinguished as a client or a server and has the same configuration as the client device 110.

[0022] The imaging device 111 of the client device 110 can capture continuous images. The imaging device 111 can utilize the captured images as digital information. The imaging device 111 may be, for example, a monocular camera, a stereo camera, or a camera equipped with a Depth sensor. The type of camera of the imaging device 111 may be an RGB camera, an infrared camera, or an event camera. The imaging device 111 may be disposed inside the client device 110 or may be disposed in another individual physically separated from the client device 110.

[0023] The input unit 112 is a device or operating member for giving instructions to the control unit 115, and specifically includes input devices such as a keyboard, a keypad, buttons, switches, a mouse, a touch pad, and a gesture input device.

[0024] The storage unit 113 is a storage medium such as a semiconductor memory, a hard disk, or a solid state drive, and stores programs and data for executing processing by the client device 110. The data stored in the storage unit 113 includes, for example, camera parameters of the imaging device 111 used for calculating the position and orientation of the imaging device 111 at the time of imaging from the captured image by the imaging device 111.

[0025] In addition, the data stored in the storage unit 113 includes key frame information and map data generated from the key frame information. The map generated by the client device 110 is also referred to as a local map. Note that a part of the programs and data used by the client device 110 is not limited to the storage unit 113 and may be acquired from an external data source (for example, a data server, a network storage, or an external memory).

[0026] The display unit 114 is a display. The control unit 115 is a processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor). The control unit 115 realizes various functions of the client device 110 by reading out and executing a program stored in the storage unit 113 or another storage medium into a memory such as a RAM (Random Access Memory).

[0027] Bus 116 connects the imaging device 111, the input unit 112, the storage unit 113, the display unit 114, and the control unit 115 to each other. Bus 116 is not limited to the wiring inside the client device 110 that physically connects the components of the client device 110 in close proximity. Bus 116 may be a transmission path such as a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet. The method of providing information via Bus 116 may be, in addition to the method of continuously transmitting information, a method of transmitting information buffered in a file or memory to the control unit 115 or the like at a desired timing.

[0028] The input unit 122 of the server device 120 is a device or operating member for giving instructions to the control unit 125, and specifically includes input devices such as a keyboard, a keypad, buttons, switches, a mouse, a touch pad, and a gesture input device.

[0029] The storage unit 123 is a storage medium such as a semiconductor memory, a hard disk, or a solid state drive, and stores programs and data for executing processing by the server device 120. The data stored in the storage unit 123 includes, for example, data of a server map generated based on a local map acquired from the client device 110. Note that a part of the programs and data used by the server device 120 is not limited to the storage unit 123 and may be acquired from an external data source (for example, a data server, a network storage, or an external memory).

[0030] The control unit 125 is a processor such as a CPU or a DSP. The control unit 125 realizes various functions of the server device 120 by reading a program stored in the storage unit 123 or another storage medium into a memory such as a RAM and executing it.

[0031] Bus 126 interconnects the input unit 122, the memory unit 123, and the control unit 125. Bus 126 is not limited to the wiring inside the server device 120 that physically connects each component of the server device 120 in close proximity. Bus 126 may be a transmission path such as a LAN, a WAN, or the Internet. The method of providing information via Bus 126 may be, in addition to the method of continuously transmitting information, a method of transmitting the information buffered in a file or memory to the control unit 115 or the like at a desired timing.

[0032] Note that the server device 120 is not limited to being configured by a single device, and the processing according to each of the above configurations may be realized by a plurality of devices.

[0033] Network 130 is a communication path that interconnects the client device 110 and the server device 120. Network 130 is, for example, a network such as a LAN, a WAN, or the Internet. Also, the client device 110 and the server device 120 may be physically included as separate virtual machines in one computer. When the client device 110 and the server device 120 are physically included in one computer, network 130 is configured as a virtual network.

[0034] [Functional Configuration] FIG. 2 is a diagram illustrating the functional configuration of the information processing system 100 according to Embodiment 1. The client device 110 includes an imaging unit 211, a position and orientation acquisition unit 212, a local map management unit 213, and a local map storage unit 214. The server device 120 includes a configuration management unit 221, a server map management unit 223, and a server map storage unit 224.

[0035] The imaging unit 211 (imaging device 111) constantly images the external world to be the object of map creation and converts the captured image into digital information. The imaging unit 211 transmits the captured image to the position and orientation acquisition unit 212. The imaging unit 211 may sequentially transmit the captured image to the position and orientation acquisition unit 212, or may save the captured image in a file format and transmit it to the position and orientation acquisition unit 212 collectively.

[0036] The position and orientation acquisition unit 212 acquires the image captured by the imaging unit 211 and acquires the position and orientation of the imaging unit 211 (imaging device 111) when the image is captured. The position and orientation of the imaging unit 211 can be calculated, for example, by a method called SLAM (Simultaneous Localization and Mapping). The position and orientation acquisition unit 212 extracts and tracks a plurality of features from the space, and uses camera parameters, which are known values specific to the imaging unit 211, to calculate the relative position and orientation of the imaging unit 211 with respect to the extracted features. The position and orientation acquisition unit 212 transmits the calculated position and orientation to the application, and the application can draw a CG object according to the received position and orientation.

[0037] The position and orientation acquisition unit 212 executes mapping processing. The mapping processing is a process of creating key frame information (map information) corresponding to the position and orientation of the imaging unit 211 based on predetermined creation conditions. The key frame information includes the captured image by the imaging unit 211 (hereinafter also referred to as the key frame image) and the information on the position and orientation of the imaging unit 211 when the captured image is captured.

[0038] The position and orientation acquisition unit 212 creates key frame information according to creation conditions based on a general algorithm. For example, the position and orientation acquisition unit 212 may create key frame information at regular intervals, or may create key frame information when there is a movement of a predetermined distance or a change in posture of a predetermined amount from the position and orientation when the previous key frame was created. Note that the timing for creating the key frame information is not limited to the timing based on the creation conditions other than the above, and may be a timing determined by combining various conditions.

[0039] The key frame information will be described with reference to FIG. 3. In the example of FIG. 3, when the imaging unit 211 moves along the trajectory 301, the position and orientation acquisition unit 212 creates key frames KF1 to KF6. The position and orientation acquisition unit 212 creates key frame information 302 for, for example, the key frame KF1.

[0040] The key frame information 302 includes a key frame image 303 which is an image captured when creating a key frame. The position and orientation acquisition unit 212 extracts one or more feature points 304 indicated by star marks in the key frame image 303. The position and orientation acquisition unit 212 acquires information such as the position of the extracted feature points, and records it in the storage unit 123 as the feature point information of the key frame information 302. Further, the key frame information 302 includes the position and orientation of the imaging unit 211 when the key frame image 303 is captured, the position coordinates of the key frame, and connection information with other key frames.

[0041] The map 305 is an aggregate of the key frame information 302. The map 305 generated using the key frame information in the client device 110 is also referred to as a local map. Note that the feature point information extracted for each key frame image may be stored in association with the map 305 independently of each key frame information 302.

[0042] The local map management unit 213 receives the key frame information created by the position and orientation acquisition unit 212. The local map management unit 213 generates a local map using the received key frame information.

[0043] When generating a server map from multiple local maps, from the perspective of information security, etc., it is desirable to separate the destinations where each local map is stored for each use, such as for in-house development use, off-site demo use, etc. Specifically, it is desirable that each local map is physically stored in separate storage or physically managed by separating the network for each use. Also, each local map is not limited to being physically separated, and the storage area or network may be virtually separated and stored. Also, each local map may be logically separated and managed by restricting access to maps stored in unauthorized spaces using storage encryption, user account authentication, and network domains, etc. Each area where the storage or network is physically or virtually separated is hereinafter referred to as a data space. In Embodiment 1, the data space of the server map is separated by physically or virtually separating the storage.

[0044] The local map management unit 213 adds information indicating a predetermined origin (hereinafter referred to as origin information) to the key frame information received from the position and orientation acquisition unit 212. The predetermined origin is S It includes an imaging device (HMD equipped with a camera, vehicle, robot, etc.) operating LAM, a PC (personal computer), a user, an area (room or place), etc. The local map is generated, for example, for each predetermined origin, and each key frame information included in the same local map is associated with the same origin information. Also, the predetermined origin is associated with any of a plurality of data spaces. Note that the predetermined origin may be associated with a plurality of data spaces.

[0045] The information of the predetermined origin represents the origin of the key frame information. The information of the predetermined origin may include an identifier indicating the origin, such as the ID of the imaging device operating SLAM, the ID of the PC, the ID of the user, the ID of the room or place, or character information.

[0046] The key frame information can identify which data space it corresponds to (which data space it belongs to) based on the origin information set by the local map management unit 213. The local map management unit 213 records the key frame information with the origin information added as a local map in the local map storage unit 214.

[0047] The local map management unit 213 transmits the key frame information (local map) stored in the local map storage unit 214 to the server map management unit 223. The local map management unit 213 may transmit the key frame information to the server map management unit 223 at the timing when new key frame information is received from the position and orientation acquisition unit 212. Also, the local map management unit 213 may transmit the key frame information received from the position and orientation acquisition unit 212 to the server map management unit 223 at regular intervals. Further, the local map management unit 213 may transmit the key frame information to the server map management unit 223 at the timing when the data volume of the key frame information received from the position and orientation acquisition unit 212 exceeds a predetermined amount.

[0048] Also, when a change occurs in the configuration of the data space, the local map management unit 213 extracts and transmits the requested key frame information from the local map storage unit 214 in response to a request from the server map management unit 223. A change in the configuration of the data space occurs, for example, when the data space associated with a predetermined origin is changed to another data space.

[0049] The local map storage unit 214 stores the key frame information with the origin information added by the local map management unit 213. The local map is an aggregate of the key frame information stored in the local map storage unit 214.

[0050] The configuration management unit 221 receives an instruction to change the definition information of the data space (hereinafter also referred to as configuration information) that stores the local map. The configuration information is information that defines the association between the local map and the data space generated for each predetermined source. The configuration management unit 221 can receive a change in the configuration information from an external system or a user via the input unit 122 of the server device 120.

[0051] The instruction to change the configuration information includes information on the sources (such as imaging devices, PCs, persons, areas, etc.) associated with each data space, and information on which data space the data space corresponding to which source has been changed to. The configuration management unit 221 updates the current configuration information according to the received change instruction. The configuration management unit 221 transmits the updated configuration information to the server map management unit 223. The configuration management unit 221 may transmit only the information on the data space in which there has been a change in the configuration information to the server map management unit 223.

[0052] The server map management unit 223 receives key frame information (local map) from the local map management units 213 of one or more client devices 110 and generates a server map. The server map management unit 223 determines which data space the key frame information corresponds to based on the source information of the key frame information. The server map management unit 223 records the key frame information in the determined data space in the server map storage unit 224. The server map is generated from one or more local maps. The server map management unit 223 may generate a server map by combining a plurality of local maps as an aggregate of key frame information, or may generate a server map by applying a predetermined combination process to each key frame information by a known technique.

[0053] As a predetermined combination process, the server map management unit 223 can generate one server map, for example, by rearranging the key frames of individual local maps in a unified coordinate system and combining overlapping feature point information. The server map management unit 223 may further optimize the information on the position and orientation and the coordinates of the feature points. The server map management unit 223 can generate an accurate server map over a wide range by mutually correcting and combining a plurality of local maps.

[0054] When there is one data space to be managed, the server map management unit 223 records the key frame information in the data space. When there are a plurality of data spaces to be managed, the server map management unit 223 records the key frame information in association with the data space corresponding to the source information of the key frame information.

[0055] The server map management unit 223 updates the configuration information of the data space based on the change instruction of the configuration information received by the configuration management unit 221. When there is a change instruction for the configuration information, that is, when the data space corresponding to a predetermined source is changed, the server map management unit 223 receives the configuration information of the updated data space from the configuration management unit 221.

[0056] The server map management unit 223 extracts the key frame information associated with the information of a predetermined source where the corresponding data space has been changed from the server map stored in the server map storage unit 224. The server map management unit 223 deletes the extracted key frame information from the server map. As a result, the key frame information of the source is deleted from the data space associated with the predetermined source before the change of the configuration information. Note that the server map management unit 223 may invalidate the extracted key frame information without deleting it, or may change the access authority according to the data space after the change.

[0057] The server map management unit 223 instructs the client device 110 that created the key frame information (local map) associated with the deleted predetermined source (PC, person, area, etc.) to transmit the key frame information associated with the deleted predetermined source. The client device 110 that has received the instruction from the server map management unit 223 transmits the local map generated from the key frame information associated with the deleted predetermined source to the server map management unit 223. Note that the client device 110 may determine whether to transmit the local map to the server map management unit 223 based on an instruction from the user.

[0058] The server map storage unit 224 receives and stores the key frame information from the server map management unit 223. The server map storage unit 224 can restrict access to the server map from an unauthenticated client device 110 or an unauthenticated user using known techniques such as encryption and authentication processing. When the server map storage unit 224 receives a request to extract key frame information by specifying specific source information from the server map management unit 223, it can extract the key frame information associated with the specified source information and transmit it to the server map management unit 223.

[0059] [Example 1] In Example 1, a local map is created for each client device 110 as a predetermined source. This is an example. For example, when the data space corresponding to the client device 110 is changed from the data space for customer demos to the data space for in-house development, the key frame information associated with the client device 110 stored in the data space for customer demos is deleted from the server map. The server device 120 updates the server map by updating the local map of the employee whose data space has been changed. The server device 120 can update the server map by reacquiring the local map of the client device 110 whose data space has been changed and recording the reacquired local map in the data space for in-house development. In the first embodiment, the plurality of data spaces are a plurality of separated storages. The server map is separately stored in the plurality of storages for each data space.

[0060] The server map generation process according to the first embodiment will be described with reference to FIG. 4. In step S401, the imaging unit 211 of the client device 110 captures an image for generating a local map. The imaging unit 211 continuously captures images (for example, at 60 fps), and transmits the captured image data (captured image) to the position and orientation acquisition unit 212. The imaging unit 211 may transmit the captured images sequentially, buffer them temporarily in the memory and then transmit them, or transmit a desired imaging sequence in the form of a file or the like all at once.

[0061] In step S402, the position and orientation acquisition unit 212 estimates the position and orientation of the imaging unit 211 when the captured image is captured, and creates key frame information based on the captured image and the information on the position and orientation of the imaging unit 211. The position and orientation acquisition unit 212 can estimate the position and orientation of the imaging unit 211 by a known technique called SLAM. The position and orientation acquisition unit 212 creates key frame information, for example, at regular intervals of time or for every certain distance movement of the imaging unit 211. The position and orientation acquisition unit 212 transmits the created key frame information to the local map management unit 213.

[0062] In step S403, the local map management unit 213 adds source information to the key frame information received from the position and orientation acquisition unit 212. The key frame information with the added source information is stored in the local map storage unit 214 as a local map.

[0063] The source information in the first embodiment includes the identifier of the client device 110 that generated the local map. The identifier of the client device 110 is an ID that can identify the client device 110, such as the IP address of the client device 110 or the PC name. The local map management unit 213 adds the IP address or PC name of the client device 110 as source information to the key frame information and records it in the local map storage unit 214.

[0064] When the local map management unit 213 receives key frame information from the position and orientation acquisition unit 212, it may sequentially add source information and record it in the local map storage unit 214. Also, the local map management unit 213 may buffer the key frame information temporarily and then record it according to desired conditions, or record the key frame information collectively after a series of imaging sequences by the imaging unit 211. In this case, the local map management unit 213 may add source information to the key frame information sequentially as it receives the key frame information, or add source information to each key frame information at the timing of recording it in the local map storage unit 214.

[0065] In step S404, the local map management unit 213 transmits the data of the local map stored (recorded) in the local map storage unit 214 to the server device 120. The local map management unit 213 instructs the server map storage unit 224 to store the transmitted local map in the data space corresponding to the source of the local map.

[0066] Note that when a single server map is managed by a plurality of server devices 120, the local map The uplink management unit 213 transmits the local map to a plurality of server devices 120. Further, the local map management unit 213 may transmit the local map only to the server device 120 that manages the data space corresponding to the origin of the local map.

[0067] In order to suppress the traffic volume, the client device 110 does not need to resend the key frame information that has been transmitted to the server device 120 once, unless it is changed. In this case, the client device 110 extracts the unsent key frames from the key frame information in the local map and transmits them to the server device 120.

[0068] The server device 120 may be an information processing device that is arranged at a location different from the client device 110 and is connected to the client device 110 via a network. Further, the server device 120 may be a virtualized information processing device.

[0069] In step S405, the server map management unit 223 records the local map received from the local map management unit 213 in the data space associated with the origin of the local map in the server map storage unit 224. One or more data spaces managed by the server map storage unit 224 are spaces protected using encryption with a password and authentication by various known techniques. Each data space restricts access from an unauthenticated client device 110 or an unauthenticated user. When the server map storage unit 224 has a plurality of data spaces, the plurality of data spaces may be separate storage devices.

[0070] Which data space the local map is recorded in is determined based on the origin information included in each key frame information. The origin information includes information such as the identifier of the imaging device or the identifier of the PC. The origin information can be, for example, the IP address of the PC. The IP address is associated with any of the data spaces managed by the server map storage unit 224.

[0071] Using FIGS. 5 to 7, the correspondence relationship between the client device 110 as a predetermined source and the data space will be described.

[0072] FIG. 5 is a conceptual diagram showing the correspondence relationship between a PC and the data space. The PC is an example of the client device 110. Three PCs are respectively associated with the in-house development data space 501 or the customer demo data space 502. Specifically, the PC with the IP address 192.168.11.1 is associated with the in-house development data space 501. The PCs with the IP addresses 192.168.12.1 and 192.168.12.2 are associated with the customer demo data space 502. The in-house development data space 501 and the customer demo data space 502 may be configured as separate storages from each other.

[0073] Note that FIG. 5 shows an example in which the PC (client device 110) has the imaging unit 211 and generates a local map using the captured image, but the PC may be connected to an external imaging device having the imaging unit and acquire the captured image from the imaging device.

[0074] FIG. 6 is a table exemplifying the relationship between a PC and a subnet mask. PC601 is an identifier such as the IP address of the PC. The table in FIG. 6 defines the relationship between PC601 and the subnet mask 602 to which PC601 belongs. FIG. 7 is a table exemplifying the relationship between the data space and the subnet mask. The table in FIG. 7 defines the relationship between the data space 701 and the subnet mask 702 indicating the belonging range of the PC.

[0075] FIG. 8 is a diagram for explaining the source information of the key frame information according to the first embodiment. FIG. 8 The table shows the source information 802 corresponding to the key frame 801. The source information 802 is the identifier of the PC, for example, the IP address of the PC. The source information of KF1 - KF3 is PC1 with an IP address of 192.168.11.1, the source information of KF4 is PC2 with an IP address of 192.168.12.1, and the source information of KF5 is PC3 with an IP address of 192.168.12.2. From FIGS. 6 and 7, PC1 has a subnet mask of 192.168.11.x and is associated with the data space for in - house development. Also, PC2 and PC3 have a subnet mask of 192.168.12.x and are associated with the data space for customer demos.

[0076] KF1 - KF3 are stored as the local map 1 with the source information being PC1 in the in - house development data space of the server map storage unit 224. KF4 is stored as the local map 2 with the source information being PC2 in the customer demo data space of the server map storage unit 224. In this case, when the local map management unit 213 of PC3 transmits the key frame information of KF5 to the server device 120, the server map management unit 223 records the key frame information of KF5 as the local map 3 in the customer demo data space.

[0077] As shown in FIG. 8, the source information is saved in association with the key frame information. However, the source information of each key frame may be stored as a part of the server map separately from the respective key frame information. Note that in FIG. 8, for simplicity of explanation, five key frames are used for the explanation. However, the local maps generated for each source are assumed to actually contain more key frame information.

[0078] Using FIG. 9, the server map update process of Embodiment 1 will be described. In step S901, the configuration management unit 221 (receiving means) receives a change in configuration information. The configuration information is information indicating the association between a predetermined source and a data space, and is changed when the data space corresponding to the predetermined source is changed. The configuration management unit 221 receives, for example, an instruction to change the data space corresponding to the source of the PC from the user via the input unit 122 (such as a keyboard or a mouse).

[0079] For example, PC2 with an IP address of 192.168.12.1 belongs to the data space for customer demos. Suppose that, according to an instruction from the user, the data space to which PC2 belonging to the data space for customer demos belongs is changed to the data space for in-house development. The configuration management unit 221 notifies the server map management unit 223 that the data space corresponding to PC2 with an IP address of 192.168.12.1 has been changed.

[0080] In step S902, the server map management unit 223 deletes the key frame information associated with a predetermined source (PC2) whose belonging data space has been changed. For example, the server map management unit 223 uses the IP address 192.168.12.1 of PC2 whose data space has been changed as a search key, extracts and deletes the key frame information (local map 2) associated with PC2 from the server map storage unit 224. In the example of FIG. 8, KF4 associated with PC2 with an IP address of 192.168.12.1 is extracted and deleted.

[0081] In step S903, the server map management unit 223 acquires the local map (key frame information) of a predetermined source (PC2) whose belonging data space has been changed. For example, the local map management unit 213 of PC2 extracts the key frame information associated with PC2 from the local map storage unit 214. The local map management unit 213 transmits the extracted key frame information to the server device 120.

[0082] Note that the local map management unit 213 presents the key frame information extracted in step S903 to the user, and before transmitting it to the server device 120, it may confirm with the user whether to update the local map corresponding to a predetermined location (PC2) in the server map. The information presented to the user may be the content obtained by visualizing the extracted key frame information, and may include various information such as a key frame image, the position coordinates of the key frame, the creation date and time, and the source information.

[0083] When the user instructs not to update the local map corresponding to PC2 in the server map, in step S903, the local map management unit 213 does not transmit the key frame information associated with the predetermined location (PC2) to the server device 120. In this case, the server map management unit 223 may not execute the process of step S902. That is, after receiving an instruction from the user on whether to update the local map, the server map management unit 223 may determine whether to delete the key frame information associated with the predetermined location (PC2) where the belonging data space has been changed.

[0084] In step S904, the server map management unit 223 updates the server map with the local map obtained in step S903. The server map management unit 223 records the obtained local map in the newly associated data space. For example, the server map management unit 223 records the key frame information (local map) associated with PC2 in the newly associated data space for in-house development.

[0085] The server map management unit 223 may combine the local map obtained in step S903 with other local maps already stored as the server map using known techniques.

[0086] In the first embodiment, the data of the server map used in SLAM is stored separately as storage. Therefore, the server device 120 can securely manage the data of the server map for each data space (storage) corresponding to the location.

[0087] Also, when the data space of the local map that constitutes the server map is changed, it is only necessary to update the part corresponding to the local map whose belonging data space has been changed, and the load of the update process of the server map is reduced. Specifically, even when the belonging (department, etc.) of the client device 110 that generates the local map changes, the server device 120 can reconfigure the server map by updating some of the local maps without having to recreate the combined server map from scratch.

[0088] [Embodiment 2] Embodiment 2 is an example in which a local map is created for each person as a predetermined source. For example, when the data space corresponding to a certain employee is changed from the data space for customer demos to the data space for in-house development, the key frame information associated with the employee (predetermined source) stored in the data space for customer demos is deleted from the server map. The server device 120 updates the server map by updating the local map of the employee whose data space has been changed. The server device 120 can update the server map by reacquiring the local map of the employee whose data space has been changed and recording the reacquired local map in the data space for in-house development. In Embodiment 2, as in Embodiment 1, the plurality of data spaces are a plurality of separated storages. The server map is separately stored in a plurality of storages for each data space.

[0089] The process of acquiring the position and orientation of the imaging unit 211 by SLAM in the client device 110, creating key frame information, and generating a local map is the same as in Embodiment 1. However, the source information added to the key frame information includes the identifier of the person in Embodiment 2, whereas it includes the identifier of the client device 110 in Embodiment 1. The identifier of the person is an ID that can identify the person, such as the login ID of the person (user) who operates the client device 110 that generated the local map. That is, it is an ID that can identify the person, such as the login ID of the person (user) who operates the client device 110 that generated the local map.

[0090] Among the server map generation processes in FIG. 4, the parts different from those in the first embodiment will be described. In step S403, the local map management unit 213 adds an identifier such as a person's login ID as origin information to the key frame information and records it in the local map storage unit 214.

[0091] In step S405, the server map management unit 223 records the local map received from the local map management unit 213 in the data space associated with the origin of the local map in the server map storage unit 224.

[0092] Which data space the local map is recorded in is determined based on the origin information included in each key frame information. In the second embodiment, the origin information includes an identifier of the person operating the client device 110. The origin information can be, for example, the login ID of the person (user) operating the client device 110. The identifier of the person is associated with any one of the data spaces managed by the server map storage unit 224.

[0093] With reference to FIGS. 10 to 12, the correspondence relationship between the person as a predetermined origin and the data space will be described.

[0094] FIG. 10 is a conceptual diagram showing the correspondence relationship between the person and the data space. Three employees are the persons operating the client device 110, and each is associated with the in-house development data space 1001 or the customer demo data space 1002. Each employee may independently use one client device 110 having the imaging unit 211, or may share and use one client device 110.

[0095] Employee A is associated with the in-house development data space 1001. Employees B and C are associated with the customer demo data space 1002. The in-house development data space 1001 and the customer demo data space 1002 may be configured as separate storages from each other.

[0096] FIG. 11 is a table illustrating the relationship between a person and their affiliation. Person 1101 is an identifier of a user such as an employee's name or login ID. The table in FIG. 11 defines the relationship between Person 1101 and the affiliation 1102 of Person 1101. FIG. 12 is a table illustrating the relationship between a data space and an affiliation. The table in FIG. 12 defines the relationship between the data space 1201 and the affiliation 1202 of a person (employee).

[0097] FIG. 13 is a diagram for explaining the source information of key frame information according to Example 2. The table in FIG. 13 shows the source information 1302 corresponding to the key frame 1301. The source information 1302 is an identifier of a person (employee), for example, an employee's name or login ID. The source information of KF1 to KF3 is Employee A, the source information of KF4 is Employee B, and the source information of KF5 is Employee C. From FIGS. 11 and 12, Employee A has an affiliation with the Development Department and is associated with the data space for in-house development in the server map storage unit 224. Also, Employee B and Employee C have an affiliation with the Sales Department and are associated with the data space for customer demos in the server map storage unit 224.

[0098] KF1 to KF3 are stored as the local map A with the source information being Employee A in the data space for in-house development in the server map storage unit 224. KF4 is stored as the local map B with the source information being Employee B in the data space for customer demos in the server map storage unit 224. In this case, assume that the local map management unit 213 of the client device 110 operated by Employee C transmits the key frame information of KF5 to the server device 120. The server map management unit 223 records the key frame information of KF5 as the local map C in the data space for customer demos.

[0099] As shown in FIG. 13, the source information is stored in association with the key frame information. However, the source information of each key frame may be stored as a part of the server map separately from the respective key frame information. In FIG. 13, for the sake of simplicity of explanation, five key frames are used for explanation. However, the local map generated for each source is assumed to actually include more key frame information.

[0100] Among the server map update processes in FIG. 9, the parts different from those in the first embodiment will be described. In step S901, the configuration management unit 221 receives a change in the configuration information. The configuration information is information indicating the association between a predetermined source and a data space, and is changed when the data space corresponding to the predetermined source is changed.

[0101] The configuration management unit 221 receives, for example, an instruction from the user via the input unit 122 (such as a keyboard or a mouse) to change the data space corresponding to employee B from the data space for customer demos to the data space for in-house development. Note that the configuration management unit 221 may change the data space corresponding to employee B from the data space for customer demos to the data space for in-house development by receiving an instruction from the user to change the department to which employee B belongs from the sales department to the development department.

[0102] For example, employee B shown in FIG. 10 is in charge of tasks related to customer demos. The local map generated and saved by employee B is recorded in the data space for customer demos when transmitted to the server device 120. When employee B is to be in charge of tasks related to in-house development due to a personnel transfer, the configuration management unit 221 associates employee B with the data space for in-house development according to an instruction from the user. The configuration management unit 221 transmits the identifier of employee B to the server map management unit 223 as the information of the source where the data space has been changed. The identifier of employee B is information added as the source information of the key frame information associated with employee B in the server map, and is the name, login ID, etc. of employee B. The key frame information associated with employee B is, for example, the key frame information created based on the operations of employee B.

[0103] In step S902, the server map management unit 223 deletes the key frame information associated with a predetermined origin (employee B) whose belonging data space has been changed. For example, the server map management unit 223 extracts and deletes the key frame information (local map B) associated with employee B using the identifier of employee B whose data space has been changed as a search key. In the example of FIG. 13, KF4 associated with employee B is extracted and deleted.

[0104] In step S903, the server map management unit 223 acquires the local map (key frame information) of a predetermined origin (employee B) whose belonging data space has been changed. The server map management unit 223 instructs the client device 110 that created the key frame information associated with employee B to transmit the key frame information associated with the predetermined origin (employee B).

[0105] The client device 110 that created the key frame information associated with employee B can be determined, for example, based on the identifier of employee B and the login information of the client device 110, where employee B has login authority. Also, by pre-assigning the IP address of the client device 110 to the identifier of a person, the server device 120 can determine the client device 110 that created the key frame information associated with each employee. There may be a plurality of client devices 110 that create the key frame information associated with one employee.

[0106] The local map management unit 213 of the client device 110 extracts the key frame information associated with employee B from the local map storage unit 214 in response to an instruction from the server map management unit 223. The local map management unit 213 transmits the extracted key frame information to the server device 120.

[0107] Note that the local map management unit 213 may present the key frame information extracted in step S903 to the user and confirm with the user whether to update the local map corresponding to a predetermined location (employee B) in the server map before transmitting it to the server device 120. The information presented to the user may be the content obtained by visualizing the extracted key frame information, and may include various information such as key frame images, position coordinates of key frames, creation dates and times, and location information.

[0108] When the user instructs not to update the local map corresponding to employee B in the server map, in step S903, the local map management unit 213 does not transmit the key frame information associated with the predetermined location (employee B) to the server device 120. In this case, the server map management unit 223 may not execute the process of step S902. That is, the server map management unit 223 may determine whether to delete the key frame information associated with the predetermined location (employee B) whose affiliated data space has been changed after receiving an instruction from the user on whether to update the local map.

[0109] In step S904, the server map management unit 223 updates the server map with the local map obtained in step S903. The server map management unit 223 records the obtained local map in the newly associated data space. For example, the server map management unit 223 records the key frame information (local map) associated with employee B in the newly associated data space for in-house development.

[0110] Similar to the first embodiment, the server map management unit 223 may combine the local map obtained in step S903 with other local maps already stored as the server map using known techniques.

[0111] In Example 2, similar to Example 1, the server device 120 can securely manage the data of the server map for each data space (storage) corresponding to the origin. Also, when the data space of the local map constituting the server map is changed, the load of the update process of the server map is reduced. Specifically, even when the affiliation (department, etc.) of the person who generates the local map changes, the server device 120 can reconstruct the server map by updating a part of the local map without having to recreate the combined server map from scratch.

[0112] [Example 3] Example 3 is an example in which a local map is created for each area as a predetermined origin. For example, when the data space corresponding to a certain area (room) is changed from the data space for customer demos to the data space for in-house development, the key frame information associated with the area (predetermined origin) stored in the data space for customer demos is deleted from the server map. The server device 120 updates the server map by updating the local map of the area where the data space has been changed. The server device 120 can update the server map by reacquiring the local map of the area where the data space has been changed and recording the reacquired local map in the data space for in-house development. In Example 3, similar to Examples 1 and 2, the plurality of data spaces are separate storage devices. The server map is stored separately in a plurality of storage devices for each data space.

[0113] The process of obtaining the position and orientation of the imaging unit 211 by SLAM in the client device 110 and creating key frame information to generate a local map is the same as in Example 1. However, the origin information added to the key frame information includes the identifier of the area in Example 3. The identifier of the area is, for example, the ID of the room.

[0114] Among the server map generation processes in FIG. 4, the parts different from those in the first embodiment will be described. In step S403, the local map management unit 213 adds an identifier of the area as the origin information to the key frame information and records it in the local map storage unit 214.

[0115] In step S405, the server map management unit 223 records the local map received from the local map management unit 213 in the data space associated with the origin of the local map in the server map storage unit 224.

[0116] Which data space the local map is recorded in is determined based on the origin information included in each key frame information. In the third embodiment, the origin information includes information (identifier) for identifying the area where the client device 110 is located. The origin information can be, for example, the ID of the room. The identifier of the area is associated with any data space managed by the server map storage unit 224.

[0117] The correspondence relationship between the area as a predetermined origin and the data space will be described with reference to FIGS. 14 and 15.

[0118] FIG. 14 is a conceptual diagram showing the correspondence relationship between the area and the data space. Development Room A and Development Room B are associated with the in-house development data space 1401. The corridor and the demo space are associated with the customer demo data space 1402. The in-house development data space 1401 and the customer demo data space 1402 may be configured as separate storages from each other.

[0119] FIG. 15 is a table exemplifying the correspondence relationship between the area and the data space. Area 1501 is an identifier of the area such as the ID of the room. The table in FIG. 15 defines the relationship between area 1501 and the data space 1502 with which area 1501 is associated.

[0120] FIG. 16 is a diagram for explaining the source information of the key frame information according to Example 3. The table in FIG. 16 shows the source information 1602 corresponding to the key frame 1601. The source information 1602 is an identifier of a region, such as a room ID. The source information of KF1 is Development Room A, the source information of KF2 and KF3 is Development Room B, the source information of KF4 is the corridor, and the source information of KF5 is the demo space. From FIG. 15, Development Room A and Development Room B are associated with the data spaces for in-house development. Also, the corridor and the demo space are associated with the data spaces for customer demos.

[0121] KF1 to KF3 are stored in the data space for in-house development in the server map storage unit 224. KF4 is stored in the data space for customer demos in the server map storage unit 224. In this case, it is assumed that the local map management unit 213 of the client device 110 arranged in the demo space transmits the key frame information of KF5 to the server device 120. The server map management unit 223 records the key frame information of KF5 in the data space for customer demos.

[0122] As shown in FIG. 16, the source information is stored in association with the key frame information, but the source information of each key frame may be stored as a part of the server map separately from the respective key frame information. Note that in FIG. 16, for simplicity of explanation, five key frames are used for the explanation, but the local maps generated for each source are actually assumed to include more key frame information.

[0123] Among the server map update processes in FIG. 9, the parts different from Example 1 will be described. In step S901, the configuration management unit 221 receives a change in the configuration information. The configuration information is information indicating the association between a predetermined source and a data space, and is changed when the data space corresponding to the predetermined source is changed.

[0124] The configuration management unit 221 receives, for example, an instruction from the user via the input unit 122 (such as a keyboard or a mouse) to change the data space corresponding to the demo space from the data space for customer demos to the data space for in-house development.

[0125] For example, the demo space shown in FIG. 14 is used for customer demos, and the local map generated by the client device 110 arranged in the demo space is transmitted to the server device 120 and recorded in the data space for customer demos. When the demo space is changed to a development room due to a layout change, the configuration management unit 221 associates the demo space with the data space for in-house development according to an instruction from the user. The configuration management unit 221 transmits the identifier of the demo space to the server map management unit 223 as information on the source where the data space has been changed. The identifier of the demo space is information added as the source information of the keyframe information associated with the demo space in the server map, such as the ID of the room. The keyframe information associated with the demo space is, for example, the keyframe information created by the client device 110 arranged in the demo space.

[0126] In step S902, the server map management unit 223 deletes the keyframe information associated with a predetermined source (demo space) where the belonging data space has been changed. For example, the server map management unit 223 extracts and deletes the keyframe information associated with the demo space using the identifier of the demo space where the data space has been changed as a search key. In the example of FIG. 16, KF5 associated with the demo space is extracted and deleted.

[0127] In step S903, the server map management unit 223 acquires the local map (keyframe information) of a predetermined source (demo space) where the belonging data space has been changed. The server map management unit 223 instructs the client device 110 that created the keyframe information associated with the demo space to transmit the keyframe information associated with the predetermined source (demo space).

[0128] The client device 110 that creates keyframe information associated with the demo space is, for example, the client device 110 arranged in the demo space and can be determined based on the position information of the client device 110. The position information of the client device 110 can be obtained, for example, by GPS (Global Positioning System). Also, by pre-assigning the IP address of the client device 110 to the identifier of the area, the server device 120 can determine the client device 110 that has created keyframe information associated with each area. There may be multiple client devices 110 that create keyframe information associated with one area.

[0129] The local map management unit 213 of the client device 110 extracts the keyframe information associated with employee B from the local map storage unit 214 in response to an instruction from the server map management unit 223. The local map management unit 213 transmits the extracted keyframe information to the server device 120.

[0130] Note that the local map management unit 213 presents the keyframe information extracted in step S903 to the user and may confirm with the user whether to update the local map corresponding to the demo space in the server map before transmitting it to the server device 120. The information presented to the user may be the content obtained by visualizing the extracted keyframe information and may include various information such as keyframe images, position coordinates of the keyframes, creation date and time, and source information. The information presented to the user may be the content obtained by visualizing the extracted keyframe information and may include various information such as keyframe images, position coordinates of the keyframes, creation date and time, and source information.

[0131] When the user instructs not to update the local map corresponding to the demo space in the server map, in step S903, the local map management unit 213 does not transmit the key frame information associated with a predetermined source (demo space) to the server device 120. In this case, the server map management unit 223 may not execute the process of step S902. That is, after receiving an instruction on whether to update the local map, the server map management unit 223 may determine whether to delete the key frame information associated with a predetermined source (demo space) whose belonging data space has been changed.

[0132] In step S904, the server map management unit 223 updates the server map with the local map acquired in step S903. The server map management unit 223 records the acquired local map in the newly associated data space. For example, the server map management unit 223 records the key frame information (local map) associated with the demo space in the newly associated data space for in-house development.

[0133] Similar to the first embodiment, the server map management unit 223 may combine the local map acquired in step S903 with other local maps already stored as the server map using known techniques.

[0134] In the third embodiment, similar to the first embodiment, the server device 120 can securely manage the data of the server map for each data space (storage) corresponding to the source. Also, when the data space of the local map constituting the server map is changed, the load of the update process of the server map is reduced. Specifically, even when the belonging (department, etc.) of the area where the local map is generated changes, the server device 120 can reconfigure the server map by updating a part of the local map without rebuilding the combined server map from scratch.

[0135] In Examples 1 to 3, examples where the source information is a PC, a user, and a region were described, but the source information is not limited to these classifications. Also, the source information may be information combining a plurality of classifications. For example, key frame information including PC1 and employee A as source information, key frame information including PC1 and employee B as source information, and key frame information including PC2 and employee A as source information are all associated with different data spaces. Thus, the correspondence relationship between the source information and the data space can be set flexibly.

[0136] <Embodiment 2> In Embodiment 1, the data space of the server map is separated by physically or virtually dividing the storage. In contrast, in Embodiment 2, the data space of the server map is separated by dividing a physical network or a virtual network. The network space in Embodiment 2 corresponds to the data space in Embodiment 1.

[0137] The hardware configuration of the information processing system according to Embodiment 2 is the same as the configuration of the information processing system shown in FIG. 1. The functional configuration of the information processing system according to Embodiment 2 is substantially the same as the functional configuration of the information processing system shown in FIG. 2. Using FIG. 2, the details of the processing different from Embodiment 1 will be described.

[0138] The local map management unit 213 receives the key frame information created by the position and orientation acquisition unit 212. The local map management unit 213 uses the received key frame information to generate a local map. The local map management unit 213 adds source information to the key frame information received from the position and orientation acquisition unit 212. The local map management unit 213 stores the key frame information with the added source information in the local map storage unit 214 as a local map.

[0139] The local map management unit 213 transmits the key frame information (local map) stored in the local map storage unit 214 to the server map management unit 223. The local map management unit 213 may transmit the key frame information to the server map management unit 223 at the timing when new key frame information is received from the position and orientation acquisition unit 212. Also, the local map management unit 213 may transmit the key frame information received from the position and orientation acquisition unit 212 to the server map management unit 223 at regular intervals. Further, the local map management unit 213 may transmit the key frame information to the server map management unit 223 at the timing when the data amount of the key frame information received from the position and orientation acquisition unit 212 exceeds a predetermined amount.

[0140] The network for transmitting the key frame information from the client device 110 to the server device 120 is a network space divided into one or more parts (network slicing) according to the purpose. The divided networks (communication paths) are securely separated from each other. The transmission / reception network between the client device 110 and the server device 120 may be a physically divided communication path or a virtual independent communication path provided by a tunneling protocol.

[0141] The client device 110 can determine the network for transmitting the key frame information based on the origin information added to the key frame information. Similar to the correspondence relationship between the predetermined origin and the data space in Embodiment 1, the correspondence relationship between the predetermined origin and the network (network space) in Embodiment 2 is defined in advance.

[0142] Also, when a change occurs in the configuration of the network space, the local map management unit 213 extracts and transmits the requested key frame information from the local map storage unit 214 in response to a request from the server map management unit 223. A change in the configuration of the network space occurs, for example, when the network space associated with a predetermined origin is changed to another network space.

[0143] The configuration management unit 221 receives an instruction to change the definition information (configuration information) of the network space for storing the local map from a user or an external system. The instruction to change the configuration information includes information on the source (imaging device, PC, person, etc.) associated with each network space and information on which network the information of which source is transmitted to has been changed to. The configuration management unit 221 updates the current configuration information according to the received change instruction. The configuration management unit 221 transmits the updated configuration information to the server map management unit 223. The configuration management unit 221 may transmit only the information on the network space with changes in the configuration information to the server map management unit 223.

[0144] The server map management unit 223 receives key frame information (local map) from the local map management units 213 of one or more client devices 110 and generates a server map. The server map management unit 223 determines, based on the source information of the key frame information, in which network the key frame information is transmitted and received (which network space it corresponds to). The server map management unit 223 records the key frame information in the data space of the server map storage unit 224 corresponding to the determined network space.

[0145] When the number of network spaces to be managed by the server map management unit 223 is one, the key frame information is recorded in the data space corresponding to the network space. When there are multiple network spaces to be managed, the key frame information is transmitted to the server device 120 via the network determined based on the source information and the network space associated with the source information. The server map management unit 223 records the received key frame information in the data space corresponding to the network space associated with the source information.

[0146] Based on the change instruction of the configuration information received by the configuration management unit 221, the server map management unit 223 updates the configuration information of the network space. When there is a change instruction for the configuration information, that is, when the network space corresponding to a predetermined source is changed, the server map management unit 223 receives the configuration information of the updated network space from the configuration management unit 221.

[0147] The server map management unit 223 extracts the key frame information associated with the information of the predetermined source where the corresponding network space has changed from the server map stored in the server map storage unit 224. The server map management unit 223 deletes the extracted key frame information from the server map. As a result, the key frame information of the corresponding source is deleted from the data space corresponding to the network space to which the predetermined source was associated before the change of the configuration information. Note that the server map management unit 223 may invalidate the extracted key frame information without deleting it, or change the access authority according to the network space after the change.

[0148] The server map management unit 223 instructs the client device 110 that generated the key frame information (local map) associated with the deleted predetermined source (such as a PC, a person, an area, etc.) to transmit the key frame information associated with the deleted predetermined source. The client device 110 that has received the instruction from the server map management unit 223 transmits the local map generated using the key frame information associated with the deleted predetermined source to the server map management unit 223. Note that the client device 110 may determine whether to transmit the local map to the server map management unit 223 in response to the instruction from the server map management unit 223 based on an instruction from the user.

[0149] Using FIG. 17, the server map update process of Embodiment 2 will be described. FIG. 17 shows an example of the server map update process when the network space to which the client device 110 belongs is switched to another network space. The processes different from the server map update process of Embodiment 1 described in FIG. 9 will be described, and the detailed description of the processes similar to those of Embodiment 1 will be omitted. Note that the plurality of data spaces in Embodiment 1 correspond to the plurality of separated networks (network spaces) in Embodiment 2.

[0150] In step S1701, the configuration management unit 221 (reception means) receives a change in configuration information. The configuration information is information indicating the association between a predetermined source and a network space (corresponding to the data space of Embodiment 1), and is changed when the network space corresponding to the predetermined source is changed.

[0151] Using FIGS. 18(A) and 18(B), the change in the network space will be described. In the examples of FIGS. 18(A) and 18(B), the predetermined source is a client device as in Example 1 of Embodiment 1. FIGS. 18(A) and 18(B) show an example in which the server map is managed by two servers, a server device 1801 and a server device 1802. The server device 1801 and the server device 1802 have the same configuration as the server device 120 of Embodiment 1. The client device 1803 is a terminal device having a PC name "Client α" as an identifier, and has the same configuration as the client device 110 of Embodiment 1.

[0152] The client device 1803 records the created keyframe information (local map) in the local map storage unit 214. The server device 1801 is a server that manages the network space A (NW space A), and the client device 18 via the network NW_A The local map can be received from 03. The server device 1801 records the local map received from the client device 1803 in the data space of the server map 1804. The server device 1802 is a server that manages the network space B (NW space B), and can receive the local map from the client device 1803 via the network NW_B. The server device 1802 records the local map received from the client device 1803 in the data space of the server map 1805.

[0153] FIG. 18(A) shows the state before the network space of the client device 1803 is changed. The client device 1803 is associated with the network space A (NW space A). FIG. 18(B) shows the state after the network space of the client device 1803 is changed. The client device 1803 is associated with the network space B (NW space B).

[0154] In the state of FIG. 18(A), the configuration management unit 221 accepts a change in configuration information. Specifically, the configuration management unit 221 accepts an instruction from the user to change the network space to which the client device 1803 belongs from the NW space A to the NW space B. The configuration management unit 221 notifies the server device 1801 and the server device 1802 of the identifier of the client device 1803 and instructs a change in the configuration information according to the received change instruction.

[0155] In step S1702, the server map management unit 223 of the server device 1801 deletes the key frame information associated with the client device 1803 whose belonging network space has been changed. The server map management unit 223 extracts and deletes the key frame information associated with the client device 1803 from the server map 334 using the PC name "Client α" of the client device 1803 whose network space has been changed as a search key. The server map management unit 223 of the server map management unit 223 updates the configuration information with the network space corresponding to the client device 1803 being the NW space B.

[0156] In step S1703, similar to the server device 1801, the server map management unit 223 of the server device 1802 updates the configuration information with the network space corresponding to the client device 1803 being the NW space B.

[0157] Also, the server map management unit 223 acquires the local map (key frame information) of the client device 1803 whose belonging network space has been changed to the NW space B. The server map management unit 223 instructs the client device 1803 to transmit the key frame information associated with the client device 1803. The local map management unit 213 of the client device 1803 extracts the key frame information associated with the client device 1803 from the local map storage unit 214. The local map management unit 213 transmits the extracted key frame information (local map) to the server device 1802 in the NW space B via the network NW_B.

[0158] Note that the local map management unit 213 may present the extracted key frame information to the user and confirm with the user whether to update the local map of the client device 1803 (predetermined origin) in the server map before transmitting it to the server device 1802. The information presented to the user may be the content obtained by visualizing the extracted key frame information, and may include various information such as key frame images, position coordinates of key frames, creation dates and times, and origin information.

[0159] When the user instructs not to update the local map of the client device 1803 in the server map, in step S1703, the local map management unit 213 does not transmit the key frame information associated with the client device 1803 to the server device 1802. In this case, the server map management unit 223 may not execute the process of step S1702. That is, the server map management unit 223 may determine whether to delete the key frame information associated with the client device 1803 after receiving an instruction from the user on whether to update the local map. That is, the server map management unit 223 may determine whether to delete the key frame information associated with the client device 1803 after receiving an instruction from the user on whether to update the local map.

[0160] In step S1704, the server map management unit 223 of the server device 1802 updates the server map with the local map acquired in step S1703. The server map management unit 223 records the acquired local map in the data space of the server map 1805.

[0161] Similar to Example 1 of Embodiment 1, the server map management unit 223 of the server device 1802 may combine the local map acquired in step S1703 with other local maps already stored as the server map using known techniques.

[0162] In Embodiment 2, the data of the server map used for SLAM has a separated network space for transmission and reception for each local map constituting the server map. Therefore, the server devices 1801 and 1802 can securely manage the data of the server map for each network space. Also, when the network space of the local map constituting the server map is changed, only the part corresponding to the local map whose belonging network space has been changed needs to be updated, and the load of the update process of the server map is reduced.

[0163] <Modification Example> In Embodiment 1 and Embodiment 2, the client device 110 and the server device 120 have been described separately. However, in the modification example, the client device 110 and the server device 120 are configured to communicate with each other in a P2P communication method without distinguishing their roles. In the modification example, each device has a hardware configuration and a functional configuration capable of realizing the functions of the client device 110 and the server device 120.

[0164] In the P2P communication method, each device has its own server map. When an update of the server map occurs in a certain device, each device performs update confirmation and update processing of the server map through P2P communication. Note that the P2P communication method is not limited to a general method without a server, and may be a method including a server that manages index information.

[0165] Note that the above embodiments (including modification examples) are merely examples, and configurations obtained by appropriately modifying or changing the configurations of the above embodiments within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the configurations of the above embodiments are also included in the present invention.

[0166] <Other Embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and causing one or more processors in a computer of the system or device to read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0167] The disclosure of the present embodiment includes the following configurations, methods, and programs. (Configuration 1) An acquisition means for acquiring a local map generated using map information including a captured image and information on the position and orientation of an imaging unit when the captured image was captured, for each predetermined location associated with any of a plurality of data spaces; A first generation means for generating a server map from one or more of the local maps; A reception means for receiving a change in a data space corresponding to the predetermined location; An update means for updating the server map by updating the local map of the predetermined location where the data space has been changed, among the server maps; An information processing apparatus characterized by having the above. (Configuration 2) The plurality of data spaces are a plurality of separated storages. The information processing apparatus according to Configuration 1, characterized by the above. (Configuration 3) The information processing apparatus according to Configuration 1 or 2, characterized in that the information indicating the predetermined location includes an identifier of a terminal device that generated the local map. (Configuration 4) The information indicating the predetermined origin includes information for identifying the user who operates the terminal device that generated the local map. The information processing apparatus according to any one of Configurations 1 to 3, characterized by this. (Configuration 5) The information indicating the predetermined origin includes information for identifying the area where the terminal device that generated the local map is located. The information processing apparatus according to any one of Configurations 1 to 4, characterized by this. (Configuration 6) The plurality of data spaces are separate networks. The information processing apparatus according to Configuration 1, characterized by this. (Configuration 7) The information indicating the predetermined origin is stored in association with the map information. The information processing apparatus according to any one of Configurations 1 to 6, characterized by this. (Configuration 8) The update means determines whether to update the local map at the predetermined origin where the data space has been changed in the server map, based on a user instruction. The information processing apparatus according to any one of Configurations 1 to 7, characterized by this. (Configuration 9) The acquisition means acquires the local map from an external terminal device by means of a Peer to Peer communication method. The information processing apparatus according to any one of Configurations 1 to 8, characterized by this. (Configuration 10) The imaging unit, A position and orientation acquisition means for acquiring information on the position and orientation of the imaging unit when the imaging image is captured, A second generation means for generating the local map for each predetermined origin, In the information processing apparatus according to any one of Configurations 1 to 9, a transmission means for transmitting the local map at the predetermined origin where the data space has been changed, A terminal device characterized by having this. (Configuration 11) Before transmitting the local map of the predetermined location where the data space has been changed by the transmission means to the information processing apparatus, confirmation means for confirming with the user whether to update the local map in the server map is further provided. The terminal device according to Configuration 10, characterized in that. (Method) For each predetermined location associated with any one of a plurality of data spaces, obtaining a local map generated using map information including a captured image and information on the position and orientation of the imaging unit when the captured image was captured; Generating a server map from one or more of the local maps; Receiving a change in the data space corresponding to the predetermined location; Updating the server map by updating the local map of the predetermined location where the data space has been changed in the server map; An information processing method, characterized by comprising the steps of. (Program) A program for causing a computer to function as each means of the information processing apparatus according to any one of Configurations 1 to 9.

Explanation of Reference Numerals

[0168] 120: Server device (information processing apparatus), 122: Input unit, 125: Control unit 221: Configuration management unit, 223: Server map management unit

Claims

1. Obtaining means for obtaining a local map generated using map information including a captured image and information on the position and orientation of an imaging unit when the captured image was captured, for each predetermined location associated with any of a plurality of data spaces; First generation means for generating a server map from one or more of the local maps; Receiving means for receiving a change to a data space corresponding to the predetermined location; Updating means for updating the server map by updating the local map of the predetermined location in which the data space has been changed, among the server maps An information processing apparatus characterized by comprising the above.

2. The plurality of data spaces are a plurality of separated storages The information processing apparatus according to claim 1, characterized in that.

3. The information indicating the predetermined location includes an identifier of a terminal device that generated the local map, the information processing apparatus according to claim 1, characterized in that.

4. The information indicating the predetermined location includes information for identifying a user who operates the terminal device that generated the local map The information processing apparatus according to claim 1, characterized in that.

5. The information indicating the predetermined location includes information for identifying an area where the terminal device that generated the local map is disposed The information processing apparatus according to claim 1, characterized in that.

6. The plurality of data spaces are a plurality of separated networks The information processing apparatus according to claim 1, characterized in that.

7. The information indicating the predetermined location is stored in association with the map information The information processing apparatus according to claim 1, characterized in that.

8. The updating means determines whether to update the local map of the predetermined location in which the data space has been changed, in the server map, based on a user's instruction The information processing apparatus according to claim 1, characterized in that.

9. The obtaining means obtains the local map from an external terminal device by a Peer to Peer communication method The information processing apparatus according to claim 1, characterized in that.

10. The imaging unit; Position and orientation obtaining means for obtaining information on the position and orientation of the imaging unit when the captured image was captured; Second generation means for generating the local map for each predetermined location In the information processing apparatus according to any one of claims 1 to 9, transmission means for transmitting the local map of the predetermined location where the data space has been changed A terminal device characterized by comprising the same.

11. Before transmitting the local map of the predetermined location where the data space has been changed to the information processing apparatus by the transmission means, confirmation means for confirming to the user whether or not to update the local map in the server map is further provided The terminal device according to claim 10, characterized by the above.

12. A step of obtaining a local map generated using map information including a captured image and information on the position and orientation of the imaging unit when the captured image was captured, for each predetermined location associated with any one of a plurality of data spaces; A step of generating a server map from one or more of the local maps; A step of receiving a change in the data space corresponding to the predetermined location; A step of updating the server map by updating the local map of the predetermined location where the data space has been changed in the server map An information processing method characterized by comprising the same.

13. A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 9.

Images