Location Estimation System

Abstract

Position estimation in three-dimensional real space is realized based on captured images, and various guidance information is then provided to the user. SOLUTION: A space management server stores data of an integrated space 100 configured as a three-dimensional virtual space corresponding to real space. The space management server compares a user image 102, which is a captured image acquired from a user terminal, with the data of the integrated space 100 to identify position information indicating the three-dimensional position in real space of a point corresponding to the user image 102. The space management server provides this position information to the user terminal.

Description

[Technical field]

[0001] The present invention relates to techniques for estimating position based on images. [Background technology]

[0002] There is a technology called SLAM (Simultaneous Localization And Mapping) that can simultaneously estimate the location of the vehicle and create a map of the environment around the vehicle. There are various methods for SLAM, including those that use lasers and ToF (Time of Flight) sensors, as well as VisualSLAM, which uses cameras.

[0003] There is also a technology called Structure from Motion (SfM) that reconstructs the three-dimensional shape of an object from multiple captured images. This technology is made possible largely by advances in artificial intelligence (AI), especially deep learning technology. Comparing VisualSLAM and SfM, it is said that VisualSLAM is superior in real-time processing performance, while SfM is superior in image analysis capabilities. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6662979 Summary of the Invention [Problem to be solved by the invention]

[0005] By utilizing VisualSLAM technology, it is possible to create a three-dimensional map. If a three-dimensional map can be realized, it is believed that it will be possible to provide users with more useful information than a two-dimensional map.

[0006] On the other hand, the computational burden of creating a 3D map is much greater than that of a 2D map, so currently the map's coverage is inevitably limited. Even if it were possible to create a 3D map that covers a vast real space, the real space would change from moment to moment. Therefore, even if a 3D map could be created, it would have to be continually updated to match the changes in the real space.

[0007] The present invention was completed based on the recognition of the above problems, and its main objective is to provide a technology for realizing position estimation in three-dimensional real space based on captured images, and then providing various guidance information to the user. [Means for solving the problem]

[0008] A position estimation system in one aspect of the present invention includes an integrated space storage unit that stores data of an integrated space configured as a three-dimensional virtual space corresponding to real space, a position identification unit that identifies position information indicating the three-dimensional position in real space of a point corresponding to a user image by comparing the integrated space data with a user image, which is an image captured from a user terminal, and a guidance unit that provides the identified position information to the user terminal. Effect of the Invention

[0009] According to the present invention, it is possible to realize position estimation using a three-dimensional map corresponding to real space, and to provide various kinds of guidance information to the user. [Brief description of the drawings]

[0010] [Figure 1] FIG. 13 is a schematic diagram for explaining a method for generating an integrated space. [Diagram 2] FIG. 13 is a schematic diagram for explaining the expansion and updating of an integrated space. [Diagram 3] FIG. 1 is a schematic diagram for explaining the principle of position estimation. [Figure 4] FIG. 2 is a hardware configuration diagram of a space management server. [Diagram 5] FIG. 2 is a functional block diagram of a user terminal and a space management server. [Figure 6] FIG. 11 is a schematic diagram for explaining a process of generating a user space. [Figure 7] FIG. 11 is a data structure diagram of space configuration information. [Figure 8] FIG. 4 is a data structure diagram of object information. [Figure 9] FIG. 13 is a conceptual diagram for explaining a method for expanding an integrated space. [Figure 10] 11A and 11B are schematic diagrams for explaining directional correction of a user image and time parallax. [Figure 11] FIG. [Figure 12] 11 is a flowchart showing a process of position estimation based on a user image. [Figure 13] 13 is a flowchart showing a process of generating a user space. [Figure 14] FIG. 13 is a conceptual diagram for explaining an update condition. [Figure 15] 1 is a flowchart showing the expansion and updating process of the integrated space. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] In this embodiment, a large number of captured images are collected from various users, and a three-dimensional virtual space (hereinafter, referred to as an "integrated space 100") corresponding to real space is constructed from these captured images.

[0012] FIG. 1 is a schematic diagram for explaining a method for generating an integrated space 100. As shown in FIG. First, assume that a user R1 captures an image of his / her surroundings using a user terminal 200. The user terminal 200 may be a portable device equipped with an imaging function and a communication function, such as a smartphone, smart glasses, or a spherical camera. Smart glasses are a wearable device configured in the shape of glasses with a display built into the lenses. This display can display image information superimposed on the field of view.

[0013] A plurality of captured images (hereinafter, referred to as "user images 102") are acquired by the user terminal 200. More specifically, the user R1 captures a moving image of his / her surroundings by the user terminal 200, and a large number of user images 102 are acquired from this moving image. Although details will be described later, the position estimation system 500 in this embodiment includes a space management server 400 and a plurality of user terminals 200 that are interconnected via the Internet.

[0014] A three-dimensional virtual space showing the surrounding environment of the user R1 is generated by SfM technology from the user images 102 corresponding to various imaging directions. The three-dimensional virtual space generated by the multiple user images 102 is called the "user space 104." The user space 104 may be constructed as a hemispherical or cylindrical space, but for convenience of explanation, it is represented as a cube in Figs. 1 and 2.

[0015] A huge integrated space 100 is constructed as a collection of many user spaces 104 generated in this way. For example, if a user space 104 at a point P1 and a user space 104 at a point P2 adjacent to the point P1 are obtained, a larger integrated space 100 that covers the points P1 and P2 can be constructed by connecting the two user spaces 104.

[0016] Various pieces of information are associated with the user image 102. Specifically, these pieces of information include a user ID, an image capture date and time, an image capture angle, and a resolution. These pieces of information are collectively referred to as "image attribute information." A user ID, an acquisition date and time, image quality, and a three-dimensional image capture position (latitude, longitude, and altitude) in the real space are also associated with the user space 104. These pieces of attribute information associated with the user space 104 are referred to as "space attribute information." Note that the acquisition date and time of the user space 104 may be the image capture date and time of the user image 102 that is the basis of the user space 104, or may be the generation date and time of the user space 104, but the acquisition date and time in this embodiment will be described as the generation date of the user space 104.

[0017] In summary, a large number of user spaces 104 are generated from user images 102 obtained from user terminals 200 in various locations. Then, a huge integrated space 100 is generated from the large number of user spaces 104. Through such a mechanism, a huge integrated space 100 that ultimately covers most of the earth is constructed.

[0018] FIG. 2 is a schematic diagram for explaining the expansion and updating of the integrated space 100. As shown in FIG. As described above, the integrated space 100 is configured as a combination of multiple user spaces 104. A user space 104 that is part of the integrated space 100, in other words, a three-dimensional virtual space that represents a part of the integrated space 100, is called a "subspace 106." Although the user space 104 and the subspace 106 are the same thing, in the following description, the space before being integrated into the integrated space 100 will be referred to as the "user space 104," and the space after being integrated will be referred to as the "subspace 106."

[0019] The subspaces 106 include regular subspaces 106 (hereinafter referred to as "regular spaces") and reserve subspaces 106 (hereinafter referred to as "reserve spaces"). The integrated space 100 is expressed as a collection of regular spaces, but the subspaces 106 that were not adopted as regular spaces are also attached as reserve spaces. Hereinafter, when distinguishing between the normal space and the subspace, the subspace 106 as the normal space may be written as the "subspace 106 (normal)", and the subspace 106 as the spare space may be written as the "subspace 106 (spare)".

[0020] In FIG. 2, for point P1, the subspace 106 generated at acquisition date and time t2 (hereinafter, referred to as "subspace 106(P1, t2)") is the normal space. On the other hand, for point P2, there are three subspaces 106, namely, subspace 106(P2, t11), subspace 106(P2, t7), and subspace 106(P2, t9). Of these, subspace 106(P2, t11) is the normal space, and the remaining two are spare spaces. In this way, there is only one normal space at a certain point, and all other subspaces 106 at the same point are spare spaces. There is no limit to the number of spare spaces.

[0021] Here, assume that a user space 104 (P7, t14) is newly generated for the point P7. Since the subspace 106 of the point P7 is not included in the integrated space 100, the user space 104 (P7, t14) is added as a new subspace 106 (P7, t14: normal), and the integrated space 100 is expanded.

[0022] As another example, assume that a user space 104 (P3, t15) is newly generated for the point P3. The integrated space 100 already includes a subspace 106 (P3, t3: regular), a subspace 106 (P3, t1: spare), a subspace 106 (P3, t8: spare), and a subspace 106 (P3, t10: spare) for the point P3. When the update condition for the user space 104 (P3, t15) is satisfied, the user space 104 (P3, t15) is registered as a regular space of the integrated space 100, and the subspace 106 (P3, t3), which is the original regular space, is changed to a spare space. On the other hand, when the update condition is not satisfied, the user space 104 (P3, t15) is registered as a spare space of the integrated space 100. Details of the update condition will be described later in relation to FIG. 14.

[0023] Through the above-mentioned processing steps, the integrated space 100 is expanded and updated as appropriate. Since multiple preliminary spaces for the same point, in other words, preliminary spaces at multiple points in time, are included, the integrated space 100 can also be considered as a "space-time" that expands not only in the spatial direction but also in the time direction.

[0024] FIG. 3 is a schematic diagram for explaining the principle of position estimation. One of the purposes of creating the integrated space 100 is "position estimation." Suppose that user R2 captures a user image 102 at a certain point in a certain imaging direction. The space management server 400 searches for a spatial image 108 that is the same as or similar to the user image 102 within the integrated space 100. As will be described in detail later, the integrated space 100 is configured as a virtual space similar to the real space, that is, a so-called digital twin, in which the captured image (real scenery) is texture-mapped. Suppose that the spatial image 108 when viewed from a certain line of sight direction from point P1 is similar to this user image 102. In this case, the location of user R2 can be estimated to be "point P1."

[0025] Position information (latitude, longitude, altitude) is assigned in advance to each part of the integrated space 100. Therefore, by searching for a spatial image 108 similar to an image captured by user R2 within the integrated space 100, the position of user R2 can be identified (position estimation).

[0026] In practice, in order to reduce the calculation load, the point cloud data extracted from the user image 102 is compared with the point cloud data extracted from the spatial image 108. The similarity determination of the point cloud data is realized by applying known techniques established in VisualSLAM and the like.

[0027] In this embodiment, since real-time estimation is required for position estimation, VisualSLAM technology is used. Meanwhile, since accuracy is required for the user space 104, SfM technology is used to generate the user space 104. Also, position estimation may be performed not only from one captured image but also from multiple captured images captured as a video.

[0028] FIG. 4 is a diagram showing the hardware configuration of the space management server 400. As shown in FIG. The space management server 400 includes a storage 312 as a non-volatile memory for storing computer programs, a volatile memory 304 for developing programs and data, and a processor 300 (CPU) that incorporates (not shown) registers, an arithmetic unit, an instruction decoder, etc., and reads and executes programs from the memory 304. The processor 300 is connected to a relatively high-speed first bus 302. In addition to the memory 304, a NIC (Network Interface Card) is connected to the first bus 302. In addition to the above, other devices such as a GPU may be connected to the first bus 302.

[0029] The first bus 302 is connected to a relatively slow second bus 310 via a bridge 308. In addition to a storage 312, an output device 316 such as a monitor or a speaker is connected to the second bus 310. In addition, an input device 314 such as a mouse or a keyboard, and a peripheral device 318 such as a printer may also be connected to the second bus 310. The hardware configuration of the user terminal 200 is basically the same. It will be understood by those skilled in the art that the method of connection between first bus 302 and second bus 310 shown in FIG. 4 is just one example, and that other connection methods or configurations including other buses may also be used.

[0030] FIG. 5 is a functional block diagram of the user terminal 200 and the space management server 400. Each component of the user terminal 200 and the space management server 400 is realized by hardware including computing units such as a CPU and various co-processors, storage devices such as memory and storage, and wired or wireless communication lines connecting them, and software stored in the storage devices and supplying processing instructions to the computing units. The computer programs may be composed of device drivers, an operating system, various application programs located at higher layers than these, and libraries that provide common functions to these programs. Each block described below does not represent a hardware configuration but represents a functional block.

[0031] (User terminal 200) The user terminal 200 includes a user interface processing unit 206 , an imaging unit 202 , a communication unit 204 , a data processing unit 208 , and a data storage unit 210 . The user interface processing unit 206 accepts operations from the user via an input device such as a touch panel, and is responsible for processing related to the user interface, such as image display and audio output. The imaging unit 202 captures the external environment as a video using a camera. The communication unit 204 is responsible for communication processing with the space management server 400. The data storage unit 210 stores various data. The data processing unit 208 executes various processes based on the data input from the user interface processing unit 206, the user image captured by the imaging unit 202, the data received by the communication unit 204, and the data stored in the data storage unit 210. The data processing unit 208 also functions as an interface between the imaging unit 202, the communication unit 204, the user interface processing unit 206, and the data storage unit 210.

[0032] (Space management server 400) The space management server 400 includes a data processing unit 402 , a communication unit 404 , and a data storage unit 406 . The communication unit 404 is responsible for communication processing with the user terminal 200. The data storage unit 406 stores various information. The data processing unit 402 executes various processes based on the data acquired by the communication unit 404 and the data stored in the data storage unit 406. The data processing unit 402 also functions as an interface between the communication unit 404 and the data storage unit 406.

[0033] The data processing unit 402 includes a position identification unit 408 , a guidance unit 410 , an image analysis unit 414 , a user space generation unit 416 , an integrated space management unit 418 and a reward provision unit 424 . The position identification unit 408 identifies the current location of the user terminal 200 by comparing the user image 102 and the spatial image 108, as described in relation to FIG. 3. The guidance unit 410 provides various types of guidance information according to the user's current location. The guidance unit 410 includes a route generation unit 412. The route generation unit 412 identifies a route from the user's current location to the destination point. The guidance unit 410 provides the user with route information as part of the guidance information. The guidance information including the route information will be described later in relation to FIG. 11.

[0034] The image analysis unit 414 analyzes the images included in the integrated space 100 and generates guidance information. The user space generation unit 416 generates the user space 104 based on the user image 102.

[0035] The integrated space management unit 418 manages the integrated space 100. The integrated space management unit 418 includes an expansion unit 420 and an update unit 422. When a user space 104 is generated for a point (hereinafter referred to as an "unknown point") that is not registered in the integrated space 100, the expansion unit 420 expands the integrated space 100 by adding the user space 104 to the integrated space 100. The update unit 422 determines whether an update condition (described later) is satisfied for the newly generated user space 104, and updates a part of the integrated space 100 as necessary. The reward granting unit 424 grants a reward (described later) to a user who cooperates in the expansion and update of the integrated space 100. The data store 406 includes an integrated spatial store 426 that stores the spatial data for the integrated space 100 .

[0036] FIG. 6 is a schematic diagram for explaining the process of generating the user space 104. As shown in FIG. When generating the user space 104, first, the user terminal 200 captures a plurality of user images 102 as a moving image. The imaging unit 202 of the user terminal 200 adds the above-mentioned image attribute information to each user image 102. The imaging unit 202 of the user terminal 200 transmits the user images 102 with the added image attribute information to the space management server 400.

[0037] The user space generation unit 416 of the space management server 400 extracts feature points from the user image 102 and generates three-dimensional point cloud data (S1). Feature points are points extracted by a method known in SfM technology, and are extracted as landmarks on an image, such as edge points of a building. The point cloud data is three-dimensional data.

[0038] Next, the user space generation unit 416 adds frames (line segments) connecting feature points included in the point cloud data (S2). By adding the frames, a three-dimensional frame image is generated. The user space generation unit 416 specifies the closed area by the frames as a "plane" (S3). Finally, the user space generation unit 416 generates a user space 104 as a virtual space similar to the real space by texture mapping the user image 102 onto the plane specified in S3 (S4). The user space 104 generated in this way is integrated into the integrated space 100 as a partial space 106.

[0039] The space management server 400 may generate the user space 104 by itself, or may request the generation of the user space 104 from other servers. The user space 104 does not need to be generated in real time at the same time as the user image 102 is captured. The user space generation unit 416 generates the user space 104 by batch processing the user images 102 that have been received in advance during a time period with low computational load.

[0040] FIG. 7 is a data structure diagram of the space configuration information 110. As shown in FIG. The space configuration information 110 is stored in the integrated space storage unit 426 of the space management server 400. A position ID is associated with the partial space 106, and the position ID is associated with three-dimensional position information of the center point of the partial space 106. The space configuration information 110 shown in FIG. 7 shows a list of the partial space 106 (hereinafter, written as "partial space 106 (A01)") corresponding to position ID=A01. There are multiple partial spaces 106 (A01), one of which is a normal space and the rest are auxiliary spaces. In the space configuration information 110, the partial space 106 is associated with space attribute information.

[0041] The partial space 106 is identified by a space ID. The partial space 106 (A01) is associated with the partial space 106 (hereinafter, referred to as "partial space 106 (T01)") having a space ID of T01, the partial space 106 (T05), the partial space 106 (T08), the partial space 106 (T11), etc. When the user space 104 (A01) is acquired, the integrated space management unit 418 assigns a space ID to the user space 104 (A01) and registers it as the partial space 106 (A01).

[0042] According to FIG. 7, the partial space 106 (A01, T01) corresponds to the user space 104 acquired on "October 15, 2024". The user ID of the user who captured the user image 102 that is the basis of this partial space 106 (A01, T01) is "R01", and the image quality of the user image 102 is "A (high image quality)". The user space generation unit 416 classifies the image quality into a plurality of ranks based on the number of user images 102 (frame rate), resolution (number of pixels), clarity, etc. In this way, the user space generation unit 416 generates space attribute information of the partial space 106 from the image attribute information of the user image 102.

[0043] The partial space 106 (A01, T05) corresponds to the user space 104 acquired (generated) on June 4, 2021 from the user image 102 of the user (R11). Similarly, the partial space 106 (A01, T08) corresponds to the user space 104 acquired (generated) on October 18, 2024 from the user image 102 of the user (R04). The integrated space management unit 418 sets the partial space 106 (A01, T08) as a normal space and sets the other partial spaces 106 as reserve spaces. Details of the method of selecting a normal space will be described later.

[0044] FIG. 8 is a diagram showing the data structure of the object information 120. As shown in FIG. The object information 120 is stored in the integrated space storage unit 426 of the space management server 400. The integrated space 100 includes various "objects." An object here refers to a building, a tree, a fountain, a traffic light, and other physical objects. An object ID is assigned to each object.

[0045] The integrated space management unit 418 identifies objects included in the partial space 106. The integrated space management unit 418 identifies a space enclosed by a "plane" as an object. The integrated space management unit 418 assigns an object ID to the object and associates the object ID with position information of the object. The object may be automatically identified by the integrated space management unit 418, or the operator of the space management server 400 may manually specify the object included in the partial space 106.

[0046] According to FIG. 8, the position information of an object with object ID=J01 (hereinafter, referred to as "object (J01)") is (36, 133, 10). For example, assume that object (J01) is a "five-story pagoda." During position estimation, if a five-story pagoda is shown in user image 102, position identification unit 408 knows that the user is near the five-story pagoda, and can quickly identify the user's current location using the position information associated with object (J01: five-story pagoda).

[0047] The object is associated with guidance information as various auxiliary information related to the object. Each piece of guidance information is identified by a guidance ID. The object (J01: five-story pagoda) is associated with guidance information with a guidance ID of G01 (hereinafter, written as "guidance information (G01)"). The guidance information (G01) is explanatory information about the five-story pagoda, which states that "the pagoda represents five worlds, earth, water, fire, wind, and sky, in that order from the bottom." Therefore, if the user goes near the five-story pagoda, the user can learn an explanation about the five-story pagoda from the above guidance information (G01). The guidance information will be described in more detail with reference to FIG. 11. The guidance information may be set by the administrator of the space management server 400, or may be automatically generated by the image analysis unit 414 (described later).

[0048] FIG. 9 is a conceptual diagram for explaining a method for enlarging the integrated space 100. As shown in FIG. Assume that the integrated space 100 includes a partial space 106 (A01) but does not include a partial space 106 (A02). Assume that a user moves a distance E from the center of point A01 in the V1 direction as shown in FIG. 9 and reaches point A02. The user terminal 200 transmits a user image 102 (A02) to the space management server 400. The user space generation unit 416 generates a user space 104 (A02).

[0049] The position identification unit 408 estimates the user's moving direction V1 and moving distance E from the user image 102 obtained in time series. The integrated space management unit 418 checks whether the integrated space 100 includes the partial space 106 (A02), in other words, whether the user image 102 (A02) is a "known scene". If the partial space 106 (A02) does not exist, the enlargement unit 420 adds the user space 104 (A02) to the integrated space 100 as the partial space 106 (A02: normal). Since the moving direction V1 and the moving distance E have been estimated, the enlargement unit 420 can also estimate the position information of the A02 point. The enlargement unit 420 sets a space ID to the partial space 106 (A02: normal) and registers it in the space configuration information 110. The enlargement unit 420 also registers the position information in association with the A02 point.

[0050] Furthermore, suppose that the user moves a distance E from point A02 in the V2 direction to reach point A03. The user space generation unit 416 generates the user space 104 (A03) in a similar manner. If the partial space 106 (A03) does not exist, the enlargement unit 420 adds the user space 104 (A03) to the integrated space 100 as the partial space 106 (A03: normal). The enlargement unit 420 sets a space ID for the partial space 106 (A03) and registers it in the space configuration information 110. The enlargement unit 420 registers A03 in association with position information.

[0051] Through the above process, the integrated space 100 is gradually expanded. The user can receive location estimation services at point A01, which is already included in the integrated space 100. When moving to point A02, the expansion unit 420 identifies point A02, which is an unknown point, based on the moving direction V1 and moving distance E from point A01, which is a known point, and expands the integrated space 100 by adding a partial space 106 (A02: regular) to the integrated space 100. After the partial space 106 (A02: regular) is added, other users can receive location estimation services when they enter point A02. The same applies to point A03.

[0052] When the user terminal 200 is equipped with an acceleration sensor, a pedometer, or the like, the user space generation unit 416 may estimate the movement direction V1 and the movement distance E based on information obtained from these sensors.

[0053] FIG. 10 is a schematic diagram for explaining the directional correction and time parallax of the user image 102. In FIG. In FIG. 10, the user's current location is P. The user faces in a direction D1 at a certain timing T1, and captures a user image 102 in the direction D1 (hereinafter, referred to as "user image 102(D1:T1)"). Next, at timing T2, the user faces in a direction D2, and captures a user image 102(D2:T2). At this time, an object 130 appears in the user image 102(D1:T1) and the user image 102(D2:T2).

[0054] The imaging directions of the object 130 are different between the user image 102 (D1:T1) and the user image 102 (D2:T2). Therefore, triangulation of the object 130 is possible using the user image 102 (D1:T1) and the user image 102 (D2:T2). If the angle difference between the direction D1 and the direction D2 (hereinafter referred to as "time parallax") can be identified, three-dimensional measurement of the object 130 is possible. The principle of three-dimensional measurement is as described above, but in reality, in VisualSLAM or SfM, the three-dimensional shape of the object 130 is estimated from a large number of user images 102 whose imaging directions are slightly different.

[0055] The above-mentioned three-dimensional measurement by "time parallax" is premised on the assumption that the change from the direction D1 to the direction D2 is at a constant speed. Therefore, when the imaging direction changes suddenly, the three-dimensional measurement becomes inaccurate. In FIG. 10, the imaging direction changes suddenly from the direction D2 (timing T2) to the direction D3 (timing T3). The object 130 is reflected in the user image 102 (D2:T2), but the object 130 is not reflected at all in the user image 102 (D3:T3). If it is assumed that the angle difference (time parallax) between the user image 102 (D2:T2) and the user image 102 (D3:T3) is small, it becomes difficult to appropriately generate the user space 104 from the partial space 106 (D2:T2) in the vicinity of the partial space 106 (D3:T3). The three-dimensional measurement based on time parallax has a problem in that it is vulnerable to sudden changes in the user image 102, particularly sudden changes in the imaging direction.

[0056] In this embodiment, the user space generation unit 416 compares the user image 102 with the spatial image 108, identifies the correct imaging direction of the user image 102, and then generates the user space 104. It is assumed that the partial space 106 of the point P has already been registered in the integrated space 100, and first, the user space generation unit 416 identifies the imaging direction D1 by comparing the user image 102 (D1:T1) with the spatial image 108. Similarly, the imaging directions D2 and D3 are also identified.

[0057] Since the correct imaging direction of each user image 102 can be identified while referring to the integrated space 100, the user space generation unit 416 can correctly recognize that the imaging directions of the user image 102(D2) and the user image 102(D3) are significantly different, in other words, the angle difference between each user image 102. By checking the imaging direction of each user image 102 (hereinafter referred to as "direction correction"), the user space generation unit 416 can generate an appropriate user space 104.

[0058] However, when generating the user space 104 at an unknown point, the direction cannot be corrected by comparing the user image 102 and the spatial image 108. Therefore, when generating the user space 104 at an unknown point, careful and careful imaging is required. Even if the user space 104 generated for the unknown point is inappropriate, the partial space 106 is updated by the newly acquired user space 104 as described in relation to FIG. 2, so that the integrated space 100 will be optimized in due course.

[0059] In addition to the direction correction, when the user suddenly accelerates, there is a problem that the imaging points of the user image 102 obtained at one timing and the user image 102 obtained at the next timing are largely different. In this case, too, the user space generation unit 416 can identify the correct imaging points of each user image 102 by comparing the user image 102 with the space image 108 (hereinafter, referred to as "position correction").

[0060] FIG. 11 is a screen diagram of the guide screen 140. The guidance screen 140 is displayed as an AR (Augmented Reality) image by the user interface processing unit 206 of the user terminal 200. Fig. 11 shows the guidance screen 140 displayed when the user is in an underground mall of a train station. When the user terminal 200 is a pair of smart glasses, the guidance screen 140 is displayed as an image of the user's field of view.

[0061] The user sets a destination point to be traveled to on the user terminal 200. In this example, a station ticket gate is set as the destination point. The user interface processing unit 206 accepts input from the user, and the communication unit 204 transmits the destination point "ticket gate" to the space management server 400. Information specifying the destination point is called a "route condition."

[0062] The position identification unit 408 first identifies the user's current location based on the user image 102. The route generation unit 412 references the integrated space 100 and generates route information indicating a travel route from the user's current location to the destination. The guidance unit 410 transmits the route information to the user terminal 200, thereby displaying route guidance 142 on the guidance screen 140. At this time, the guidance unit 410 also displays guidance information 144 on the screen, such as "50m to the ticket gate," as the distance from the current location to the destination. The user can reach the destination "ticket gate" by moving according to the route guidance 142 displayed on the guidance screen 140.

[0063] Some of the various buildings shown on the guidance screen 140 are registered in advance as objects 130 in the integrated space 100. For example, the object 130 (J01) in FIG. 11 is a "sushi restaurant" and is associated with guidance information 144 (J01). The guidance information 144 (J01) is "sushi." The guidance unit 410 displays "sushi," which is the guidance information 144 (J01), near the object 130 (J01) shown on the guidance screen 140. Similarly, the guidance information 144 (J11) is displayed on the object 130 (J11) of the "pasta restaurant," and the guidance information 144 (J08) is displayed on the object 130 (J08) of the "bookstore." While walking through the underground mall, the user can simultaneously receive information about the various facilities (objects 130) that are visible.

[0064] The image analysis unit 414 may automatically generate the guidance information 144 by performing image analysis on various objects 130 included in the integrated space 100. For example, when the character "pasta" is detected in an image (e.g., a portion of a signboard) of an object 130 (J11) in the integrated space 100, the image analysis unit 414 may generate the guidance information 144 (J11) "pasta restaurant."

[0065] Famous trademarks from around the world may be registered in advance in the data storage unit 406 of the space management server 400. When the logo of a famous pasta restaurant "SX" is detected from the captured image texture-mapped onto the object 130, the image analysis unit 414 can identify the brand "SX" of this pasta restaurant.

[0066] The image analysis unit 414 may search for various information from Internet sites based on trademarks or exterior images of buildings, and generate information posted on external sites as guide information 144. For example, for an object 130 corresponding to a "five-story pagoda," the image analysis unit 414 may execute an image search based on the exterior image of the "five-story pagoda" to obtain an explanatory article such as "The five worlds of earth, water, fire, wind, and sky are represented in that order from the bottom up." The image analysis unit 414 may associate this explanatory article with the "five-story pagoda" object 130 as guide information 144.

[0067] FIG. 12 is a flow chart showing the process of position estimation based on the user image 102. In estimating the position, first, the space management server 400 acquires a plurality of user images 102 from the user terminal 200 (S10). The position identification unit 408 compares the space images 108 included in the integrated space 100 with the received user images 102 to search for a space image 108 similar to the user image 102 (S12). If a space image 108 similar to the user image 102 is detected, the position identification unit 408 identifies the user's current location based on the partial space 106 including this space image 108 (S14). At this time, the position identification unit 408 identifies not only the three-dimensional position information of the current location (imaging location) in the partial space 106 but also the imaging direction. The communication unit 404 of the space management server 400 notifies the user terminal 200 of the position information corresponding to the current location (S16).

[0068] If an object 130 registered in the object information 120 is shown in the user image 102 (Y of S18), the guiding unit 410 transmits guidance information 144 associated with this object 130 to the user terminal 202. As shown in the guidance screen 140 of Fig. 11, the guidance information 144 is superimposed and displayed on the guidance screen 140 displayed by the user terminal 200 (S20). If a registered object 130 is not shown (N of S18), the processing of S20 is skipped. When the user sets a destination point, route guidance 142 is also notified from the space management server 400 as a type of guidance information 144 .

[0069] The position identification unit 408 does not estimate the position of one user image 102, but estimates the position of the current location based on multiple user images 102. For example, assume that, of the tenth user image 102, the partial space 106 (A01) is estimated by nine user images 102, and the partial space 106 (A02) is estimated by the remaining one user image 102. In this case, the position identification unit 408 estimates the user's current location as the partial space 106 (A01). In this way, the position identification unit 408 performs position estimation using multiple user images 102, and estimates the most probable position, in other words, the position estimated by the largest number of user images 102, as the current location.

[0070] FIG. 13 is a flowchart showing the process of generating the user space 104. The space management server 400 generates a user space 104 based on a user image 102 collected in advance during a time period when the calculation load is low, more specifically, during a time period when the utilization rates of the CPU, memory, and storage are equal to or lower than a predetermined threshold. In this embodiment, the guide unit 410 performs position estimation based on the VisualSLAM technology from the user image 102 acquired as appropriate. Meanwhile, the user image 102 acquired in this manner is stored in the data storage unit 406. The user space generation unit 416 generates the user space 104 during an appropriate time period based on the user image 102 accumulated in the data storage unit 406.

[0071] The user space generation unit 416 corrects the imaging direction and imaging position of each user image 102 by comparing it with the spatial image 108 included in the integrated space 100 (S32). That is, the direction and position of the user image 102 are corrected based on the already obtained spatial image 108. The user space generation unit 416 generates point cloud data from the corrected user space 104 (S34). The user space generation unit 416 then identifies a frame from the point cloud data (S36) and generates a surface (S38). Finally, the user space generation unit 416 texture maps the user image 102 onto the surface of the user space 104 (S40). Through the above process, a user space 104 that imitates the real space is created.

[0072] FIG. 14 is a conceptual diagram for explaining the update condition. Assume that a subspace 106 (A01) has already been obtained for the same point, for example, point A01. Assume that a new user space 104 (A01) is now acquired. Usually, the subspace 106 (A01: normal) and the newly generated user space 104 (A01) do not completely match. More specifically, all feature points detected in the subspace 106 (A01: normal) and all feature points detected in the user space 104 (A01) are not necessarily located at the same position.

[0073] The difference between the partial space 106 (A01: normal) and the user space 104 (A01) (the magnitude of the deviation in the positions of the feature points) may be due to a measurement error, or may be due to a large change in the real space. If the real space has changed, it is desirable to register the latest user space 104 (A01) as the normal space. On the other hand, if the difference between the user space 104 (A01) and the partial space 106 (A01: normal) is not so large, there is no need to bother to register the latest user space 104 (A01) as the normal space.

[0074] For example, if the image quality of the subspace 106 (A01: regular) is better than that of the user space 104 (A01), it is reasonable to maintain the subspace 106 (A01: regular) as the regular space.

[0075] The update unit 422 compares the surface 146a included in the partial space 106 (A01: normal) with the surface 146b in the user space 104 (A01) that is in contact with the surface 146a or is within a predetermined distance from the surface 146a, thereby calculating a Z4 value, which will be described later, indicating the magnitude of the difference, and determines the update condition based on the Z4 value.

[0076] The four vertices (characteristic points) of the surface 146a (normal space) are Q1 to Q4. The update unit 422 specifies the surface 146b (user space 104) as the corresponding surface of the surface 146a. The four vertices (characteristic points) of the surface 146b are B1 to B4.

[0077] Next, the update unit 422 identifies four vectors V1 to V4 that connect corresponding vertices. The vector V1 is a vector whose starting point is the vertex Q1 and whose ending point is the vertex B1. The same applies to the vectors V2 to V4. The update unit 422 identifies the following for the surfaces included in the user space 104 and the partial space 106: Z1 = the sum of deviations from the average angle of the vector Calculate. Specifically, first, the average angle of the four vectors V1 to V4, for example the average value of the angle difference from the vertical axis, is calculated for the surfaces 146a and 146b. Next, the update unit 422 calculates the deviation (absolute value) between the vector V1 and the average angle. The same is true for the other vectors, and the update unit 422 calculates the Z1 value as the total deviation from the average angle of the four vectors. If the surfaces 146a and 146b are the same size, the four vectors will face in the same direction, and the Z1 value will be close to zero. On the other hand, if the sizes of the surfaces 146a and 146b are significantly different, the Z1 value will be large.

[0078] The update unit 422 calculates the Z1 value for all faces, Z2 = (sum of Z1 values) / number of faces The closer the surfaces included in the user space 104 (A01) and the partial space 106 (A01: normal) are to the same size, the smaller the Z2 value becomes. In other words, the Z2 value is an index value indicating the similarity between the user space 104 (A01) and the partial space 106 (A01: normal).

[0079] Next, the update unit 422 Z3 = Z2 × interval coefficient k Here, the interval coefficient k means the time difference between the acquisition date and time of the partial space 106 (A01: regular) and the acquisition date and time of the user space 104 (A01). Specifically, k=1.0 if within 1 hour, k=1.2 if within 1 day, k=1.5 if within 1 month, k=2.0 if within 1 year, k=3.0 if within 5 years, and k=10.0 if 5 years or more. In other words, the older the partial space 106 (A01: regular) is, the larger the Z3 value is likely to be.

[0080] Furthermore, the update unit 422 Z4 = Z3 × correction coefficient m The correction coefficient m is a coefficient defined as (1 + image quality difference) when the image quality of the user space 104 (A01) is better than that of the partial space 106 (A01: normal). For example, when the image quality of the user space 104 (A01) is "A" and the image quality of the partial space 106 (A01: normal) is "B", the image quality difference is "1", so the correction coefficient m = 1 + 1 = 2. When the image quality difference is the same, or when the image quality of the partial space 106 (A01: normal) is higher than that of the user space 104 (A01), the correction coefficient m = 1. In other words, the higher the image quality of the user space 104 compared to the normal space, the larger the Z4 value is likely to be.

[0081] In summary, the Z4 value is large when: - The difference between the subspace 106 (A01: normal) and the user space 104 (A01) is large (the reality may have changed significantly) The subspace 106 (A01: regular) is older than the user space 104 (A01). The image quality of the user space 104 (A01) is better than that of the subspace 106 (A01: regular).

[0082] When the Z4 value is greater than a predetermined threshold D1, the update unit 422 determines that the update condition is satisfied. When the update condition is satisfied, the update unit 422 registers the user space 104 (A01) as a new normal space in the integrated space 100, and changes the setting of the partial space 106 (A01), which was a normal space, to a reserve space.

[0083] FIG. 15 is a flow chart showing the process of expanding and updating the integrated space 100. The user space generation unit 416 generates the user space 104 (S50) through the process described in relation to Fig. 13. When the subspace 106 for the same point as the user space 104 does not exist, that is, when the user space 104 for an unknown point is obtained (Y in S52), the expansion unit 420 expands the integrated space 100 by adding the user space 104 as a new subspace 106 (normal) (S54).

[0084] If a partial space 106 already exists at a point corresponding to the user space 104 (N in S52), the update unit 422 determines whether the update condition is met (S64). If the update condition is met (Y in S64), the update unit 422 registers the user space 104 as a partial space 106 (regular) and updates the integrated space 100 by changing the partial space 106 that was registered as a regular space to a reserve space (S66). If the update condition is not met (N in S64), the update unit 422 adds the user space 104 to the integrated space 100 as a reserve space (S68).

[0085] When the subspace 106 is updated or added (S54, S66), the image analysis unit 414 performs image analysis of the new subspace 106 (normal) and identifies the object 130 (S56). The image analysis unit 414 further generates guidance information 144 regarding the object 130 from image data of the object 130 included in the subspace 106 (S58). The image analysis unit 414 registers the object 130 and the guidance information 144 in the object information 120 (S60).

[0086] The reward giving unit 424 gives a reward to the user who provided the user image 102 that is the basis of the subspace 106 (regular) (S62). The reward here may be anything, information, or rights that bring benefits to the user, and may be, for example, digital content such as images, music, and games, cryptocurrency, points that can be exchanged for various services or products, cryptocurrency, and the like.

[0087] [Summary] The position estimation system 500 has been described above based on the embodiment. According to this embodiment, the integrated space 100 can be appropriately expanded and updated by collecting user images 102 from various users around the world and generating a user space 104 from the collected user images 102. Since the integrated space 100 is a three-dimensional map that continues to expand while undergoing metabolism, the position estimation function of the integrated space 100 is naturally strengthened over time.

[0088] The position estimation system 500 identifies the current location of the user by comparing the user image 102 with the spatial image 108. GPS (Global Positioning System), which is often used as a type of position estimation technology, has difficulty in estimating the location indoors or underground because it requires communication with artificial satellites, and requires supplementation with other sensors such as a gyro sensor. On the other hand, the position estimation system 500 generates an integrated space 100 similar to the real space, and identifies the current location of the user by comparing the user image 102 in the real space with the spatial image 108 in the integrated space 100 (a virtual space similar to reality), so it can be used indoors as well. In addition, the estimation accuracy of GPS is about ±30 meters, while that of SLAM is about ±0.5 meters, and the position estimation accuracy of SLAM is far superior to that of GPS. Therefore, position estimation system 500 can estimate the position with higher accuracy not only in the vertical direction but also in the horizontal direction compared to GPS.

[0089] Since the integrated space 100 is a three-dimensional space, three-dimensional route guidance is possible. For example, if the office "YX" on the third floor of a building is set as the destination, detailed three-dimensional guidance is possible from the building entrance to the elevator location and the route after getting off the elevator. In addition, one advantage of a three-dimensional map based on captured images is that detailed information such as steps, obstacles, doors, and fences can be incorporated into the integrated space 100.

[0090] Even if a new user space 104 is acquired, if the update condition is not met, this user space 104 will not be registered as a regular space. The space management server 400 registers only user spaces 104 that have information value as regular spaces in the integrated space 100, and registers user spaces 104 that have low information value as spare spaces. By making such a distinction, the calculation load associated with updating the integrated space 100 can be reasonably suppressed. For example, when there is almost no difference between the regular space and the latest user space 104, there is little need to incorporate the user space 104 as a new regular space. In addition, when accuracy and image quality are also taken into consideration, the latest user space 104 is not necessarily appropriate as a regular space.

[0091] By incorporating the interval coefficient k into the decision element of the update condition, the old partial space 106 is unlikely to be maintained as a regular space for a long period of time. For example, when the user space 104 (A15) for the same point is acquired in the partial space 106 (A15) after a long time, the newly generated user space 104 (A15) is likely to be adopted as the regular space. The interval coefficient k promotes the renewal (update) of the integrated space 100.

[0092] When the user space 104 is added as a regular space, the reward granting unit 424 grants a reward to the user. The user can obtain a reward by taking an image of an unknown point and contributing to the expansion of the integrated space 100. Alternatively, even if the point is known, the user can obtain a reward if the user can generate a user space 104 of higher quality than the regular space. By granting a reward, it becomes easier to stimulate a desire to cooperate in improving the quality of the integrated space 100. In addition, it becomes easier to collect high-quality user images 102 necessary to generate a suitable user space 104. For example, a user who owns a high-resolution camera or a user who slowly and thoroughly captures the surrounding environment is more likely to obtain a reward.

[0093] Guidance information 144 can be attached to objects 130 included in the integrated space 100. The image analysis unit 414 identifies what the object 130 is by, for example, analyzing characters or images displayed on a sign or the like of the object 130. The information identified in this manner may be the guidance information 144, or the guidance information 144 may be generated by searching for related information based on the identified information. Since the user can view the guidance information 144 for various objects 130 displayed on the guidance screen 140, the user can grasp in real time what he or she is currently looking at.

[0094] The integrated space 100 includes a reserve space as well as a regular space. By storing various reserve spaces at the same point, real spaces at various times can be reproduced. By storing reserve spaces, the integrated space 100 not only expands spatially, but also becomes multi-layered temporally.

[0095] The present invention is not limited to the above-mentioned embodiment and modified examples, and the components can be modified without departing from the gist of the present invention. Various inventions can be formed by appropriately combining multiple components disclosed in the above-mentioned embodiment and modified examples. In addition, some components can be deleted from all the components shown in the above-mentioned embodiment and modified examples.

[0096] [Variations] In this embodiment, it has been described that the integrated space 100 is generated, updated, managed, etc. by one space management server 400, but the management of the integrated space 100 may be shared among a plurality of space management servers 400. For example, a space management server 400 may be provided for each region, such as a space management server 400 in charge of the integrated space 100 in Hiroshima prefecture and a space management server 400 in charge of the integrated space 100 in Gifu prefecture.

[0097] In this embodiment, the image measurement by VisualSLAM and SfM is assumed. As a modified example, the user space 104 may be generated by measuring the surrounding environment by LiDAR (Light Detection And Ranging). Alternatively, the user space 104 may be generated by using LiDAR and a camera in combination. Image measurement based on captured images is not good at measuring in dark places, but by using LiDAR in combination, three-dimensional measurement becomes possible even in such places. Three-dimensional measurement by an infrared camera may be used in combination.

[0098] In the present embodiment, the partial spaces 106 are described as not overlapping with each other, but the partial spaces 106 may overlap. For example, a part of the partial space 106 (A01) centered on the point A01 may overlap with a part of the partial space 106 (A02) centered on the adjacent point A02. By overlapping parts of the partial spaces 106, it becomes easier to form an integrated space 100 without gaps.

[0099] In the present embodiment, the position estimation is performed by comparing the user image 102 and the spatial image 108. As a modified example, the position estimation may be assisted by utilizing existing sensors such as a GPS and an IMU (Inertial Measurement Unit). For example, when the position information "near point A01" is obtained by the GPS, the position identification unit 408 may search for a spatial image 108 similar to the user image 102 within a predetermined range from point A01 in the integrated space 100, and identify the current position of the user. By narrowing the search range to the vicinity of a point where the user is likely to be present in the vast integrated space 100, the position estimation can be speeded up.

[0100] When the user terminal 200 is equipped with a compass and an altimeter, the user space generation unit 416 may estimate the imaging direction of the user image 102 based on information obtained from the compass, etc. When the user terminal 200 is equipped with a compass, etc., the imaging direction of the user image 102 can be included as part of the image attribute information, so that an appropriate user space 104 can be generated without performing the orientation correction described in relation to FIG.

[0101] A mark such as a QR code (registered trademark) may be set in advance in the real space. For example, a mark X1 is attached to a building A. The mark X1 includes a mark ID for identifying the mark. If the mark X1 is shown in the user image 102, the data processing unit 208 of the user terminal 200 extracts the mark ID from the user image 102 from the mark X1. The communication unit 204 transmits the user image 102 and the mark ID to the space management server 400. In the data storage unit 406, the mark ID and the position information are associated in advance. The position identification unit 408 can identify the current location of the user by the position information corresponding to the mark ID. The position identification unit 408 can also estimate the distance from the mark X1 to the user and the relative position of the user with respect to the mark X1 based on the size and shape of the mark X1 shown in the user image 102. By using mark X1 as a known point, the user's current location can be estimated, so the position identification unit 408 simply searches for a spatial image 108 that is similar to the user image 102 within a specified range from the point where mark X1 is located in the integrated space 100.

[0102] Once the user's current location can be identified, even if the user moves thereafter, the location identification unit 408 can easily grasp where the user is located. For example, if the user's current location can be identified as point A03, the location identification unit 408 can easily estimate the location by searching the space image 108 from the integrated space 100 near point A03 even if the user moves. When the user starts up the user terminal 200 and starts position estimation, the position specifying unit 408 may start position estimation centered on the point that was last estimated as the user's current position.

[0103] The position may be estimated using a landmark structure as a known point. For example, since the position of Tokyo Tower is known, the user terminal 200 or the space management server 400 can estimate that the user is at least in the vicinity of Tokyo Tower when Tokyo Tower is shown in the user image 102. By using landmarks in this way, the position identification unit 408 can be executed more efficiently.

[0104] The location specifying unit 408 may specify, as the current location of the user, the current location estimated based on a large number of user images 102 among the plurality of user images 102. For example, assuming that 50 user images 102 are obtained, among these user images 102, 40 are similar to the spatial image 108 in the partial space 106 (A05), and the others are similar to the spatial images 108 in the partial spaces 106 at other locations. At this time, the location specifying unit 408 estimates that the user's current location is the A05 location. In this way, by estimating the most probable location as the current location even from a large number of user images 102, the location estimation accuracy can be improved. The more user images 102 are used, the higher the location estimation accuracy becomes.

[0105] In the present embodiment, it has been described that the update condition is satisfied when the Z4 value is greater than the threshold D1. As a modification, two thresholds D1 and D2 may be set. When the Z4 value is greater than the threshold D1, the update condition is satisfied, and when it is less than or equal to the threshold D2, the update condition is not satisfied.

[0106] When D2 < Z4 ≤ D1 (hereinafter, this range is referred to as the "intermediate range"), the update unit 422 determines the update condition by comparing the preliminary space and the user space 104. For example, assume that the Z4 values for the user space 104 (A08) and the partial space 106 (A08: normal) are in the intermediate range. At this time, the update unit 422 calculates the Z4 value not only for the partial space 106 (A08: normal) but also for the partial space 106 (A08: preliminary). If a predetermined ratio or more, for example, the majority, of the Z4 values for each of the plurality of preliminary spaces is less than or equal to the threshold D2, the update unit 422 determines that the update condition is satisfied.

[0107] More specifically, it is assumed that the Z4 value (spare 1) for the partial space 106 (A08: spare 1) and the user space 104 (A08), the Z4 value (spare 2) for the partial space 106 (A08: spare 2) and the user space 104 (A08), the Z4 value (spare 5) for the partial space 106 (A08: spare 5) and the user space 104 (A08) are calculated. If three or more (majority) of the Z4 values ​​(spare 1) to (spare 5) are equal to or less than the threshold D2, the update condition is met, and the update unit 422 registers the user space 104 (A08) in the integrated space 100 as a normal space.

[0108] If the Z4 value for the normal space is in the intermediate range and the user space 104 (A08) is similar to many spare spaces, it is likely that the subspace 106 (A08: normal) was abnormal. In this case, it is appropriate to set the user space 104 (A08) as a new normal space. This mechanism can eliminate the risk of an inappropriate normal space remaining forever.

[0109] In the present embodiment, the Z1 value is calculated based on the deviation in the direction of vectors V1 to V4, and the Z2, Z3, and Z4 values ​​are calculated from the Z1 value, and the success or failure of the update condition is determined based on the Z4 value. As a modified example, the C3 value described below may be calculated based on the length of the vector instead of the direction, and the update condition may be determined based on the C3 value. Specifically, the update unit 422 calculates the following for all vectors included in the user space 104 and the partial space 106: C1 = total vector length / total number of vectors When the similarity between the user space 104 and the subspace 106 is high, the C1 value is small (see FIG. 14).

[0110] Next, the update unit 422 C2 = C1 value × interval coefficient k C3 = Z2 × correction coefficient m The update unit 422 may compare the C3 value calculated in this manner with the thresholds D1 and D2 to determine whether the update condition is met.

[0111] The update unit 422 may determine whether the update condition is met based on both the Z4 value and the C3 value. For example, the update unit 422 may determine whether the update condition is met by comparing the Z4 value+C3 value with predetermined thresholds D1 and D2.

[0112] The integrated space management unit 418 may reproduce the integrated space 100 corresponding to the "past world" based on the preliminary space. For example, when the year 2050 arrives, if the integrated space 100 is constructed only from the partial space 106 with the "acquisition date and time 2024", the "world of 2024 (past)" can be reproduced. By utilizing the integrated space 100 as a so-called metaverse, the user can experience the "past world".

[0113] In order to reduce the amount of data in the integrated space 100, the integrated space management unit 418 may delete spare spaces that satisfy a predetermined deletion condition from the integrated space 100. Possible deletion conditions include, for example, "image quality is below a predetermined level" and "generated before a predetermined date and time."

[0114] A condition other than a destination point may be set as a route condition. For example, assume that the user sets "the nearest toilet" as a destination point. The route generating unit 412 searches for an object 130 corresponding to "toilet" in the integrated space 100, and generates route information from the user's current location to the nearest "toilet."

[0115] The user may set a route condition such as "the nearest toilet that can be reached without steps." Since the integrated space 100 is a three-dimensional space, it includes three-dimensional information such as steps (height differences). The route generation unit 412 may check whether there are steps on the route from the user's current location to the toilet, and then guide the user to a route to the toilet that can be reached without steps. This type of control makes it possible to provide route guidance that is useful for wheelchair users, for example.

[0116] In addition, the route generation unit 412 may generate route information according to various route conditions, such as "a route without steps (for traveling by electric cart)", "a route for a walk with a large difference in elevation and taking about 15 minutes", and "a route that avoids passing outdoors as much as possible".

[0117] The user space generation unit 416 may generate a user space 104 that takes privacy into consideration by "blurring" the face of a person reflected in the user image 102. The user space generation unit 416 may set in advance ranges (ranges of latitude, longitude, and altitude) of areas where image capture is prohibited (for example, inside a toilet, a military base, a changing room, a swimming pool, etc.), and may not generate a user space 104 from a user image 102 that targets the areas where image capture is prohibited.

[0118] Similarly, user space generation unit 416 may set entry-prohibited areas (for example, military bases, research laboratories, factories, etc.). When estimating the position, guidance unit 410 may notify user terminal 200 of warning information when the user enters an entry-prohibited area. Benefit granting unit 424 may set a penalty such that a privilege is not granted for a certain period of time to a user who violates a preset prohibited act such as entering a prohibited area or taking a picture in a prohibited area.

[0119] The integrated space management unit 418 may delete "non-fixed objects" included in the real space, such as automobiles and humans, that is, moving objects, from the partial space 106. For example, suppose that object X appears only in two auxiliary spaces and one normal space in the partial space 106 (A10: normal), partial space 106 (A10: auxiliary 1), and partial space 106 (A10: auxiliary 10). In this case, object X exists only at the acquisition date and time of these two auxiliary spaces and one normal space, and is therefore considered to be a moving object. In this case, the integrated space management unit 418 deletes object X from the two auxiliary spaces.

[0120] More specifically, the integrated space management unit 418 deletes an object that appears only in a predetermined percentage or less of the partial spaces 106 among the multiple partial spaces 106 that target the same point. For example, the integrated space management unit 418 may delete an object that appears only in 20% or less, i.e., two or less, of the ten partial spaces 106. By deleting a moving object that interferes with position estimation from the partial spaces 106, the accuracy of position estimation can be improved. [Explanation of symbols]

[0121] 100 integrated space, 102 user image, 104 user space, 106 partial space, 108 space image, 110 space configuration information, 120 object information, 130 object, 140 guidance screen, 142 route guidance, 144 guidance information, 146a surface, 146b surface, 200 user terminal, 202 imaging unit, 204 communication unit, 206 user interface processing unit, 208 data processing unit, 210 data storage unit, 300 processor, 302 first bus, 304 memory, 308 bridge, 310 second bus, 312 storage, 314 input device, 316 output device, 318 peripheral device, 400 space management server, 402 data processing unit, 404 communication unit, 406 data storage unit, 408 position identification unit, 410 guidance unit, 412 route generation unit, 414 Image analysis unit, 416 user space generation unit, 418 integrated space management unit, 420 expansion unit, 422 update unit, 424 privilege grant unit, 426 integrated space storage unit, 500 position estimation system

Claims

1. an integrated space storage unit for storing data of an integrated space configured as a three-dimensional virtual space corresponding to a real space; a position identification unit that identifies position information indicating a three-dimensional position in real space of a point corresponding to a user image by comparing a user image, which is a captured image acquired from a user terminal, with a plurality of spatial images that constitute the integrated space; a guidance unit that provides the specified location information to the user terminal; a user space generation unit that generates data of a user space as a three-dimensional virtual space from a plurality of user images acquired by the user terminal at a first point in real space; A position estimation system comprising: an integrated space management unit that updates data of the integrated space by updating a partial space including the first point, which is part of the integrated space, with a user space corresponding to the first point.

2. the user space generation unit generates, when a user moves in a predetermined direction from a second point and reaches a third point, a user space including the third point from a plurality of user images acquired by the user terminal at the third point; 2. The position estimation system of claim 1, wherein when the integrated space includes a subspace targeting the second location and does not include a subspace targeting the third location, the integrated space management unit expands the integrated space by newly registering a user space corresponding to the third location as a subspace adjacent to the second location in the specified direction in the integrated space.

3. The location estimation system of claim 2 , further comprising a reward granting unit that grants a reward to a user when a portion of the integrated space is updated by a user space or when a user space is added as part of the integrated space.

4. The position identification unit further identifies an imaging direction of the user image by comparing the user image with a plurality of space images that configure the integrated space; The position estimation system according to claim 1 , wherein the user space generation unit stores the user image and the imaging direction in association with each other, and generates data of the user space from a plurality of user images corresponding to a plurality of imaging directions.

5. Guidance information is associated in advance with objects included in the integrated space; The position estimation system according to claim 1 , wherein the guidance unit, when providing the position information to the user terminal, provides guidance information related to a location where the user terminal is located.

6. An image analysis unit that performs image analysis of an object included in the user image to generate guidance information for the object, The position estimation system according to claim 5 , wherein the integrated space management unit maps the user image to an object included in the integrated space and registers the guidance information in association with the object.

7. The position estimation system of claim 1, wherein the integrated space management unit constructs the integrated space by combining multiple user spaces, and when multiple user spaces are obtained targeting the same location, sets one of the multiple user spaces as a regular subspace that is part of the integrated space, and stores the other user spaces as spare subspaces.

8. The integrated space management unit When a predetermined update condition is satisfied, the regular partial space corresponding to the first location is changed to a reserve partial space, and the user space corresponding to the first location is set as a regular partial space of the integrated space; The position estimation system according to claim 7 , wherein when the update condition is not satisfied, a regular subspace corresponding to the first location is maintained, and a user space corresponding to the first location is stored as a reserve subspace.

9. The position estimation system of claim 8, wherein the integrated space management unit calculates a degree of agreement between a verification surface existing in a user space of the first point and a comparison surface that is included in a normal subspace corresponding to the first point and exists within a predetermined range from the verification surface, and determines whether the update condition is met based on the degree of agreement between each of the multiple verification surfaces and the multiple comparison surfaces.

10. The integrated space management unit If the degree of match is less than a first threshold, it is determined that the update condition is satisfied; if the degree of agreement is less than a second threshold value that is greater than the first threshold value, determining whether an update condition is satisfied based on the degree of agreement between a plurality of surfaces included in each of one or more auxiliary partial spaces corresponding to the first point and a plurality of surfaces included in the user space; The position estimation system according to claim 9 , wherein if the degree of coincidence is equal to or greater than the second threshold, it is determined that the update condition is not satisfied.

11. The user space and the subspace are registered with the corresponding data acquisition date and time. The position estimation system according to claim 9 , wherein the integrated space management unit further determines whether the update condition is satisfied based on a difference between an acquisition date and time of the user space and an acquisition date and time of the partial space.

12. The position estimation system according to claim 1 , wherein the guidance unit, when a destination point and route conditions are set by a user, provides guidance on a route from a current position of the user to the destination point in accordance with the route conditions.

13. an integrated space storage unit for storing data of an integrated space configured as a three-dimensional virtual space corresponding to a real space; a position identification unit that identifies position information indicating a three-dimensional position in real space of a point corresponding to a user image by comparing a user image, which is a captured image acquired from a user terminal, with a plurality of spatial images that constitute the integrated space; a guidance unit that provides the specified location information to the user terminal; A position estimation system in which, when a known point in real space is included in a user image, the position identification unit searches for a point corresponding to the user image preferentially from data of a subspace of the integrated space that includes the known point.

Images