Position specification system and recording medium

Abstract

Provided is a system in which a control unit 22 of a server 20 that has acquired an image captured by a terminal device 10 specifies the position of the terminal device. The server comprises a database 23 which stores a large number of 3D maps indicating respective areas and feature data related to the 3D maps. The feature data includes data obtained by vectorizing an image of a 3D map and text information related thereto, and context information of a 3D map, etc. A map narrowing unit 222 of the control unit vectorizes the acquired image, etc., performs vector retrieval with the feature data stored in the database, and provides the acquired image and the context information of the 3D map to an LLM to narrow down candidates of the 3D map (exclude 3D maps of similar shapes). A map matching unit 221 can accurately specify a position and a posture by matching the shape with the narrowed 3D map candidates.

Description

Location identification system and recording medium

[0001] The present invention relates to a location identification system and a recording medium.

[0002] Measuring the position and orientation of an imaging device based on image information is used for various purposes, such as aligning a real space with a virtual object in mixed reality / augmented reality, and estimating the position of a user carrying the imaging device, a robot, a car, a drone, or other mobility. Position detection using a VPS (Viral Positioning System) can identify a location even in an environment where GPS satellite signals cannot be received. Therefore, for example, if a user carrying an imaging device is inside a building or underground, the user's location within that space can be identified.

[0003] This type of system for identifying a position using captured image information is disclosed in, for example, Japanese Patent Application Laid-Open No. 2003-222999.

[0004] Patent No. 7361075

[0005] Sales floors in supermarkets and convenience stores tend to have similar display shelves arranged in a similar layout. Similarly, indoor stores in commercial complexes tend to use rooms of the same size along the aisles, and some have similar entrances. Exhibitions held at exhibition centers often have numerous booths with the same dimensions and layouts.

[0006] In such cases, for example, a supermarket may have many similarly shaped sales areas, and the feature points of images taken at each sales area may be similar. When similar feature points exist, the accuracy of location identification using VPS decreases. This also occurs indoors in commercial complexes, exhibition halls, etc.

[0007] Furthermore, when performing image-based localization within a wide indoor area, there are many images to compare. If there are many images with similar features, the system may not be able to accurately locate the location or may freeze up without being able to identify a single image.

[0008] The above-mentioned problems are described as being independent of each other, and the present invention does not necessarily have to solve all of the problems described, but it is sufficient if it can solve at least one of the problems. Furthermore, we intend to obtain rights to the configurations for solving these problems separately through divisional applications, amendments, etc.

[0009] (1) A location identification system that identifies the location where an acquired image was taken should preferably be equipped with a map narrowing-down function that narrows down candidates for 3D maps to be searched using feature data abstracted based on the image, and a map matching function that matches the shape of the image with the narrowed-down candidates for 3D maps to identify the location where the image was taken.

[0010] In this way, it is possible to perform shape feature point matching with high accuracy after narrowing down the area in terms of content (narrowing down the 3D maps). The process of narrowing down the 3D map candidates includes, for example, selecting a predetermined number of 3D maps for shape matching from multiple 3D maps, selecting a predetermined number of 3D map portions of an area for shape matching within one 3D map, or a combination thereof.

[0011] (2) The map narrowing function may be configured to perform the abstraction using a large-scale language model with the image and context information of the 3D map as input, thereby narrowing down the candidates for the 3D map.

[0012] While searches using large-scale language models cannot pinpoint location or pinpoint posture or orientation, they can identify similar shapes by taking their content into account. This allows for accurate selection of 3D map candidates. By narrowing down the 3D map candidates, the map matching function in the subsequent stage can accurately identify locations using shape matching.

[0013] (3) The map narrowing function may be configured to vectorize information based on the image to create the feature data.

[0014] (4) The map refinement function may be configured to create the feature data by vectorizing text information about the image. The text information about the image may be created by character recognition of characters contained in the image, by image recognition to indicate objects present in the image, or by a description of the image. The description may be created in advance by a person or generated by providing the image to a generation AI.

[0015] (5) The map refinement function may be configured to perform the abstraction using the image and context information related to the image.

[0016] (6) The map narrowing function may be configured to be performed using location information detected by the terminal that captured the image.

[0017] (7) It is preferable to configure the system to include a database that stores feature data to be compared and 3D maps and that can be accessed by the map narrowing function and the map matching function.

[0018] (8) The system may include a camera and a terminal having a communication function, and the terminal may be configured to transmit images captured by the camera using the communication function to a server having the map filtering function. The terminal may also have a function to acquire a location identified by the map matching function and notify the server of the acquired location.

[0019] (9) The recording medium of the present invention may be a computer-readable recording medium on which a program for causing a computer to realize the functions of the location identification system described in any one of (1) to (8) is recorded.

[0020] The constituent elements of each invention described above in (1) to (8) can be arbitrarily combined in two or more, and the invention can be constituted by such combinations.

[0021] The present invention narrows down the candidate 3D maps to be searched using abstracted feature data, such as vector search or inference using a large-scale language model, and then matches feature points with the narrowed-down 3D maps to accurately identify a location even within a space where similar shapes exist.

[0022] FIG. 1 is a diagram showing a preferred embodiment of a position specifying system according to the present invention. FIG. 2 is a diagram showing an example of an actual system configuration. FIG. 3 is a flowchart illustrating processing by a control unit of a server. FIG. 4 is a diagram showing an example of a place where an indoor position is specified. FIG. 5 is a diagram showing an example of a place where an indoor position is specified. FIG. 6 is a diagram showing an example of a place where an indoor position is specified. FIG. 7 is a diagram showing an example of a place where an indoor position is specified. FIG. 8 is a diagram showing an example of a place where an indoor position is specified. FIG. 9 is a diagram showing an example of a place where an indoor position is specified.

[0023] Preferred embodiments of the present invention will be described in detail below with reference to the drawings. However, the present invention is not limited to these embodiments, and various changes, modifications, and improvements can be made based on the knowledge of those skilled in the art without departing from the scope of the present invention.

[0024] The positioning system according to the present invention accurately identifies positions indoors where GPS signals cannot be received accurately, such as in exhibition halls, commercial facilities, various buildings, underground shopping malls, and underground parking lots. Based on information acquired in advance using a camera or LiDAR (Light Detection and Ranging) for the indoor area where positioning is to be performed, various computational processes, statistical models, and machine learning models are used to create and store a 3D map showing a specific indoor area, along with feature data related to the 3D map. The positioning system then acquires image data actually captured on-site, narrows down indoor locations based on the acquired image data and the stored feature data, and accurately identifies the location by matching feature points with the stored 3D map.

[0025] Fig. 1 is a block diagram showing a preferred embodiment of a location identification system 1 according to the present invention, and Fig. 2 is a diagram showing an example of an actual configuration. As shown in Fig. 1, the location identification system 1 has a function in which a server 20 acquires image data captured by a terminal device 10 via a network 2 and identifies the location of the terminal device 10 based on the acquired image data. The network 2 may be, for example, a public line or the Internet, and may be accessible even when the devices are located far apart from each other.

[0026] The terminal device 10 includes a control unit 11, a camera 12, an input unit 13, a communication unit 14, a display unit 15, a GPS receiving unit 16, etc. When the terminal device 10 is carried by a person, for example, a smartphone, smart glasses, etc. When the terminal device 10 is a smartphone, the hardware constituting the terminal device 10 (the control unit 11, the camera 12, the input unit 13, the communication unit 14, the display unit 15, and the GPS receiving unit 16) uses equipment that is standard equipment on a smartphone.

[0027] The control unit 11 controls each unit of the terminal device 10. The control unit 11 is, for example, a computer including a processor and a memory. The memory is, for example, a main storage device having a RAM (Random Access Memory) and a ROM (Read Only Memory). The processor temporarily stores a program read from the ROM in the RAM. The RAM provides a working area for the processor. The processor performs various controls by temporarily storing data generated during program execution in the RAM while performing arithmetic processing. If the terminal device 10 is a smartphone, the process for determining the location may be performed by the control unit activating a smartphone app, for example.

[0028] The communication unit 14 is connected to the network 2 and has a function of transmitting and receiving data and information to and from other devices connected to the network 2, a function of connecting to a Wi-Fi access point, and the like.

[0029] The control unit 11 displays images captured by the camera 12 on the display unit 15. The control unit 11 also transmits image data being captured to the server 20 via the network 2. The control unit 11 also transmits sensor information based on the current location to the server 20 via the network 2. The sensor information includes, for example, location information (longitude and latitude) received and acquired by the GPS receiving unit 16, and information identifying a Wi-Fi access point acquired by the communication unit 14. The timing for transmitting this information may be periodically at predetermined intervals or triggered by a transmission instruction received by the input unit 13. The control unit 11 also has a function for displaying predetermined information on the display unit 15 based on the current location acquired from the server 20.

[0030] The server 20 includes a communication unit 21 having an interface for connecting to and communicating with a network 2 such as an internet connection function, a control unit 22 that controls the operation of the server 20, and a database 23 that stores data for location identification. While Fig. 1 illustrates an example in which the server 20 is configured from a single server computer, it may also be configured from multiple computers connected directly or via a network, as shown in Fig. 2. The server that implements the database is not limited to a physical server, and various types of servers may be used, such as those configured from software programs or implemented in the cloud.

[0031] The database 23 stores data for identifying a location based on images captured by the camera 12 of the terminal device 10. The database 23 includes a 3D map storage unit 231 and a feature data storage unit 232. The 3D map storage unit 231 stores a 3D map of a target area for which location identification is to be performed, which has been created in advance. That is, the 3D map is data based on a three-dimensional shape composed of a collection of feature points (x, y, z) or a polygon mesh created using a large number of images and LiDAR depth data associated with each image. Therefore, a system administrator or the like moves around the indoor area for which location identification is to be performed in advance, taking images using a camera to acquire image data of each location, and acquiring depth data using LiDAR (Light Detection and Ranging) as needed. Various calculations and other processes are performed based on multiple image data and depth data acquired at a certain indoor location to create a 3D map. This 3D map creation process can be performed using existing technology.

[0032] Within the same indoor space, there may be multiple areas with similar shapes. As a result, there may be similar shapes and feature points in 3D maps of different locations. In particular, in the case of an exhibition hall, there are many booths with similar dimensions and shapes, and the layout of multiple booths within the venue is often the same. Furthermore, in large commercial facilities and shopping malls, there may be multiple stores with similar entrances. Furthermore, even within the same store, if many identical display shelves are prepared and arranged in the same layout, there will be many similar 3D maps.

[0033] The process of creating a 3D map for a given location involves, for example, dividing the area so that the characteristics of the 3D map data are uniquely determined. A 3D map of the divided area can be created by performing three-dimensional reconstruction using a sequence of slightly overlapping images acquired in the divided area and the posture information associated with each image. Furthermore, if seismic intensity data acquired using LiDAR is available, this data can also be used to create a 3D map of the divided area. If LiDAR data is not available, the 3D map can be created based solely on image data.

[0034] For example, in the case of an exhibition hall, the divided area may be a fixed area such as a corner of an event venue or a booth. In the case of a shopping mall, the divided area may be, for example, an area including the entrance of each store. Furthermore, the divided area within a store may be, for example, an area including one or more display shelves. The divided area may be selected as appropriate, and in either case, one 3D map is created for each divided area. Each 3D map is assigned an ID, and location information identifying the actual indoor area is also associated and registered.

[0035] In this way, 3D maps are created for each of the areas obtained by dividing a given indoor area into multiple sections, so that, for example, multiple 3D maps are prepared for the same exhibition or event venue held at a single exhibition center.Even within a single commercial facility, 3D maps are created for each of the multiple divided areas.

[0036] The server 20 stores 3D maps for multiple areas within different buildings and facilities. Therefore, the server 20 stores the 3D maps in a manner that indicates which exhibition hall or commercial facility the 3D map pertains to. For example, the 3D maps may be stored in separate storage areas for each building, or identification information for identifying the building or the like may be registered as supplementary information to the 3D map. Furthermore, since the 3D maps will differ depending on the exhibition being held, even if the exhibition hall is the same, the 3D maps are stored separately for each exhibition in the 3D map storage unit 231.

[0037] The feature data storage unit 232 stores feature data abstracted into an intermediate representation from one or more image data captured of the area for which the 3D map was created. Each 3D map has one or more pieces of this feature data. The abstracted feature data may include, for example, data vectorized using a multimodal embedding model from the images used to create the 3D map, explanatory information (context) present in the image of each divided area, and associated location information for identifying the approximate location. The context may be text data, but it is preferable to use a vectorized multidimensional vector. The vector representing the feature data may be created, for example, by detecting characters or objects from the image to generate explanatory text, and then vectorizing the text information using an embedding model. Furthermore, the vector representing the feature data is not limited to text information; for example, it may be generated directly by convolving the image. Generating a vector based on an image allows for the generation of feature data based on information that cannot be generated by character detection, such as the appearance of walls, which reduces information loss. Additionally, information combining explanatory information from multiple images taken from different angles within the same area may be used as context information for a single 3D map. The text used to create this explanatory information may be generated, for example, by image recognition of image data and using OCR technology. It is also preferable to generate explanatory text using a multimodal large-scale language model, using images as input. This creates text that describes the image information as if it were a human, and can also include non-textual visual information such as shape and color, generating feature data that can be more accurately distinguished from other locations. Furthermore, it is also preferable for a worker to review the created explanatory information (explanatory text), correct the text, re-take photos, and create new explanatory information.

[0038] The accompanying location information includes, for example, Wi-Fi access points, GPS information, etc. For example, since there are areas where GPS signals can be received even indoors, the approximate location can be identified from GPS-based location information. Furthermore, Wi-Fi access points are installed at appropriate locations in underground shopping malls, etc. Wi-Fi access points have narrow communication areas. Therefore, information identifying the access point is used as feature data to associate the access point with a 3D map of the communication area.

[0039] The creation of the 3D map and the feature data to be stored in the database 23 may be performed in any order, as long as they are based on the same range.

[0040] The control unit 22 is, for example, a computer including a processor and a memory. The memory is, for example, a main storage device having a RAM (Random Access Memory) and a ROM (Read Only Memory). The processor temporarily stores a program read from the ROM in the RAM. The RAM also provides a working area for the processor. The processor performs various controls by performing arithmetic processing while temporarily storing data generated during program execution in the RAM.

[0041] The control unit 22 includes a map matching unit 221 and a map narrowing unit 222 as functions for performing position identification. The map matching unit 221 performs shape-based matching. That is, the map matching unit 221 matches feature points extracted based on image data captured by the terminal device 10 with a group of 3D maps stored in the 3D map storage unit 231, identifies one 3D map with a high degree of match, and derives the position and orientation at the time of capture by the camera 12 of the terminal device 10. Furthermore, the map matching unit 221 may have a function for calculating coordinates on the 3D map of the terminal device 10 or coordinates within the entire group of 3D maps based on the derived capture position and orientation. Furthermore, the shape-based matching process may utilize an existing position identification algorithm based on, for example, comparing image data captured by a VPS or the like with a pre-prepared 3D map. The control unit 22 then sends position information regarding the identified position, orientation, etc. to the terminal device 10 via the network 2.

[0042] Prior to shape matching by the map matching unit 221, the map narrowing down unit 222 narrows down the area where the image was captured using the acquired image data, etc. In other words, the map narrowing down unit 222 narrows down the map by abstracting the content represented by the image data. The map narrowing down unit 222 extracts candidates for the 3D map based on the acquired image data, etc. and the feature data stored in the feature data storage unit 232. The detailed functions of this map narrowing down unit 222 will be described later.

[0043] FIG. 3 is a flowchart illustrating the processing of the control unit 22 when an image captured by the camera 12 of the terminal device 10 at the site is acquired by a user who actually uses the system. The user who uses the system launches a predetermined smartphone app installed on a smartphone, which is an example of the terminal device 10. The smartphone app accepts a predetermined input instruction, launches the camera 12, switches the camera to a continuous shooting mode, and displays the image being captured on the display unit 15. The smartphone app also sends the image being captured to the server 20 at a predetermined timing. This predetermined timing may be a predetermined time interval, or the acceptance of a transmission instruction given from the input unit 13, such as a tap on the screen. The control unit 11 also transmits the predetermined sensor information along with the image data.

[0044] The control unit 22 of the server 20 acquires the image captured by the terminal device 10 and the sensor information (S1). The control unit 22 stores the acquired image data in a predetermined storage unit. The storage unit may be, for example, a temporary storage unit such as a volatile memory.

[0045] The map narrowing-down unit 222 of the control unit 22 compares the acquired image with the context using a multimodal large-scale language model to determine the area, converts the image into feature information using a machine learning model, and then narrows down the 3D map candidates where the image whose photographed position is to be detected is located (S2). The map narrowing-down unit 222 sends the IDs of the narrowed-down 3D map candidates to the map matching unit 221.

[0046] Next, the map matching unit 221 reads the image data stored in the temporary storage unit and extracts feature points using a machine learning model or a SLAM technique (S3).The map matching unit 221 then matches the obtained feature points with the candidate 3D maps narrowed down by the map narrowing down unit 222, identifies the 3D map that best matches, and derives the position and orientation within that 3D map (S4).

[0047] The control unit 22 calculates the coordinates of the terminal on the 3D map or the coordinates of the entire group of 3D maps based on the derived shooting position and orientation (S5). The calculated coordinate data is then sent to the terminal device 10. The smartphone app provides, for example, route guidance based on the position specified by the received coordinate data.

[0048] The map narrowing unit 222 that performs the above-described process S2 has (1) an image recognition and vector search function, and (2) a search function that uses multimodal LLMs (Large Language Models).

[0049] The image recognition and vector search function (1) converts various information into feature vectors using some method and performs location identification based on the vector search, which is a similarity search. This vector search function vectorizes the acquired input image (sequence), calculates the degree of match with vectors associated with 3D maps stored in the feature data storage unit 232, and extracts feature data that meets a predetermined criterion. The control unit 22 determines the 3D maps associated with the extracted feature data as map candidates. The predetermined criterion may be, for example, a match exceeding a threshold or a predetermined number of highly matched maps. The vectorization process for the input image may involve vectorizing the image directly using a multimodal embedding model (generating one abstract feature data from multiple pieces of information), or detecting characters in the image and abstracting and vectorizing the obtained text information. The text information to be abstracted may be, for example, characters present in the image extracted by image recognition processing. Furthermore, the text information may be provided to the generation AI directly or through image recognition, and the description of the image generated by the generation AI may be used. This vector search can be processed at high speed, so when there are many 3D maps to compare, it is possible to efficiently determine in a short time whether a 3D map is a target.

[0050] The search function using LLM (2) uses image and text information as input and performs logical inference using a large-scale language model to identify a location. This search function using LLM provides information (such as descriptions) of candidate maps stored in the feature data storage unit 232 as context to the multimodal LLM, and inputs images captured by the terminal device 10 via the network 2 to estimate candidate 3D maps. In other words, the context information of the 3D map and image information captured by the terminal device 10 are provided to the LLM, which then searches for 3D map candidates. The LLM model itself may be fine-tuned. Searches using LLMs are characterized by higher accuracy but longer processing times compared to vector searches. Searches using LLMs are limited in the number of areas they can cover.

[0051] Furthermore, when a user actually using the system takes a photograph on-site with the camera 12 of the terminal device 10, the captured image data will differ if the orientation of the camera 12 is different, even if the camera 12 is in the same position when the feature data, etc., was created in advance. Even in such cases, for example, in the case of a multimodal large-scale language model, image information from multiple angles is combined and edited to create feature data that effectively describes the range that the 3D map corresponds to, making it particularly tolerant to differences in angle. Therefore, even if the orientation of the camera 12 is different, if the range is within a certain range, the map can be extracted as a 3D map candidate by narrowing down the map.

[0052] On the other hand, in the case of vector search, even if images are taken from the same position, if the orientation is different, the degree of match will be low, and if the difference in angle is large, detection will be impossible. Therefore, in the case of vector search, it is advisable to create feature vectors for images taken in multiple orientations within the range of a single 3D map and associate each vector with the ID of the same map. In other words, feature data is created in advance, and multiple feature vectors are associated with one 3D map when stored in the database 23. In this way, the 3D map linked to the image taken during use can be appropriately identified by matching the feature vectors.

[0053] Either of the functions (1) and (2) may be used, or both may be combined. When using one, it is preferable to use the more versatile (2). Furthermore, since (2) has limitations in processing speed and the number of areas that can be processed, it is also possible to narrow down the search results using only the image recognition and vector search (1). Furthermore, when combining these functions, for example, when there are a large number of 3D map groups, it is preferable to narrow down the search results using (1) and make the final decision using (2). Furthermore, instead of dividing the order of use in this way, it is preferable to operate the search functions (1) and (2) in parallel, calculate the reliability for each, and obtain multiple 3D map candidates based on the reliability.

[0054] Furthermore, as described above, search (1) can be processed quickly and is therefore capable of handling a wide area. On the other hand, search (2) can be sophisticated and highly accurate, but it takes time, and if the target area is wide, it may not be possible to narrow down the search results in real time. Therefore, it is advisable to provide a function for selectively using either search depending on the location to be located. For example, if the area within a certain building A is relatively small and the number of 3D maps and feature data to be compared is small, search (2) is used to narrow down the search results. On the other hand, if the area within building B is relatively large and the number of 3D maps and feature data to be compared is large, search (1) is used to narrow down the search results. To determine which search to perform, the control unit 22 may determine whether the user is in building A or building B based on the current location based on sensor information such as GPS signals and Wi-Fi access points, and then switch the search function to operate based on the current location.

[0055] The map narrowing unit 222 may also have a function of narrowing down the target 3D map group in advance based on various sensor information such as GPS (GNSS) information, Wi-Fi information, and geomagnetic information. Furthermore, the control unit 22 may have a function of, if it determines that the transmitted image data is outside the target area, transmitting a notification to that effect to the terminal device 10 and prompting the terminal device 10 to perform necessary processing. The determination of whether or not the target area is outside the target area may be made, for example, based on sensor information, by determining whether or not there are facilities or the like to be located around the current location of the terminal device 10. The determination of whether or not the target area is outside the target area may also be made if no 3D map candidates are detected as a result of a search by the map narrowing unit 222.

[0056] The map narrowing down unit 222 sends information, such as IDs, of one or more extracted 3D map candidates to the map matching unit 221. The map matching unit 221 then performs matching processing only on the 3D map candidates. In this way, since the matching targets are narrowed down, the map matching unit 221 can identify accurate positions and attitudes (three-dimensional orientations).

[0057] In this way, the control unit 22 of this embodiment narrows down the area in terms of content (narrows down the 3D map) and then performs shape feature point matching. Furthermore, the map narrowing down unit 222 can narrow down the area by abstracting not specific objects but comprehensive features including peripheral information, including vector search and LLM, into vectors or intermediate representations in a large-scale language model (also vectors), and then comprehensively performing comparative inference, etc. Then, the map matching unit 221 performs shape-based matching, thereby enabling the current location to be identified with high accuracy.

[0058] That is, as mentioned above, 3D maps of areas with similar shapes have similar features, so they may not be able to be accurately identified by matching only the feature points of the shapes. On the other hand, even if the shapes are similar, when the images are abstracted to include colors, characters, etc., the features differ. For example, the vector directions are different, so the two can be distinguished.

[0059] Therefore, the map narrowing unit 222 narrows down the search, thereby eliminating 3D maps of different areas. On the other hand, abstracted search processing can narrow down the search to similar 3D maps, etc., but cannot identify the exact position or orientation. Therefore, by eliminating similar 3D maps in advance, the map matching unit 221 can identify the position with high accuracy.

[0060] Therefore, the control unit 22 can distinguish between repetitive shapes and similar shapes. For example, the control unit 22 can distinguish between areas that cannot be distinguished by 3D map feature point matching alone. Furthermore, since the number of 3D maps to be processed for 3D map feature point matching can be reduced, the load on the point cloud matching process performed by the map matching unit 221 can be reduced and the speed can be increased, thereby expanding the coverage area.

[0061] In other words, even if objects have similar shapes, when they are abstracted, their characteristics differ, making it possible to distinguish between similar 3D maps and extract the correct 3D map as a candidate.This makes it possible to distinguish complex areas by making a comprehensive judgment using not only specific landmarks or objects, but also multimodal LLMs (large-scale language models) and image recognition processing and machine learning models that recognize overall features.

[0062] For example, in the case of an exhibition, similar booth packaging may be used, as shown in Figures 4 and 5. In these two figures, a similar counter 51 is located at the entrance, and the surrounding partition wall 52 is also similar in shape. Therefore, discrimination based on shape alone may be indistinguishable. On the other hand, the two booths differ in the size, number, and location of exhibits 53, such as posters, attached to the wall, and also in the text information 54, such as company names. Therefore, the feature data abstracting this information is different. Therefore, the two booths can be appropriately identified by narrowing down the map by the map narrowing unit 222.

[0063] Furthermore, for example, in the case of a commercial facility, even if two stores in a shopping mall have similar entrances, if they have different colors, store names, or carry different products, the map narrowing unit 222 can distinguish between them before the map matching unit 221 performs shape feature point matching, thereby narrowing down the target 3D map candidates. This allows for appropriate and efficient shape feature point matching. For example, the stores shown in Figures 6 and 7 are located along a linear corridor, have wide, open entrances, and are similar in shape. However, because the store logos (text information) 55 and wall color schemes are different, the feature data that takes these factors into account is different. Therefore, the two stores can be appropriately identified by narrowing down the candidates using the map narrowing unit 222.

[0064] 8 and 9 have similar configurations and shapes, such as curved lines and ceiling lights 57, but carry different products 58 (for example, books and clothes). Even if the shelves are similar, the products displayed will be different, and the product shapes, colors, and text will also be different. Therefore, the two stores can be appropriately identified by narrowing down the search results using the map narrowing unit 222.

[0065] According to this embodiment, by combining image recognition processing and vector search, which are suitable for capturing general characteristics, with 3D map feature matching, which is suitable for detecting positions with high accuracy, it becomes possible to identify positions that cannot be detected by GPS, such as inside complex buildings.

[0066] In this embodiment, indoor location can be identified using images and sensor information from the smartphone. The terminal device 10 may have a function to, for example, provide guidance to a specified item or destination or present AR content by acquiring the location information identified by the server 20.

[0067] The positioning system of this embodiment can accurately identify locations even indoors where similar shapes exist, something that existing VPSs struggle with. As a result, it is possible to identify the current location without special equipment in large commercial facilities, event venues, etc., and provide guidance and AR content tailored to the location. Furthermore, a system can be built that can update content and destination information in real time through a management app or API, making it possible to use the system in places with dynamic layouts such as retail stores and event venues.

[0068] [Modifications] The position identification system can be modified in various ways as shown below. FIG. 2 is a diagram showing an example of an actual configuration. In this example, the server 20 is realized as a multiple client-server system. The map narrowing unit 222 is implemented in the first cloud system 31. The map matching unit 221 is implemented in the second cloud system 32. This second cloud system 32 performs processing to extract feature points from images and to estimate the current posture from the feature points. Furthermore, the entity data of the 3D map is stored and held in storage 33 associated with the second cloud system 32. Furthermore, the feature data of the 3D map and the 3D map ID are linked in a database 34.

[0069] Some or all of the 3D map narrowing down process and the 3D map matching process may be performed on a terminal device 10 equipped with the photographing equipment, or may be performed on computational resources such as a data center or external services (collectively referred to as the cloud).

[0070] Alternatively, a terminal device with a photographing device and a terminal device performing calculation processing may be configured to exist separately and be connected by wired or wireless communication (collectively referred to as a terminal). The map narrowing process may be performed in the cloud, and the terminal may obtain the results, after which the matching process with the 3D map may be performed again in the cloud, or both the map narrowing process and the matching process with the 3D map may be performed on the cloud and the results obtained. Cloud processing may also be performed across multiple clouds.

[0071] The process of narrowing down the map can be performed on the cloud or on the device. The computational processes, machine learning models, feature data, and other information required for the map narrowing process can be downloaded in advance to the device. The range to be downloaded can be narrowed down using sensor information such as GPS and Wi-Fi. The range can also be narrowed down by user operation.

[0072] The process of extracting features from an image or image sequence can also be performed on the cloud or on the device.

[0073] The process of extracting features from an image or image sequence may be performed before, simultaneously with, or after the map refinement process, or the same images may be used to extract features.

[0074] The process of matching the 3D map with the image features can be performed on the cloud or on the device. The 3D map can be downloaded in advance to perform the matching process on the device, or it can be obtained when the results of the map filtering process are returned.

[0075] The database that holds the feature data and the 3D maps may not be a single database, but the feature data of a specific 3D map may be associated with the 3D map and stored in separate databases or cloud systems.

[0076] It is also possible to use the final location information only for recording and utilization on the cloud without returning it to the device. It is also possible to use images already taken to refine the map or match it with a 3D map.

[0077] In addition, in the above-described embodiments and modifications, 3D maps showing a predetermined indoor area are created for each of the areas divided into multiple sections. However, a single 3D map may be created for the entire predetermined indoor area. In this case, predetermined feature data is associated with each position in the single 3D map and stored in the feature data storage unit. The map narrowing unit narrows down candidate points for shape matching based on the feature data generated based on the acquired image data, etc., and the feature data stored in the feature data storage unit. In other words, the map narrowing unit selects a portion of the 3D map corresponding to the narrowed-down point in the single 3D map as a candidate 3D map. Then, the map matching unit performs shape matching on a portion of the 3D map in the candidate single 3D map.

[0078] By doing this, even in cases where shape matching is not possible for the entire 3D map containing elements of a single, relatively broad, similar shape, shape matching can be performed within a narrowed-down range within a single 3D map, and the location can be identified with high accuracy.

[0079] In this case, a single 3D map may be considered as a collection of multiple 3D maps in the above-described embodiments.

[0080] [Device Equipped with a Camera] In the above-described embodiment, the terminal device 10 is mainly a smartphone. However, it may be a terminal equipped with a camera, such as smart glasses exemplified in the examples. Furthermore, the terminal is not limited to devices carried by people, such as smartphones and smart glasses, but may also be a mobile object that moves on its own. The mobile object may be a mobility such as an automobile or drone, or a robot. For example, application to a mobile object such as a drone or robot may be possible by recognizing which floor it is currently on and where it is located within a multi-story building, and using the information for autonomous driving. Furthermore, in the case of an automobile, it may be possible to recognize its current location in an underground parking lot, for example, and use the information for autonomous driving.

[0081] The terminal does not necessarily have to have a display unit, but may simply have a photographing device.

[0082] Furthermore, the positioning system is not limited to one that identifies the current position based on images captured in real time. For example, the positioning system may be applied to one that performs a series of processes using images that have already been captured and estimates the position at which the image to be processed was captured.

[0083] Various aspects of the present invention have been described above using embodiments and modifications. However, it should be noted that these embodiments and descriptions are provided to aid in understanding the present invention, and are not intended to limit the scope of the present invention. The scope of the present invention is not limited to the structures and manufacturing methods explicitly described in the specification, but also includes combinations of various aspects of the present invention disclosed herein. While the structures of the present invention that are sought to be patented are specified in the appended claims, it is hereby emphasized that structures not currently specified in the claims may be claimed in the future as disclosed herein.

[0084] 1: Positioning system 2: Network 10: Terminal device 11: Control unit 12: Camera 13: Input unit 14: Communication unit 15: Display unit 16: GPS receiving unit 20: Server 21: Communication unit 22: Control unit 23: Database 31: First cloud system 32: Second cloud system 33: Storage 34: Database 221: Map matching unit 222: Map narrowing unit 231: 3D map storage unit 232: Feature data storage unit

Claims

1. A location identification system that identifies the location where an acquired image was taken, comprising: a map narrowing down function that narrows down 3D map candidates to be searched using feature data abstracted based on the image; and a map matching function that matches the shape of the image with the narrowed down 3D map candidates to identify the location where the image was taken.

2. The location identification system according to claim 1, wherein the map narrowing function performs the abstraction using a large-scale language model with context information of the image and the 3D map as input, thereby narrowing down the candidates for the 3D map.

3. The location identification system according to claim 1, wherein the map refinement function creates the feature data by vectorizing information based on the image.

4. The location identification system according to claim 3, wherein the map refinement function creates the feature data by vectorizing text information related to the image.

5. The location system of claim 1, wherein the map refinement function performs the abstraction using the image and context information related to the image.

6. The location identification system according to claim 1, wherein the map narrowing function is performed using location information detected by the terminal that captured the image.

7. The location identification system according to claim 1, further comprising a database that stores feature data to be compared and 3D maps and that is accessible by the map narrowing function and the map matching function.

8. A location identification system as described in claim 1, comprising a camera and a terminal having a communication function, said terminal having a function of transmitting an image taken by said camera using said communication function to a server having said map narrowing function.

9. A computer-readable recording medium having recorded thereon a program for causing a computer to realize the functions of the location specifying system according to any one of claims 1 to 8.

Images