Location identification system and program

Abstract

Accurately identify a location based on images, even indoors SOLUTION: In this system, a control unit 22 of a server 20 that acquires an image captured by a terminal device 10 identifies the position of the terminal device. The server includes a database 23 that stores a large number of 3D maps showing each area and feature data related to the 3D maps. The feature data includes vectorized images of the 3D maps and related text information, context information of the 3D maps, etc. A map narrowing-down unit 222 of the control unit vectorizes the acquired images, etc., searches the vectors against feature data stored in the database, and also provides the acquired images and context information of the 3D maps to the LLM to narrow down the 3D map candidates (eliminates 3D maps with similar shapes). A map matching unit 221 matches the shapes of the narrowed-down 3D map candidates to accurately identify the position and orientation.

Description

[Technical field]

[0001] The present invention relates to a location identification system and a program. [Background technology]

[0002] Measurement of the position and orientation of an imaging device based on image information is used for various purposes, such as aligning real space with virtual objects in mixed reality / augmented reality, and estimating the position of users carrying imaging devices, robots, cars, drones, and other mobility devices. s Position detection using the Global Positioning System (GPS) can pinpoint a location even in an environment where GPS satellite signals cannot be received. Therefore, for example, if a user carrying an image capture device is inside a building or underground, the location within that space can be pinpointed.

[0003] This type of system for identifying a position using captured image information is disclosed in, for example, Patent Document 1. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7361075 Summary of the Invention [Problem to be solved by the invention]

[0005] In supermarkets and convenience stores, the sales floors tend to have similar display shelves arranged in a similar layout. In addition, the stores set up inside a commercial complex tend to use rooms of the same size along the corridors, and some have similar entrances. Exhibitions held at exhibition centers often have many booths with the same dimensions and shapes, arranged in the same layout.

[0006] In such cases, for example, there may be many similarly shaped sales areas in the same supermarket, and the feature points of images taken at each sales area may be similar. When features are similar in this way, the accuracy of localization using VPS decreases. This also occurs indoors in commercial complexes and exhibition halls.

[0007] Furthermore, when performing image-based localization within a wide indoor area, there are many images to compare. If there are many images with similar features, the system may not be able to correctly detect the location or may freeze up without being able to identify one.

[0008] The above-mentioned problems are described as being independent of each other, and the present invention does not necessarily have to solve all of the problems described, but it is sufficient if it can solve at least one of the problems. In addition, we intend to obtain rights to the configuration for solving this problem separately through divisional applications, amendments, etc. [Means for solving the problem]

[0009] (1) A location identification system that identifies the location where an acquired image was taken, the location identification system preferably having a map narrowing down function that narrows down candidates for 3D maps to be searched using feature data abstracted based on the image, and a map matching function that matches the shape of the image with the narrowed down candidates for 3D maps to identify the location where the image was taken.

[0010] In this way, it is possible to perform shape feature point matching with high accuracy after narrowing down the area in terms of content (narrowing down the 3D maps). The process of narrowing down the 3D map candidates includes, for example, selecting a predetermined number of 3D maps for shape matching from multiple 3D maps, selecting a predetermined number of 3D map parts of the area for shape matching within one 3D map, or a combination of these.

[0011] (2) The map narrowing down function may be configured to perform the abstraction using a large-scale language model with context information of the image and the 3D map as input, thereby narrowing down candidates for the 3D map.

[0012] Although searches using large-scale language models cannot pinpoint a location or pinpoint a posture or orientation, they can identify similar shapes by taking their content into account. This allows for accurate selection of 3D map candidates. By narrowing down the 3D map candidates, the map matching function in the subsequent stage can perform accurate location identification, etc., by matching shapes.

[0013] (3) The map narrowing function may be configured to vectorize information based on the image to create the feature data.

[0014] (4) The map refinement function may be configured to vectorize text information related to the image to create the feature data. The text information related to the image may be information created by character recognition of characters contained in the image, information showing objects present in the image through image recognition, explanatory documents for the image, etc. The explanatory documents may be information created in advance by a person or information generated by providing an image to a generation AI.

[0015] (5) The abstraction performed by the map refinement function may be configured to be performed using the image and context information related to the image.

[0016] (6) The map narrowing function may be configured to utilize location information detected by the terminal that captured the image.

[0017] (7) It is preferable to configure the system so as to include a database that stores feature data to be compared and 3D maps and that can be accessed by the map narrowing down function and the map matching function.

[0018] (8) The present invention may be configured to include a camera and a terminal having a communication function, and the terminal may be configured to transmit images captured by the camera to a server having the map filtering function using the communication function. The terminal may also have a function of acquiring a position identified by the map matching function and notifying the acquired position.

[0019] (9) The program of the present invention may be a program for causing a computer to realize the functions of the position specifying system according to any one of (1) to (8).

[0020] The components of each invention described above in (1) to (8) can be arbitrarily combined in two or more parts, and the combination can constitute an invention. Effect of the Invention

[0021] The present invention can, for example, narrow down candidate 3D maps to be searched using abstracted feature data such as vector search or inference using large-scale language models, and then match feature points with the narrowed-down 3D maps to accurately identify a location even within a space containing similar shapes. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a diagram showing a preferred embodiment of a location determination system according to the present invention. [Diagram 2] FIG. 1 illustrates an example of an actual system configuration. [Diagram 3] 13 is a flowchart illustrating a process of a control unit of the server. [Figure 4] FIG. 13 is a diagram showing an example of a place for identifying an indoor position. [Diagram 5] FIG. 13 is a diagram showing an example of a place for identifying an indoor position. [Figure 6] FIG. 13 is a diagram showing an example of a place for identifying an indoor position. [Figure 7] FIG. 13 is a diagram showing an example of a place for identifying an indoor position. [Figure 8]FIG. 13 is a diagram showing an example of a place for identifying an indoor position. [Figure 9] FIG. 13 is a diagram showing an example of a place for identifying an indoor position. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to these embodiments, and various changes, modifications, and improvements can be made based on the knowledge of those skilled in the art without departing from the scope of the present invention.

[0024] The positioning system according to the present invention performs accurate positioning in indoor locations where GPS signals cannot be received accurately, such as exhibition halls, commercial facilities, various buildings, underground shopping malls, and underground parking lots. For the indoor area where positioning is to be performed, a 3D map showing a specific indoor area and feature data related to the 3D map are created and stored using various calculation processes, statistical models, and machine learning models based on information acquired in advance using a camera or LiDAR (Light Detection And Ranging). The positioning system has a function of acquiring image data actually taken on-site, narrowing down indoor locations based on the acquired image data and the stored feature data, and further accurately identifying the location by matching feature points with the stored 3D map.

[0025] Fig. 1 is a configuration diagram showing a preferred embodiment of a position identification system 1 according to the present invention, and Fig. 2 is a diagram showing an example of an actual configuration. As shown in Fig. 1, the position identification system 1 has a function in which a server 20 acquires image data captured by a terminal device 10 via a network 2 and identifies the location of the terminal device 10 based on the acquired image data. The network 2 is, for example, a public line or the Internet, and is preferably accessible even when the devices are located far from each other.

[0026] The terminal device 10 includes a control unit 11, a camera 12, an input unit 13, a communication unit 14, a display unit 15, a GPS receiver unit 16, etc. When the terminal device 10 is carried by a person, for example, a smartphone, smart glasses, etc. are preferably used. When the terminal device 10 is a smartphone, the hardware constituting the terminal device 10 (the control unit 11, the camera 12, the input unit 13, the communication unit 14, the display unit 15, and the GPS receiver unit 16) uses equipment that is standard equipment on a smartphone.

[0027] The control unit 11 controls each part of the terminal device 10. The control unit 11 is, for example, a computer including a processor and a memory. The memory is, for example, a main storage device having a RAM (Random Access Memory) and a ROM (Read Only Memory). The processor temporarily stores a program read from the ROM of the memory in the RAM. The RAM of the memory provides a working area for the processor. The processor performs various controls by temporarily storing data generated during the execution of the program in the RAM while performing arithmetic processing. In addition, when the terminal device 10 is a smartphone, the process for identifying the location may be performed by the control unit activating a smartphone app, for example.

[0028] The communication unit 14 has a function of connecting to the above-mentioned network 2, transmitting and receiving data and information to and from other devices connected to the network 2, connecting to a Wi-Fi access point, and the like.

[0029] The control unit 11 displays an image captured by the camera 12 on the display unit 15. The control unit 11 also transmits image data being captured to the server 20 using the network 2. The control unit 11 also transmits sensor information based on the current location to the server 20 using the network 2. The sensor information includes, for example, location information (longitude and latitude) received and acquired by the GPS receiving unit 16, and information identifying a Wi-Fi access point acquired by the communication unit 14. The timing for transmitting such information may be periodically at a predetermined interval, or when a transmission instruction received by the input unit 13 is triggered. The control unit 11 also includes a function for displaying predetermined information on the display unit 15 based on the current location acquired from the server 20.

[0030] The server 20 includes a communication unit 21 having an interface for connecting to and communicating with a network 2 such as an Internet connection function, a control unit 22 for controlling the operation of the server 20, and a database 23 for storing data for identifying a location. In addition, while FIG. 1 illustrates an example in which the server 20 is configured from one server computer, as shown in FIG. 2, the server 20 may be configured from multiple computers connected directly or via a network. In addition, the server implementing the database is not limited to a physical server, and may be in various forms such as a software program or implemented in the cloud.

[0031] The database 23 stores data for identifying a position based on an image captured by the camera 12 of the terminal device 10. The database 23 includes a 3D map storage unit 231 and a feature data storage unit 232. The 3D map storage unit 231 stores a 3D map of a target area for which position identification is to be performed, which is created in advance. That is, the 3D map is data based on a three-dimensional shape composed of a collection of feature points (x, y, z) or a polygon mesh created using a large number of images and further LiDAR depth data associated with each image. Therefore, a system administrator, etc., moves within an indoor area for which position identification is to be performed in advance, photographs the image data of each point using a camera, and acquires depth data using LiDAR (Light Detection And Ranging) as necessary. Based on a plurality of image data and depth data acquired at a certain point indoors, various calculation processes are performed to create a 3D map. This 3D map creation process can be performed using existing technology.

[0032] Within the same indoor space, there may be multiple areas with similar shapes. As a result, there are similar shapes and feature points of 3D maps in different locations. In particular, in the case of the same exhibition, there are many booths with similar dimensions and shapes, and the layout of multiple booths within the venue is often the same. Also, in large commercial facilities and shopping malls, there may be multiple stores with similar entrances. Also, even within the same store, if many identical display shelves are prepared and arranged in the same layout, there will be many similar 3D maps.

[0033] The process of creating a 3D map for a given location may involve, for example, dividing the area so that the characteristics of the 3D map data are uniquely determined. A 3D map of the divided area may be created by performing three-dimensional reconstruction from a sequence of overlapping images acquired in the divided area and the attitude information associated with each image. In addition, if seismic intensity data acquired using LiDAR is available, this may also be used to create a 3D map of the divided area. If LiDAR data is not available, the 3D map may be created based only on image data.

[0034] For example, in the case of an exhibition hall, the divided range may be a fixed area such as a corner unit or a booth unit of an event venue. In the case of a shopping mall, the divided range may be, for example, an area including the entrance of each store. In addition, the divided range within a store may be, for example, an area including one or more display shelves. The divided range may be selected appropriately, and in either case, one 3D map is created for each divided area. An ID is assigned to each 3D map, and location information identifying the actual area indoors is also associated and registered.

[0035] In this way, 3D maps are created for each area that is divided into a certain indoor area, so for example, many 3D maps are prepared for the same exhibition or event venue held at a single exhibition center.Even in the interior of a single commercial facility, 3D maps are created for each of the multiple divided areas.

[0036] The server 20 stores 3D maps for multiple areas in different buildings and facilities. The server 20 stores the maps so that it is clear which exhibition center or commercial facility the 3D map is for. For example, the 3D maps may be stored in separate storage areas for each building, or identification information for identifying the building or the like may be registered as supplementary information for the 3D map. Even if the exhibition center is the same, the 3D maps will be different if different exhibitions are held, so the 3D maps are stored in the 3D map storage unit 231 for each exhibition.

[0037] The feature data storage unit 232 stores feature data that is abstracted into an intermediate representation from one or more pieces of image data captured in an area for which a 3D map is created. This feature data is associated with each 3D map, and one or more pieces of feature data are included. The abstracted feature data includes, for example, data obtained by vectorizing an image used when creating a 3D map using a multimodal embedding model, explanatory information (context) present in the image of each divided range, and associated location information for specifying an approximate location. The context may be text data, but may be a vectorized multidimensional vector. The vector representing the feature data may be created, for example, by detecting characters or objects from an image to generate an explanatory text, and then vectorizing the text information using an embedding model. In addition, the vector representing the feature data is not limited to one that uses text information, and may be, for example, a feature vector directly generated by convolving an image. By generating a vector based on an image, feature data based on information such as the appearance of a wall that cannot be generated by character detection can be generated, which is advantageous in that there is less information leakage. Also, information combining explanatory information from multiple images taken from different angles within the same range may be used as the context information for a single 3D map. The text used to create this explanatory information may be created, for example, by image recognition of image data and using OCR technology. It is more preferable that explanatory information is generated using a multimodal large-scale language model, with the image as input, to generate explanatory text. This creates text that describes the information in the image as if it were being explained by a human, and can also include non-textual information such as shape and color, so that feature data can be generated that can be more accurately distinguished from other locations. Furthermore, it is preferable that a worker who has viewed the created explanatory information (explanatory text) corrects the text, retakes the image, and creates explanatory information again.

[0038] The associated location information includes, for example, Wi-Fi access points, GPS information, etc. For example, since there are areas where GPS signals can be received even indoors, the approximate location can be identified from GPS-based location information. In addition, Wi-Fi access points are installed at appropriate locations in underground shopping malls, etc. The communication area of ​​Wi-Fi access points is narrow. Therefore, information identifying the access point is used as feature data to associate the access point with a 3D map of the communication area.

[0039] The creation of the 3D map and the feature data to be stored in the database 23 may be performed in any order as long as they are based on the same area.

[0040] The control unit 22 is, for example, a computer including a processor and a memory. The memory is, for example, a main storage device having a RAM (Random Access Memory) and a ROM (Read Only Memory). The processor temporarily stores a program read from the ROM of the memory in the RAM. The RAM of the memory also provides a working area for the processor. The processor performs various controls by temporarily storing data generated during execution of the program in the RAM while performing arithmetic processing.

[0041] The control unit 22 includes a map matching unit 221 and a map narrowing unit 222 as functions for performing position identification. The map matching unit 221 performs matching based on shape. That is, the map matching unit 221 matches feature points extracted based on image data captured by the terminal device 10 with a group of 3D maps stored in the 3D map storage unit 231, identifies one 3D map with a high degree of match, and derives the position and attitude at the time of shooting with the camera 12 of the terminal device 10. Furthermore, the map matching unit 221 may include a function for determining the coordinates on the 3D map of the terminal device 10 and the coordinates in the entire group of 3D maps based on the derived shooting position and attitude. In addition, the matching process based on shape may utilize an existing position identification algorithm based on a comparison between image data captured by a VPS or the like and a 3D map prepared in advance. Then, the control unit 22 sends position information related to the identified position, attitude, etc. to the terminal device 10 via the network 2.

[0042] Prior to shape matching by the map matching unit 221, the map narrowing down unit 222 narrows down the area where the image was taken, using the acquired image data, etc. as a preliminary step. That is, the map narrowing down unit 222 narrows down the map by abstracting the content represented by the image data. The map narrowing down unit 222 extracts candidates for the 3D map based on the acquired image data, etc. and the feature data stored in the feature data storage unit 232. The detailed functions of this map narrowing down unit 222 will be described later.

[0043] FIG. 3 is a flowchart explaining the processing of the control unit 22 when a user who actually uses the system acquires an image captured by the camera 12 of the terminal device 10 at the site. The user who uses the system starts a predetermined smartphone app installed on a smartphone, which is an example of the terminal device 10. The smartphone app accepts a predetermined input instruction, starts the camera 12, switches to a mode for continuous shooting, and displays the image being captured on the display unit 15. The smartphone app also sends the image being captured to the server 20 at a predetermined timing. This predetermined timing may be a set time interval, or the acceptance of a transmission instruction given from the input unit 13, such as a tap on the screen. The control unit 11 also transmits the predetermined sensor information together with the image data.

[0044] The control unit 22 of the server 20 acquires an image captured by the terminal device 10 and sensor information (S1). The control unit 22 stores the acquired image data in a predetermined storage unit. The storage unit may be, for example, a temporary storage unit such as a volatile memory.

[0045] The map narrowing down unit 222 of the control unit 22 compares the acquired image with the context in a multimodal large-scale language model to determine the area, converts it into feature information by processing using a machine learning model, and narrows down the 3D map candidates in which the image for detecting the photographed position is located (S2). The map narrowing down unit 222 sends the IDs of the narrowed down 3D map candidates to the map matching unit 221.

[0046] Next, the map matching unit 221 reads out the image data stored in the temporary storage unit, and extracts feature points using a machine learning model or a SLAM technique (S3).Then, the map matching unit 221 matches the obtained feature points with the candidate 3D map narrowed down by the map narrowing down unit 222, identifies the most matching 3D map, and derives the position and orientation within the 3D map (S4).

[0047] The control unit 22 calculates the coordinates of the terminal on the 3D map or the coordinates within the entire group of 3D maps based on the derived shooting position and attitude (S5). Then, the calculated coordinate data is sent to the terminal device 10. The smartphone app performs, for example, route guidance based on the position specified by the received coordinate data.

[0048] The map narrowing down unit 222 that performs the above process S2 has (1) an image recognition and vector search function, and (2) a search function that uses multimodal LLMs (Large Language Models).

[0049] The image recognition and vector search function (1) converts various information into feature vectors by some method, and performs position identification based on the vector search, which is a similarity search. This vector search function vectorizes the acquired input image (sequence), calculates the degree of match with the vector associated with the 3D map stored in the feature data storage unit 232, and extracts feature data that meets a predetermined criterion. The control unit 22 determines the 3D map associated with the extracted feature data as a map candidate. The predetermined criterion may be, for example, a match exceeding a threshold value, or a predetermined number of matches with high degrees of match. The vectorization process of the input image may vectorize the image directly using a multimodal embedding model (generating one abstract feature data from multiple pieces of information), or may detect characters in the image, abstract the obtained text information, and vectorize it. The text information to be abstracted may be, for example, characters present in the image extracted by performing image recognition processing. In addition, the text information may be a description of the image generated by the generation AI, which is given the name of the object present in the image directly or by image recognition. This vector search can be processed at high speed, so when there are many 3D maps to compare, it is possible to efficiently determine in a short time whether a 3D map is eligible.

[0050] The search function using LLM in (2) uses image and character information as input and performs logical inference using a large-scale language model to identify a position. This search function using LLM provides information on the candidate map stored in the feature data storage unit 232 (such as a description) as a context to the multimodal LLM, provides an input of an image captured by the terminal device 10 and acquired via the network 2, and estimates a candidate 3D map. In other words, the context information of the 3D map and image information captured by the terminal device 10 are provided to the LLM, and the LLM finds 3D map candidates. The LLM model itself may be fine-tuned. A search using LLM has the characteristic that it is more accurate than a vector search, but takes longer to process. A search using LLM has limitations on the number of areas it can handle.

[0051] Furthermore, when a user who actually uses the system takes a photograph on-site with the camera 12 of the terminal device 10, the captured image data will differ if the orientation of the camera 12 is different, even if the camera is in the same position when the feature data, etc., was created in advance. Even in such a case, for example, in the case of a multimodal large-scale language model, image information from multiple angles is combined and edited to be handled as feature data that effectively describes the range to which the 3D map applies, so that the model is particularly tolerant of differences in angle. Therefore, even if the orientation of the camera 12 is different, it can be extracted as a 3D map candidate by narrowing down the map if it is within a certain range.

[0052] On the other hand, in the case of vector search, if the orientation is different even when the image is taken from the same position, the degree of match is low, and if the difference in angle is large, detection is not possible. Therefore, in the case of vector search, it is advisable to create feature vectors for images in multiple orientations within the range of one 3D map, and associate each vector with the ID of the same map. In other words, when creating feature data in advance and storing it in the database 23, multiple feature vectors are associated with one 3D map. In this way, the 3D map linked to the image taken during use can be appropriately identified by matching the feature vectors.

[0053] Either one of the functions (1) and (2) may be used, or both may be combined. When using one, it is advisable to use the more versatile (2). Also, since (2) has limitations in processing speed and the number of areas that can be processed, it is also possible to narrow down the results using only the image recognition and vector search (1). When combining these, for example, when there are a large number of 3D map groups, it is advisable to narrow down the results using (1) and make the final decision using (2). Also, instead of dividing the order of use in this way, it is advisable to operate the search functions (1) and (2) in parallel, calculate the reliability for each, and obtain multiple 3D map candidates based on the reliability.

[0054] As described above, the search (1) can be processed at high speed, and therefore has the characteristic of being applicable to a wide area. On the other hand, the search (2) can be performed with high accuracy and is time-consuming, and there is a risk that the search cannot be narrowed down in real time if the target is wide. Therefore, it is preferable to provide a function for selectively using either one depending on the location to be specified. For example, when the area in a certain building A is relatively small and the number of 3D maps and feature data to be compared is small, the search (2) is used to narrow down the search. On the other hand, when the area in a building B is relatively large and the number of 3D maps and feature data to be compared is large, the search (1) is used to narrow down the search. As to which search to perform, for example, the control unit 22 recognizes whether the current location is in building A or building B based on sensor information such as GPS signals and Wi-Fi access points, and switches the search function to operate based on the current location.

[0055] The map narrowing unit 222 may also have a function of narrowing down the target 3D map group in advance based on various sensor information such as GPS (GNSS) information, Wi-Fi information, and geomagnetic information. Furthermore, the control unit 22 has a function of, when it is determined that the sent image data is outside the corresponding area, transmitting that fact to the terminal device 10 and prompting the terminal device 10 to perform necessary processing. The determination of whether or not the target area is outside may be performed, for example, based on sensor information, depending on whether or not there are facilities or the like to be located around the current position where the terminal device 10 is located. The determination of whether or not the target area is outside may also be performed by determining that the area is outside the target area when no 3D map candidate is detected as a result of the search by the map narrowing unit 222.

[0056] The map narrowing down unit 222 sends information, such as IDs, of one or more extracted 3D map candidates to the map matching unit 221. The map matching unit 221 then performs matching processing only on the 3D map candidates. In this way, since the matching targets are narrowed down, the map matching unit 221 can identify accurate positions and attitudes (three-dimensional orientations).

[0057] In this way, the control unit 22 of this embodiment narrows down the area in terms of content (narrows down the 3D map), and then performs shape feature point matching. Also, the map narrowing down unit 222 can narrow down the area by abstracting not specific things but comprehensive features including peripheral information, including vector search and LLM, into vectors or intermediate representations in a large-scale language model (which are also vectors), and then performing comprehensive comparison inference, etc. Then, the map matching unit 221 performs matching based on the shape, thereby enabling the current position to be identified with high accuracy.

[0058] That is, as mentioned above, 3D maps of areas with similar shapes have similar features, so they may not be able to be accurately identified by just matching feature points of the shapes. On the other hand, even if the shapes are similar, when the images are abstracted to include colors and characters, the features are different. For example, the vector directions are different, so the two can be distinguished.

[0059] Therefore, the map narrowing down unit 222 narrows down the search, thereby eliminating 3D maps of different areas. On the other hand, abstracted search processing can narrow down the rough location of the object, such as similar 3D maps, but cannot identify the exact location or orientation. Therefore, by eliminating similar 3D maps in advance, the map matching unit 221 can identify the location with high accuracy.

[0060] Therefore, the control unit 22 can distinguish between repetitive shapes and similar shapes. For example, the control unit 22 can distinguish between areas that cannot be distinguished by only matching feature points of 3D maps. In addition, since the number of 3D maps to be processed for matching feature points of 3D maps can be reduced, the load of the point cloud matching process performed by the map matching unit 221 can be reduced and the speed can be increased, and the corresponding area can be expanded.

[0061] In other words, even if objects have similar shapes, when abstracted, their characteristics differ, making it possible to distinguish similar 3D maps and extract the correct 3D map as a candidate. Therefore, it is possible to distinguish complex areas by making a comprehensive judgment using not only specific landmarks or objects, but also multimodal LLMs (large-scale language models) and image recognition processing and machine learning models that recognize overall features.

[0062] For example, in the case of an exhibition where 3D maps are similar, similar booth packages may be used, as shown in Figs. 4 and 5. In these two figures, a similar counter 51 is located at the entrance, and the partition wall 52 from the surrounding area is also similar in shape. Therefore, if the booths are judged based only on their shapes, they may not be able to be identified. On the other hand, the two booths differ in the size, number, and installation position of exhibits 53 such as posters pasted on the walls, and the text information 54 such as company names is also different. Therefore, the feature data that abstracts such information is different. Therefore, the two booths can be appropriately identified by narrowing down the booths using the map narrowing down unit 222.

[0063] In addition, for example, in the case of a commercial facility, even if there are stores in a shopping mall with similar entrances, if the colors or store names are different, or the products they handle are different, the map narrowing down unit 222 performs discrimination in a stage prior to shape feature point matching using the map matching unit 221, and narrows down the target 3D map candidates, thereby enabling appropriate and efficient shape feature point matching. For example, the stores shown in Figs. 6 and 7 are arranged along a straight passage, have entrances with wide open entrances, and have similar shapes. However, the store logos (text information) 55 and color schemes on the walls are different, so the feature data taking these factors into account is different. Therefore, the two stores can be appropriately identified by narrowing down the stores using the map narrowing down unit 222.

[0064] 8 and 9 are similar in configuration and shape of the curved shape and ceiling lights 57, but handle different products 58 (for example, books and clothes). Even if the shelves are similar, if the products displayed are different, the shapes and colors of the products will be different, and the characters will also be different. Therefore, the two stores can be appropriately identified by narrowing down the search by the map narrowing down unit 222.

[0065] According to this embodiment, by combining image recognition processing and vector search, which are suitable for capturing rough characteristics, with 3D map feature matching, which is suitable for detecting positions with high accuracy, it becomes possible to pinpoint positions that cannot be detected by GPS, such as inside complex buildings.

[0066] In this embodiment, indoor positioning can be performed by utilizing images and sensor information of the smartphone. The terminal device 10 may have a function of, for example, guiding to a specified item or destination or presenting AR content by acquiring the position information specified by the server 20.

[0067] The positioning system of this embodiment can accurately identify a position even indoors where similar shapes exist, something that existing VPSs have difficulty with. As a result, it is possible to identify the current position without special equipment in large commercial facilities and event venues, and provide guidance and AR content according to the location. In addition, a system can be built that can update content and destination information in real time through a management app or API, making it possible to use it in places with dynamic layouts such as retail stores and event venues.

[0068] [Variations] The position identification system can be modified in various ways as shown below. FIG. 2 is a diagram showing an example of an actual configuration. In this example, the server 20 is realized as a multiple client-server system. The map narrowing down unit 222 is implemented in the first cloud system 31. The map matching unit 221 is implemented in the second cloud system 32. This second cloud system 32 performs a process of extracting feature points from an image and a process of estimating a current posture from the feature points. Furthermore, the entity data of the 3D map is stored and held in a storage 33 related to the second cloud system 32. Furthermore, the feature data of the 3D map and the ID of the 3D map are linked in a database 34.

[0069] A part or all of the 3D map narrowing down process and the 3D map matching process may be performed on a terminal device 10 having an imaging device, or may be performed on a computational resource such as a data center or an external service (collectively referred to as the cloud).

[0070] Also, a terminal device with a photographing device and a terminal device performing calculation processing may exist separately, and may be connected by wired or wireless communication (collectively referred to as terminals). The map narrowing down process may be performed in the cloud, and the terminal may obtain the results, and then the matching process with the 3D map may be performed again in the cloud, or the map narrowing down process and the matching process with the 3D map may all be performed on the cloud, and the results obtained. Also, cloud processing may be performed across multiple clouds.

[0071] The process of narrowing down the map may be performed on the cloud or on the terminal. The calculation process, machine learning model, feature data, etc. required for the map narrowing down process can be downloaded in advance on the terminal. The range to be downloaded can be narrowed down using sensor information such as GPS and Wi-Fi. Narrowing down can also be done by user operation.

[0072] The process of extracting features from an image or image sequence can also be performed on the cloud or on the device.

[0073] The process of extracting features from an image or image sequence may be performed before, during or after the map refinement process, and the features may be extracted from the same images as those used for the map refinement.

[0074] The process of matching the 3D map with the image features can be performed on the cloud or on the device. The 3D map can be downloaded in advance to perform the matching process on the device, or it can be obtained when the results of the map filtering process are returned.

[0075] The database for holding the feature data and the 3D maps may not be a single one, and the feature data of a specific 3D map may be associated with the 3D map and stored in separate databases or in a cloud system.

[0076] It is also possible to use the final location information only for recording and utilizing it on the cloud without returning it to the device. It is also possible to use images that have already been taken to refine the map or match it with a 3D map.

[0077] In addition, in the above-described embodiments and modified examples, in this embodiment, the 3D map showing a predetermined indoor area is created for each area into which the area is divided, but a single 3D map may be created for the entire predetermined indoor area. In this case, each position in the single 3D map is associated with a predetermined feature data and stored in the feature data storage unit. The map narrowing down unit narrows down candidate points for shape matching based on the feature data generated based on the acquired image data, etc., and the feature data stored in the feature data storage unit. That is, the part of the 3D map corresponding to the narrowed down point in the single 3D map is selected as a candidate for the 3D map. Then, the map matching unit performs shape matching on a part of the 3D map in the candidate single 3D map.

[0078] In this way, even in cases where shape matching would be impossible across an entire 3D map that contains elements of a single, relatively broad, similar shape, shape matching can be performed within a narrowed-down area within a single 3D map, enabling accurate location identification.

[0079] In this case, a single 3D map may be considered as a collection of multiple 3D maps in the above-described embodiments.

[0080] [Device equipped with a camera] In the above-described embodiment, the terminal device 10 is mainly a smartphone, but may be a terminal equipped with a camera, such as smart glasses exemplified in the embodiment. The terminal is not limited to a device carried by a person, such as a smartphone or smart glasses, but may be a mobile object that moves by itself. The mobile object may be configured as a mobility such as an automobile or drone, or a robot. For application to a mobile object such as a drone or robot, for example, it may be possible to recognize which floor the person is currently on in a multi-story building and where the person is on that floor, and use the device for automatic driving. In the case of an automobile, the current position may be recognized in an underground parking lot, for example, and the device may be used for automatic driving.

[0081] The terminal does not necessarily have to have a display unit, but may simply have a photographing device.

[0082] Furthermore, the position identification system is not limited to one that identifies the current position based on an image captured and acquired in real time. For example, the position identification system may be applied to one that performs a series of processes using images that have already been captured and estimates the position at which the image to be processed was captured.

[0083] Various aspects of the present invention have been described above using embodiments and modified examples, but it should be noted that these embodiments and descriptions are not intended to limit the scope of the present invention, but are provided to aid in understanding the present invention. The scope of the present invention is not limited to the configurations and manufacturing methods explicitly described in the specification, but also includes combinations of various aspects of the present invention disclosed in this specification. Although the configurations of the present invention that are sought to be patented are specified in the appended claims, it is noted for the sake of clarity that configurations disclosed in this specification may be claimed in the future, even if they are not currently specified in the claims. [Explanation of symbols]

[0084] 1: Location system 2: Network 10: Terminal device 11: Control section 12: Camera 13: Input section 14: Communications Department 15:Display section 16: GPS receiver 20: Server 21: Communications Department 22: Control section 23: Database 31: First cloud system 32: Second cloud system 33: Storage 34: Database 221: Map matching section 222: Map Filtering Section 231: 3D map storage unit 232: Feature data storage unit

Claims

1. A location identification system for identifying a location where an acquired image was taken, comprising: A map narrowing function that narrows down candidates of 3D maps to be searched using feature data abstracted based on the image; A map matching function that performs shape matching of the image with the narrowed-down candidates of the 3D map and identifies the location where the image was taken; Equipped with The map narrowing function performs the abstraction using a large-scale language model with the image and context information of the 3D map as input, and narrows down candidates for the 3D map. Location system.

2. A location identification system for identifying a location where an acquired image was taken, comprising: A map narrowing function that narrows down candidates of 3D maps to be searched using feature data abstracted based on the image; A map matching function that performs shape matching of the image with the narrowed-down candidates of the 3D map and identifies the location where the image was taken; Equipped with The map refinement function vectorizes information based on the image to generate the feature data. Location system.

3. A location identification system for identifying a location where an acquired image was taken, comprising: A map narrowing function that narrows down candidates of 3D maps to be searched using feature data abstracted based on the image; A map matching function that performs shape matching of the image with the narrowed-down candidates of the 3D map and identifies the location where the image was taken; Equipped with The map refinement function vectorizes text information related to the image to create the feature data. Location system.

4. A location identification system for identifying a location where an acquired image was taken, comprising: A map narrowing function that narrows down candidates of 3D maps to be searched using feature data abstracted based on the image; A map matching function that performs shape matching of the image with the narrowed-down candidates of the 3D map and identifies the location where the image was taken; Equipped with The abstraction performed by the map refinement function is performed using the image and context information related to the image. Location system.

5. A program for causing a computer to realize the functions of the position specifying system according to any one of claims 1 to 4.

Images