Information processing system and information processing method

Abstract

To reduce the time required for specifying position and orientation of an information terminal on the basis of feature point group data generated by the information terminal.SOLUTION: An information processing method includes the steps of: generating, by an information terminal 1, captured image data by imaging at least a partial area of a space to which the information terminal 1 belongs; transmitting, by the information terminal 1, feature point group data composed of a plurality of feature point groups of a subject included in the captured image data; acquiring, by a plurality of computers 30, the feature point group data; and providing, by one of the computers 30, multiple pieces of registered feature point group data corresponding to the computer 30 to the information terminal when a difference between the acquired feature point group data and one of the multiple pieces of registered feature point group data corresponding to a space allocated to the computer 30 is equal to or smaller than a threshold.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an information processing system and an information processing method for displaying an augmented reality image. [Background technology]

[0002] Conventionally, a system is known in which an augmented reality image to be provided to an information terminal is identified based on feature points of a subject contained in an image generated by an information terminal by capturing an image of the subject, and the augmented reality image is superimposed on the captured image and displayed on a display unit (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-175384 A Summary of the Invention [Problem to be solved by the invention]

[0004] In order to identify the position and orientation of an information terminal based on feature point group data created by the information terminal identifying multiple feature points of a subject, it is necessary to compare the created feature point group data with a large number of registered feature point group data that are associated in advance with a space. However, when the information terminal compares the created feature point group data with a large number of registered feature point group data, there is a problem in that it takes a long time to identify the position and orientation of the information terminal.

[0005] Therefore, the present invention has been made in consideration of these points, and has the effect of shortening the time required to identify the position and orientation of an information terminal based on feature point group data created by the information terminal. [Means for solving the problem]

[0006] An information processing system according to a first aspect of the present invention comprises an information terminal, an information processing device capable of communicating with the information terminal, and a computer group including a plurality of computers, the information terminal having an imaging unit that generates captured image data by imaging at least a partial area of ​​a space to which the information terminal belongs, and a transmission processing unit that transmits feature point cloud data composed of a plurality of feature points of a subject included in the captured image data to the information processing device, the information processing device having a terminal data receiving unit that receives the feature point cloud data from the information terminal, and a feature point group transmitting unit that transmits the feature point group data received from the information terminal to the computer group, the plurality of computers having a feature point group acquisition unit that acquires the feature point group data from the information processing device, and a data analysis unit that transmits predetermined information to the information processing device when a difference between the feature point group data acquired by the feature point group acquisition unit and any of the plurality of registered feature point group data corresponding to a space assigned to each computer is equal to or smaller than a threshold, and the information processing device has an information receiving unit that receives the predetermined information, and a spatial data transmitting unit that transmits the plurality of registered feature point group data corresponding to the computer that transmitted the predetermined information to the information terminal.

[0007] The information terminal may further include a detailed position identification unit that identifies a position and orientation corresponding to registered feature point cloud data that is most similar to the feature point cloud data among the plurality of registered feature point cloud data received from the information processing device.

[0008] The information processing device may have a plurality of in-memory databases storing the feature point cloud data corresponding to each of the plurality of spaces, and the feature point cloud transmission unit may transmit the feature point cloud data to the computer associated with a space corresponding to an in-memory database among the plurality of in-memory databases in which the feature point cloud data is written.

[0009] The information terminal may further have an approximate position identification unit that generates approximate position data indicating an approximate position of the information terminal based on received radio waves, and the transmission processing unit may write the feature point group data to one of the multiple in-memory databases that corresponds to the space to which the approximate position belongs.

[0010] The feature point group transmission unit may transmit the feature point group data to the computer associated with a space corresponding to data to which the approximate position indicated by the approximate position data transmitted by the information terminal together with the feature point group data belongs.

[0011] The feature point group acquisition unit may acquire the plurality of registered feature point group data corresponding to the spaces assigned to its own computer from a storage device that stores the registered feature point group data corresponding to the plurality of spaces.

[0012] An information processing method of a second aspect of the present invention includes the steps of: an information terminal generating captured image data by capturing an image of at least a portion of a space to which the information terminal belongs; the information terminal transmitting feature point cloud data composed of a plurality of feature points of a subject included in the captured image data; a plurality of computers acquiring the feature point cloud data; and, when a difference between the acquired feature point cloud data and any of the plurality of registered feature point cloud data corresponding to a space assigned to the computer is equal to or less than a threshold, any one of the plurality of computers providing the information terminal with the plurality of registered feature point cloud data corresponding to the computer. Effect of the Invention

[0013] According to the present invention, it is possible to reduce the time required to identify the position and orientation of an information terminal based on feature point group data created by the information terminal. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an overview of an information processing system. [Diagram 2] FIG. 2 is a diagram for explaining a basic operation of the information processing system. [Diagram 3] FIG. 11 is a diagram for explaining screens displayed on a plurality of information terminals. [Figure 4] FIG. 4 is a diagram showing feature point group data corresponding to a subject in a first space. [Diagram 5] 1 is a diagram showing an overview of a process in which an information terminal acquires spatial data corresponding to a space to which the information terminal belongs. [Figure 6] FIG. 13 is a diagram illustrating an overview of a process for updating spatial data. [Figure 7] FIG. 2 is a diagram showing a configuration of an information terminal. [Figure 8] FIG. 1 is a diagram illustrating a configuration of an information processing device. [Figure 9] FIG. 2 is a diagram showing the configuration of a computer included in a computer group. [Figure 10] 11 is a sequence diagram showing a flow until a plurality of information terminals belonging to the same space acquire spatial data. FIG. [Figure 11] 11 is a sequence diagram showing a flow until a plurality of information terminals belonging to different spaces acquire spatial data. FIG. [Figure 12] FIG. 11 is a sequence diagram showing a process flow in which a plurality of information terminals belonging to the same space display an AR image. [Figure 13] 11 is a sequence diagram showing a processing flow in which the information terminal registers feature point group data as spatial data. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] [Outline of Information Processing System S] (System Configuration) 1 is a diagram showing an overview of an information processing system S. The information processing system S is a system for displaying an augmented reality image (hereinafter, referred to as an "AR image") on a plurality of information terminals 1. The information processing system S includes a plurality of information terminals 1, an information processing device 2, a computer group 3, and a storage device 4.

[0016] The information terminal 1 is a terminal having an imaging function, and is, for example, a smartphone, a tablet, or a glasses-type terminal. A user who uses the information terminal 1 can move to any location. FIG. 1 shows a state in which a plurality of information terminals 1a (1a-1 to 1a-m, m is a natural number) exist in a first space, and a plurality of information terminals 1b (1b-1 to 1b-n, n is a natural number) exist in a second space. The first space and the second space correspond to areas set to display the same AR image on the information terminals 1 of users in the same space, such as, for example, a room separated by a wall, a floor in a building, a building, or a town. A plurality of information terminals 1 that display the same AR image in a space of the same coordinate system constitute a group of information terminals belonging to the space.

[0017] The information processing device 2 is a server that provides various data to the information terminal 1. The information processing device 2 provides, for example, spatial data used by the information terminal 1 to identify its own position and orientation, and AR image data to be displayed on the information terminal 1. The spatial data includes a plurality of registered feature point group data created in advance, which is composed of feature point groups indicating features of one or more subjects that are visible when viewing the space from a plurality of positions in the space. In the spatial data, coordinate data indicating the position and orientation at which the space is viewed is associated with the plurality of registered feature point group data. In addition, pixel values ​​indicating color (R value, G value, B value) and coordinates indicating the position in the space may be associated with the plurality of feature points included in the registered feature point group data.

[0018] The computer group 3 executes a matching process for identifying the space to which the information terminal 1 belongs. The computer group 3 acquires captured image data captured by the information terminal 1 and generated therefrom, or feature point group data indicating feature points of the subject included in the captured image data, and performs a matching process for the acquired captured image data or feature point group data with a plurality of space data constituted by a plurality of registered feature point group data corresponding to the plurality of spaces, thereby identifying the space to which the information terminal 1 belongs. The plurality of feature points included in the feature point group data may be associated with pixel values ​​indicating color (R value, G value, B value).

[0019] The computer group 3 selects spatial data including a plurality of registered feature point group data corresponding to the space to which the information terminal 1 belongs, and provides the selected spatial data to the information terminal 1. Although details will be described later, the computer group 3 has a plurality of computers 30 that store spatial data corresponding to each of the plurality of spaces, and the plurality of computers 30 execute matching processing in parallel to identify the space to which the information terminal 1 belongs in a short period of time.

[0020] The storage device 4 is a device that stores spatial data corresponding to various spaces. For example, the storage device 4 stores spatial data corresponding to many spaces all over the world, and provides each computer 30 with spatial data corresponding to each of the computers 30 in the computer group 3 out of the multiple spatial data.

[0021] (Basic operation) Fig. 2 is a diagram for explaining the basic operation of the information processing system S. Fig. 2 shows a flow of operations in which, in response to an operation by a user U1 using an information terminal 1a-1 present in a first space, the same AR image is displayed superimposed on an image of the space viewed by a user U1 of the information terminal 1a-1 and a user U2 of the information terminal 1a-2 present in the same first space. Since the information terminals 1a-1 and 1a-2 face different directions at different positions in the first space, the same AR image is displayed in different directions at different positions on the screens being displayed by each of them.

[0022] The image of the space visually recognized by the user U1 and the user U2 is, for example, a captured image generated by the information terminal 1 capturing an image of the space in which the user U1 and the user U2 are present. When the information terminal 1 is a glasses-type terminal, the image of the space visually recognized by the user U1 and the user U2 may be an image seen by the eyes of the user U1 and the user U2 due to light incident on the eyes of the user U1 and the user U2 through a transparent lens of the glasses-type terminal. In the following description, a case where the image of the space visually recognized by the user U1 and the user U2 is a captured image will be mainly described as an example, but the following configuration and processing are also applied when the image of the space visually recognized by the user U1 and the user U2 is an image seen by the eyes of the user U1 and the user U2 through a lens.

[0023] The information terminal 1 acquires from the information processing device 2 spatial data indicating the positions of a feature point group of a subject visible to the user U at multiple positions in the space in which the user U exists, and stores the spatial data corresponding to the specified space at least while the user U is in the specified space. The feature point group is a collection of multiple points indicating the characteristics of the shape of the subject, and is composed of multiple points on the contour line of the subject included in the captured image data generated by capturing an image of the space. In the spatial data, a combination of a position and a direction in a three-dimensional space defined by latitude, longitude, altitude, and direction is associated with the feature point group data.

[0024] Fig. 3 is a diagram for explaining screens displayed on multiple information terminals 1. Fig. 3(a) is a diagram showing a space photographed by information terminal 1a-1 and information terminal 1a-2 from above. The left-right direction in Fig. 2(a) is the X direction, the up-down direction is the Y direction, and the depth direction is the Z direction.

[0025] In Fig. 3, it is assumed that a user U1 using information terminal 1a-1 and a user U2 using information terminal 1a-2 are located at the positions shown in Fig. 3(a). Subjects T1 to T4 in Fig. 3 are subjects existing in a first space to which information terminal 1a-1 and information terminal 1a-2 belong. Image A (person), image B (bird), and image C (arrow) in Fig. 3 are AR images displayed on information terminal 1a-1 and information terminal 1a-2.

[0026] FIG. 3(b) shows an example of a screen displayed on the information terminal 1a-1. FIG. 3(b) shows a state in which the user U1 performs an operation of shooting an image C (arrow) from an image A (person) toward an image B (bird) with the hand H of the user U1. FIG. 3(c) shows an image displayed on the information terminal 1a-2 when the operation shown in FIG. 3(b) is performed on the information terminal 1a-1. In FIG. 3(c), the image C is displayed in a different direction at a different position from that in FIG. 3(b). By the information terminals 1a-1 and 1a-2 displaying such images, the users U1 and U2 can view the same AR image from different directions on the screens of the information terminals 1 that they use, superimposed on subjects that are visible at different positions in the same space.

[0027] Hereinafter, a process flow in which the information terminal 1a-1 and the information terminal 1a-2 display the image shown in Fig. 3 will be described with reference to Fig. 2. The information terminal 1 creates feature point group data based on captured image data generated by the built-in camera by executing application software such as a game or a tourist guide that has a function of displaying AR images.

[0028] Fig. 4 is a diagram showing feature point group data corresponding to subjects T1 to T4 in the first space shown in Fig. 3. Fig. 4(a) shows the feature point group data of subjects T1 to T4 photographed by information terminal 1a-1, and Fig. 4(b) shows the feature point group data of subjects T1 to T4 photographed by information terminal 1a-2. In this way, the positions of the multiple feature points included in the feature point group data differ depending on the position and orientation at which information terminal 1 photographs.

[0029] The information terminal 1 compares the created feature point group data with a plurality of registered feature point group data included in pre-stored spatial data, and identifies, from the plurality of registered feature point group data, the registered feature point group data that is most similar to the created feature point group data. The information terminal 1 identifies the position and orientation of the information terminal 1 by identifying the position and orientation corresponding to the identified registered feature point group data.

[0030] When the information terminal 1a-1 receives a user operation that triggers display of an AR image, it notifies the information processing device 2 of operation data indicating the content of the operation and position coordinate data indicating the identified position and orientation, as shown in Fig. 2. The information terminal 1a-1 acquires an AR image corresponding to the position and orientation from the information processing device 2, and displays the acquired AR image, for example, as shown in Fig. 3(b).

[0031] The information processing device 2 has multiple memory databases corresponding to multiple spaces, and when the position coordinate data notified from the information terminal 1a-1 is written to the memory database corresponding to the space to which the information terminal 1a-1 belongs, the information processing device 2 transmits an AR image and display coordinate data indicating the position to display the AR image and the direction to move the AR image to the information terminal 1a-2, which is another information terminal 1 belonging to the same space. The information terminal 1a-2 displays the AR image corresponding to the position and direction indicated by the received display coordinate data, for example, as shown in FIG. 3(c).

[0032] In a system that displays AR images as in this embodiment, a high-speed response is required, so the database is preferably an in-memory database that is constructed on the main memory and can be accessed faster than a database stored in a disk storage mechanism. In the following explanation, a case where the database is an in-memory database is exemplified.

[0033] (Providing spatial data) 5 is a diagram showing an overview of a process in which the information terminal 1 acquires spatial data corresponding to the space to which the information terminal 1 belongs. The information terminal 1 acquires spatial data when triggered by the start of a specific application software, the user U logging in to a specific site, or the information terminal 1 moving to a specific location.

[0034] For example, when the information terminal 1 detects that a specific application software has been started, it creates feature point group data based on captured image data captured and generated by a built-in camera. The information terminal 1 transmits start-up information indicating that the application software has been started and the created feature point group data to the information processing device 2. In response to receiving the start-up information, the information processing device 2 transmits the received feature point group data to the computer group 3.

[0035] The computer group 3 inputs the received feature point group data to a plurality of computers 30, and the plurality of computers 30 execute a matching process. Among the plurality of computers 30, a computer 30 that detects that the pattern of the feature point group indicated by the received feature point group data matches the pattern of the registered feature point group indicated by the spatial data stored in itself notifies the information processing device 2 of the match and transmits the spatial data stored in itself to the information processing device 2. The information processing device 2 transmits the received spatial data to the information terminal 1.

[0036] The information terminal 1 may transmit approximate position data indicating an approximate position detected by the GPS receiver together with startup information to the information processing device 2. In this case, the information processing device 2 transmits feature point group data to the computer 30 corresponding to the space including the position indicated by the received approximate position data, thereby reducing the number of computers 30 that execute the matching process, and thus reducing the load on the computer group 3.

[0037] (Spatial data update) Objects such as buildings, goods, and plants in a space change over time. Therefore, the spatial data stored in the storage device 4 does not necessarily correspond to the latest state of the space. In addition, there may be spaces for which no spatial data has been created. However, it is not easy to collect spatial data every time the state of the space changes. Therefore, the information processing system S uses a large amount of feature point group data created by the information terminals 1 to update the spatial data to the latest state.

[0038] 6 is a diagram showing an overview of a process for updating spatial data. When the information terminal 1 does not include registered feature point group data that matches the feature point group data created by itself among a plurality of registered feature point group data included in the spatial data acquired via the information processing device 2, the information terminal 1 determines that there is an error in the spatial data or that there is no spatial data for the space to which the information terminal 1 belongs. In this case, the information terminal 1 transmits an update request to the information processing device 2, together with the feature point group data created by itself and position data indicating its own position, requesting that the registered feature point group data be updated. The position data is, for example, data indicating latitude, longitude, altitude, and direction.

[0039] It is desirable for the position data to be as accurate as possible. Therefore, the information terminal 1 calculates a position corresponding to the feature point group data to be transmitted together with the update request, for example, based on the movement distance and movement direction from the position where the registered feature point group data included in the spatial data most recently matched with the feature point group data created by the information terminal 1. The information terminal 1 may transmit the update request on the condition that the distance from the position where the registered feature point group data included in the spatial data most recently matched with the feature point group data created by the information terminal 1 is within a predetermined threshold. The predetermined threshold is determined based on at least one of the accuracy of the position indicated by the spatial data or the accuracy with which the information terminal 1 can identify the movement distance and movement direction.

[0040] When the information processing device 2 receives the update request, it transmits the received feature point cloud data and position data to the storage device 4. When the information processing device 2 receives feature point cloud data corresponding to the same position and orientation from multiple information terminals 1, it transmits feature point cloud data that is considered to have higher accuracy. Feature point cloud data that is considered to have higher accuracy is feature point cloud data that includes a larger number of feature points, feature point cloud data transmitted from an information terminal 1 using a sensor with higher accuracy for identifying the position, etc.

[0041] When the storage device 4 receives the feature point group data and position data from the information terminal 1, it stores the feature point group data as registered feature point group data included in the spatial data in association with the position and orientation (i.e., latitude, longitude, altitude and direction) indicated by the received position data. By the storage device 4 updating the registered feature point group data included in the spatial data in this manner, the information terminal 1 can specify its own position and orientation using the spatial data corresponding to the latest space, and can display an AR image in an appropriate position.

[0042] [Components] (Information terminal configuration) 7 is a diagram showing the configuration of the information terminal 1. The information terminal 1 has a communication unit 11, an operation unit 12, a position detection unit 13, an imaging unit 14, a display unit 15, a storage unit 16, and a control unit 17. The control unit 17 has a transmission processing unit 171, a reception processing unit 172, an operation acceptance unit 173, an approximate position identification unit 174, an image processing unit 175, a detailed position identification unit 176, a display processing unit 177, and a spatial data acquisition unit 178.

[0043] The communication unit 11 has a communication interface for connecting to a communication network. The communication unit 11 transmits and receives various data to and from the information processing device 2 via a wireless communication network such as a mobile wireless communication network or Wi-Fi (registered trademark). The communication unit 11 transmits data input from a transmission processing unit 171, and inputs data received via the communication network to a reception processing unit 172.

[0044] The operation unit 12 has a device for detecting an operation of the user U of the information terminal 1. The operation unit 12 has, for example, a touch panel, a trackball, or a mouse. When the operation unit 12 is a touch panel, the operation unit 12 is provided superimposed on the display unit 15. The operation unit 12 notifies the operation receiving unit 173 of operation data indicating at least either the position where the operation was performed or the amount of operation.

[0045] The position detection unit 13 has a device for detecting the approximate position of the information terminal 1. The position detection unit 13 has, for example, a GPS (Global Positioning System) receiver, and notifies the approximate position identification unit 174 of position data indicating latitude and longitude detected based on radio waves received from a GPS satellite. The position detection unit 13 has at least one of an altimeter, a compass, or a gyro sensor, and may notify the approximate position identification unit 174 of the altitude of the information terminal 1 and the direction in which the information terminal 1 is facing as position data.

[0046] The imaging unit 14 is a device, such as a camera, that captures an image of a subject around the information terminal 1 to generate captured image data. The imaging unit 14 generates captured image data by capturing an image of at least a partial area of ​​the space to which the information terminal 1 belongs. The imaging unit 14 may generate captured image data continuously at a predetermined time interval (frame rate). The imaging unit 14 may also be a stereo camera that has multiple image sensors and generates distance data indicating the distance to the subject. In this case, the feature point group data may include information indicating the distance from the information terminal 1 to the subject corresponding to each feature point.

[0047] The display unit 15 is a device for displaying various information, such as a display. For example, the display unit 15 displays a screen for the user U of the information terminal 1 to operate, or displays an image based on captured image data generated by the imaging unit 14. The display unit 15 also displays an AR image provided by the information processing device 2 superimposed on an image of the space visually recognized by the user U (for example, a captured image).

[0048] The storage unit 16 is a terminal storage unit having storage media such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The storage unit 16 stores a program executed by the control unit 17. The storage unit 16 also temporarily stores image data displayed by the display unit 15, such as captured image data and AR image data. The storage unit 16 stores a plurality of feature point group data indicating feature point groups of a subject included in a plurality of images created by capturing an image of a space from a plurality of positions and orientations in the space, in association with the positions and orientations in the space.

[0049] The control unit 17 has, for example, a CPU (Central Processing Unit). The control unit 17 executes a program stored in the storage unit 16, thereby functioning as a transmission processing unit 171, a reception processing unit 172, an operation receiving unit 173, an approximate position specifying unit 174, an image processing unit 175, a detailed position specifying unit 176, a display processing unit 177, and a spatial data acquisition unit 178.

[0050] The transmission processing unit 171 transmits various data to the information processing device 2 or other external devices via the communication unit 11. For example, the transmission processing unit 171 transmits operation data received from the operation reception unit 173 to the information processing device 2. The operation data includes operation content data indicating the content of the operation, display coordinate data indicating the position of the operation (i.e., the position where the AR image is displayed), and display orientation data indicating the orientation in which the AR image is displayed. The operation data may further include speed data indicating the operation speed of the user U corresponding to the speed at which the AR image is moved.

[0051] When the user U performs an operation to display an AR image on the display unit 15, the transmission processing unit 171 transmits to the information processing device 2, together with operation content data, display coordinate data indicating coordinates in space when the AR image is displayed on the display unit 15, display orientation data, and speed data. In addition, the transmission processing unit 171 transmits the approximate position data received from the approximate position identification unit 174 to the information processing device 2.

[0052] In order to transmit and receive data asynchronously in real time with the in-memory database of the information processing device 2, the transmission processing unit 171 may transmit the operation data using a Pub / Sub message for accessing the in-memory database.

[0053] Furthermore, the transmission processing unit 171 acquires feature point group data configured of a plurality of feature points of the subject included in the captured image data from the image processing unit 175, and transmits the acquired feature point group data to the information processing device 2. The transmission processing unit 171 may transmit the captured image data received from the image processing unit 175 to the information processing device 2 instead of or together with the feature point group data. The transmission processing unit 171 writes the feature point group data, for example, by transmitting a Pub message to an in-memory database corresponding to the space to which the approximate position identified by the approximate position identifying unit 174 belongs, among a plurality of in-memory databases stored in the information processing device 2.

[0054] Furthermore, when there is no registered feature point group data whose difference from the feature point group data is equal to or less than a threshold value among the multiple registered feature point group data stored in the memory unit 16, the transmission processing unit 171 transmits the feature point group data acquired from the image processing unit 175 to the information processing device 2 together with an update request for updating the registered feature point group data.

[0055] The reception processing unit 172 receives various types of data from the information processing device 2 or other external devices via the communication unit 11. The reception processing unit 172 receives, for example, display coordinate data transmitted to the information processing device 2 by another information terminal 1 belonging to the same space.

[0056] When the information terminal 1a-1 transmits display coordinate data to the information processing device 2 in response to an operation by a user U1 using the information terminal 1a-1 to display an AR image, the reception processing unit 172 of the information terminal 1a-2 receives the display coordinate data transmitted by a coordinate transmission unit 232 of the information processing device 2, which will be described later. The reception processing unit 172 inputs the received display coordinate data to the detailed position identification unit 176. In addition, the reception processing unit 172 receives spatial data corresponding to the space to which the information terminal 1 belongs from the information processing device 2. The reception processing unit 172 inputs the received spatial data to the spatial data acquisition unit 178.

[0057] The operation accepting unit 173 identifies the content of the operation performed by the user U based on the operation data notified from the operation unit 12. The operation accepting unit 173 creates operation content data indicating the type of AR image to be displayed based on the accepted operation. The operation accepting unit 173 notifies the information processing device 2 of the content of the operation by inputting the created operation content data to the transmission processing unit 171. For example, in response to accepting an operation for displaying an AR image on the display unit 15 superimposed on an image of space (for example, a captured image), the operation accepting unit 173 creates display coordinate data indicating coordinates in space when the AR image is displayed on the display unit 15.

[0058] The operation receiving unit 173 may acquire detailed position data indicating the detailed position identified by the detailed position identifying unit 176, and create display coordinate data based on the position and orientation of the information terminal 1 indicated by the acquired detailed position data and the position operated by the user U on the operation unit 12. In other words, the operation receiving unit 173 may create display coordinate data based on the relationship between the position and orientation identified by the detailed position identifying unit 176 and the position of the operation accepted by the operation receiving unit 173.

[0059] In the example shown in FIG. 3(b), the operation acceptance unit 173 creates display coordinate data including the X coordinate and Z coordinate of the position where the hand H touches the operation unit 12. When the position of the hand H moves, the operation acceptance unit 173 creates display coordinate data at a predetermined time interval. The operation acceptance unit 173 may create display coordinate data including speed data indicating the speed at which the hand H moves based on the display coordinate data created at multiple times. When the Y coordinate of the subject at the position where the hand H touches in FIG. 3(b) is known, such as when the imaging unit 14 is a stereo camera that can generate distance data, the operation acceptance unit 173 may create display coordinate data including the Y coordinate. The operation acceptance unit 173 inputs the created display coordinate data to the transmission processing unit 171.

[0060] The operation receiving unit 173 may create display orientation data indicating the orientation of the AR image when the AR image is displayed on the display unit 15, based on the relationship between the position and orientation of the information terminal 1 identified by the detailed position identifying unit 176 and the position of the operation accepted by the operation accepting unit 173. The operation accepting unit 173 inputs the display orientation data to the transmission processing unit 171.

[0061] The approximate position specifying unit 174 generates approximate position data indicating the approximate position of the information terminal 1 based on the received radio waves. The approximate position specifying unit 174 specifies the approximate position of the information terminal 1 based on the position data input from the position detection unit 13. The approximate position data is data indicating a position with an accuracy corresponding to the accuracy of GPS, for example, position data with an error of 10 m or more.

[0062] The approximate position identification unit 174 inputs the identified position data to the transmission processing unit 171, thereby notifying the information processing device 2 of the approximate position of the information terminal 1. The approximate position identification unit 174 may periodically notify the information processing device 2 of the approximate position data, or may notify the information processing device 2 of the approximate position data when the user U performs a predetermined operation or when it becomes necessary to obtain registered feature point group data.

[0063] The image processing unit 175 creates feature point group data consisting of multiple feature points of the subject included in the captured image data. Specifically, the image processing unit 175 first identifies pixels in the captured image data whose pixel value change amount between adjacent pixels is equal to or greater than a threshold, thereby identifying the contour line of the subject and the contour line of uneven parts of the subject. The image processing unit 175 creates feature point group data consisting of multiple feature points on the identified contour line.

[0064] 4, the image processing unit 175 may create feature point group data by identifying points where the direction of a contour line changes as feature points. The image processing unit 175 creates feature point group data every time new captured image data is input from the imaging unit 14. The image processing unit 175 inputs the created feature point group data to the detailed position identifying unit 176. The image processing unit 175 inputs the feature point group data to the transmission processing unit 171 when it is necessary to acquire spatial data from the information processing device 2 (i.e. when it is necessary to identify the space to which the information terminal 1 belongs).

[0065] Furthermore, when the image processing unit 175 receives a notification that the detailed position identification unit 176 cannot identify the position and orientation of the information terminal 1 based on the registered feature point cloud data acquired from the information processing device 2, it inputs the created feature point cloud data to the transmission processing unit 171 in order to register the feature point cloud data in the storage device 4, and instructs the transmission processing unit 171 to transmit an update request to the information processing device 2 to update the registered feature point cloud data.

[0066] The detailed position identifying unit 176 identifies a position and orientation corresponding to registered feature point group data that is most similar to the feature point group data input from the image processing unit 175, among the multiple registered feature point group data received from the information processing device 2. The detailed position identifying unit 176 identifies a position and orientation corresponding to registered feature point group data that is most similar to feature point group data indicating the feature point group of the subject included in the captured image data, among the multiple registered feature point group data stored in the storage unit 16.

[0067] Specifically, the detailed position specifying unit 176 first compares the feature point group data input from the image processing unit 175 with a plurality of registered feature point group data included in the spatial data acquired from the information processing device 2 and stored in the storage unit 16. The detailed position specifying unit 176 calculates a correlation value between the feature point group data input from the image processing unit 175 and each of the plurality of registered feature point group data included in the spatial data, and specifies the registered feature point group data for which the calculated correlation value is maximum.

[0068] The detailed position specifying unit 176 specifies the position and orientation of the information terminal 1 based on coordinate data (e.g., data indicating latitude, longitude, altitude, and direction) associated with the specified registered feature point group data. In this way, the detailed position specifying unit 176 can specify the position and orientation of the information terminal 1 with higher accuracy than, for example, the approximate position specifying unit 174. The detailed position specifying unit 176 specifies the position and orientation of the information terminal 1 with an accuracy of, for example, 1 mm. The detailed position specifying unit 176 notifies the reception processing unit 172 and the display processing unit 177 of detailed position data indicating the specified position and orientation.

[0069] The display processing unit 177 displays various information or images on the display unit 15. For example, the display processing unit 177 displays an image based on captured image data generated by the imaging unit 14 capturing an image of the space to which the information terminal 1 belongs on the display unit 15. The display processing unit 177 also determines the position and orientation at which the AR image is to be displayed based on the display coordinate data. The display processing unit 177 displays the AR image by superimposing it on an image of the space visually recognized by the user U (for example, an image based on captured image data generated by the imaging unit 14 capturing an image of the space to which the information terminal 1 belongs) at a position on the display unit 15 corresponding to the coordinates in the space indicated by the display coordinate data.

[0070] When the user U performs an operation to display an AR image, the display processing unit 177 creates display coordinate data indicating the position and orientation at which the AR image is to be displayed, based on the detailed position data input from the detailed position identification unit 176 and the operation position data notified from the operation receiving unit 173. When the user U of another information terminal 1 belonging to the same space as the information terminal 1 performs an operation to display an AR image, the display processing unit 177 acquires, from the information processing device 2, display coordinate data used to display the AR image corresponding to the operation.

[0071] 2, when information terminal 1a-1 and information terminal 1a-2 exist in the first space, display processing unit 177 of information terminal 1a-1 displays an AR image at a position and orientation based on display coordinate data notified from operation receiving unit 173. Meanwhile, display processing unit 177 of information terminal 1a-2 displays an AR image at a position and orientation based on display coordinate data notified from information processing device 2. By operating display processing units 177 of information terminals 1a-1 and 1a-2 in this manner, user U1 using information terminal 1a-1 and user U2 using information terminal 1a-2 can simultaneously view AR images at different positions and orientations depending on the position and orientation of information terminal 1.

[0072] The spatial data acquisition unit 178 acquires spatial data including registered feature point group data from the information processing device 2 via the communication unit 11 and the reception processing unit 172. In response to the operation acceptance unit 173 accepting an operation for acquiring a plurality of registered feature point group data from the information processing device (for example, an operation for starting up a predetermined application software, an operation for logging in to a predetermined site), the spatial data acquisition unit 178 acquires spatial data including a plurality of registered feature point group data corresponding to the space to which the information terminal 1 belongs from the information processing device, and stores the acquired spatial data in the storage unit 16.

[0073] The spatial data acquisition unit 178 notifies the detailed position identification unit 176 of the acquired plurality of registered feature point group data. The spatial data acquisition unit 178 notifies the detailed position identification unit 176 of the plurality of registered feature point group data via the storage unit 16, for example, by storing the acquired plurality of registered feature point group data in the storage unit 16.

[0074] (Configuration of information processing device 2) 8 is a diagram showing a configuration of the information processing device 2. The information processing device 2 has a communication unit 21, a storage unit 22, and a control unit 23. The control unit 23 has a coordinate receiving unit 231, a coordinate transmitting unit 232, a terminal data receiving unit 233, a feature point group transmitting unit 234, an information receiving unit 235, a spatial data transmitting unit 236, a spatial data management unit 237, and a number management unit 238.

[0075] The communication unit 21 has a communication interface for connecting to a communication network. The communication unit 21 transmits and receives various data between the information terminal 1, the computer group 3, and the storage device 4.

[0076] The storage unit 22 is a device storage unit having a ROM, a RAM, and an SSD (Solid State Drive). The storage unit 22 stores a program executed by the control unit 23. The storage unit 22 also stores a first space database 221 and a second space database 222 as a plurality of in-memory databases corresponding to a plurality of spaces. The storage unit 22 stores, for example, a plurality of in-memory databases (for example, the first space database 221 and the second space database 222) that store display coordinate data or feature point group data corresponding to each of a plurality of spaces (for example, the first space and the second space).

[0077] As described above, the in-memory database is a database constructed on the main memory, and can be accessed at a higher speed than a database stored in a disk storage mechanism. In Fig. 8, the storage unit 22 illustrates the first space database 221 and the second space database 222, but the storage unit 22 stores databases corresponding to more spaces.

[0078] The storage unit 22 stores a first information terminal and a second information terminal belonging to a space of the same coordinate system as multiple information terminals 1 displaying an AR image in the same space. The first information terminal is the information terminal 1 on which the user U performs an operation to display the AR image, and is the information terminal 1a-1 in the example shown in FIG. 2. The second information terminal is another information terminal 1 belonging to the same space as the first information terminal, and is the information terminal 1a-2 in the example shown in FIG. 2. The storage unit 22 stores multiple pieces of terminal identification information for identifying the multiple information terminals 1 included in the space, in association with each of the multiple spaces.

[0079] Specifically, in the in-memory database stored in the storage unit 22, space identification information for identifying a space to which each of the information terminals 1 belongs is registered in association with terminal identification information for identifying each of the multiple information terminals 1. Also, in the in-memory database, AR image identification information for identifying an AR image to be displayed on the information terminal 1 belonging to the space corresponding to the space identification information, and display coordinate data indicating the position and orientation at which the AR image is to be displayed are registered in association with the space identification information.

[0080] The multiple in-memory databases may store feature point group data received from an information terminal 1 belonging to the respective corresponding spaces in order to transfer the feature point group data to the computer group 3. When the first space database 221 corresponding to the first space receives the feature point group data from one or more information terminals 1 belonging to the first space, it transfers the received feature point group data to a computer 30 that analyzes whether or not the feature point group data matches registered feature point group data corresponding to the first space. The in-memory database may relay the feature point group data between the information terminal 1 and the computer group 3 based on a Pub / Sub message transmitted from the information terminal 1 or the computer group 3 without going through the control unit 23.

[0081] The control unit 23 has, for example, a CPU. The control unit 23 executes the programs stored in the storage unit 22 to function as a coordinate receiving unit 231, a coordinate transmitting unit 232, a terminal data receiving unit 233, a feature point group transmitting unit 234, an information receiving unit 235, a spatial data transmitting unit 236, a spatial data managing unit 237, and a number managing unit 238.

[0082] The coordinate receiving unit 231 receives operation content data and display coordinate data transmitted by the information terminal 1 in response to an operation by a user U who uses the information terminal 1, and registers the received operation content data and display coordinate data in an in-memory database. The coordinate receiving unit 231 registers the display coordinate data in an in-memory database corresponding to the space including the information terminal 1 that transmitted the display coordinate data. When the coordinate receiving unit 231 receives display coordinate data from the information terminal 1a-1 belonging to the first space, for example, the coordinate receiving unit 231 stores the display coordinate data in the first space database 221.

[0083] In response to the display coordinate data being registered in the in-memory database by the coordinate receiving unit 231, the coordinate transmitting unit 232 transmits the display coordinate data to a second information terminal different from the first information terminal among the multiple information terminals 1. The coordinate transmitting unit 232 transmits the display coordinate data registered in the in-memory database corresponding to the space including the second information terminal to the second information terminal.

[0084] The coordinate transmission unit 232 transmits the display coordinate data to a second information terminal having second terminal identification information associated with a space corresponding to the first terminal identification information of the first information terminal that transmitted the display coordinate data, by referring to the storage unit 22. The coordinate transmission unit 232 may transmit the display orientation data to the second information terminal in response to the display orientation data transmitted by the first information terminal being registered in the in-memory database. As a result, a plurality of information terminals 1 belonging to the same space share the position and display orientation at which the AR image is displayed.

[0085] In addition, in response to the operation content data received from the first information terminal being registered in the in-memory database, the coordinate transmission unit 232 transmits data corresponding to the second AR image corresponding to the operation content data or data for identifying the second AR image to the second information terminal. The second AR image is an image corresponding to the first AR image, showing the same object as the first AR image, but with a different orientation or size. This allows multiple information terminals 1 belonging to the same space to share the same AR image.

[0086] The coordinate transmission unit 232 transmits the display coordinate data and the AR image stored in the first space database 221 to the information terminal 1a-2. When the coordinate transmission unit 232 has already transmitted an AR image to the information processing device 2, the coordinate transmission unit 232 may transmit AR image identification information for identifying an AR image to be displayed on the information terminal 1a-2 from among the multiple AR images already transmitted, instead of transmitting the AR image.

[0087] The terminal data receiving unit 233 receives feature point group data from the information terminal 1 via the communication unit 21. Furthermore, the terminal data receiving unit 233 receives approximate position data indicating the approximate position of the information terminal 1 from the information terminal 1 that transmitted the feature point group data. The terminal data receiving unit 233 notifies the feature point group transmission unit 234 of the received feature point group data and approximate position data. The terminal data receiving unit 233 may store the feature point group data and approximate position data in the storage unit 22, and notify the feature point group transmission unit 234 of the feature point group data and approximate position data via the storage unit 22.

[0088] The feature point group transmission unit 234 transmits the feature point group data received from the information terminal 1 to the computer group 3. For example, when the terminal data receiving unit 233 receives the feature point group data transmitted by the information terminal 1 together with startup information as shown in Fig. 5, the feature point group transmission unit 234 transmits the feature point group data to the computer group 3 together with a message requesting spatial data.

[0089] When the feature point group transmitting unit 234 acquires the approximate position data from the terminal data receiving unit 233, it may select one or more computers 30 corresponding to a space including the approximate position indicated by the approximate position data from the multiple computers 30 included in the computer group 3, and transmit the feature point group data to the selected one or more computers 30. In other words, the feature point group transmitting unit 234 may transmit the feature point group data to a computer 30 associated with a space corresponding to data to which the approximate position indicated by the approximate position data transmitted by the information terminal 1 together with the feature point group data belongs.

[0090] The feature point group transmission unit 234 may relay the feature point group data between the information terminal 1 and the computer group 3 via an in-memory database corresponding to each of the multiple spaces. The feature point group transmission unit 234 transmits the feature point group data to the computer 30 associated with the space corresponding to the in-memory database in which the feature point group data has been written, among the multiple in-memory databases.

[0091] The information receiving unit 235 receives, from the computer 30 that has identified the space to which the information terminal 1 belongs, among the one or more computers 30 to which the feature point group transmitting unit 234 has transmitted the feature point group data, predetermined information indicating that the space to which the information terminal 1 belongs has been identified. Specifically, the information receiving unit 235 receives, from one computer 30 among the one or more computers 30 to which the feature point group data has been transmitted, predetermined information indicating that the correlation value between any of the multiple registered feature point group data stored in the computer 30 and the feature point group data transmitted by the feature point group transmitting unit 234 is equal to or greater than a threshold value. In other words, the information receiving unit 235 receives, from the computer 30, predetermined information indicating that the feature point group data transmitted by the feature point group transmitting unit 234 is equivalent to the registered feature point group data of a subject included in the space in which the computer 30 is responsible for matching processing.

[0092] The information receiving unit 235 may acquire, together with predetermined information, spatial data including a plurality of registered feature point group data corresponding to the space for which the computer 30 is in charge of the matching process. The information receiving unit 235 inputs the acquired spatial data to the spatial data transmitting unit 236 together with the terminal identification information of the information terminal 1 that created the feature point group data that the feature point group transmitting unit 234 transmitted to the computer group 3. The information receiving unit 235 may also acquire space identification information for identifying the space for which the computer 30 is in charge of the matching process, and store the space identification information in the storage unit 22 in association with the terminal identification information of the information terminal 1 that created the feature point group data that the feature point group transmitting unit 234 transmitted to the computer group 3.

[0093] The spatial data transmitting unit 236 transmits a plurality of registered feature point cloud data included in the spatial data corresponding to the computer 30 that transmitted the predetermined information to the information terminal 1. The spatial data transmitting unit 236 transmits the plurality of registered feature point cloud data to the information terminal 1 corresponding to the terminal identification information acquired from the information receiving unit 235.

[0094] The spatial data management unit 237 manages the spatial data stored in the computer group 3 or the storage device 4 to keep it up to date. The spatial data management unit 237 keeps the spatial data up to date, for example, by storing the feature point group data received from the information terminal 1 in the storage device 4 in association with a position in space.

[0095] When the spatial data management unit 237 determines that the feature point group data received from the information terminal 1 is the latest feature point group data, it stores the feature point group data received from the information terminal 1 as registered feature point group data in the storage device 4. When the spatial data management unit 237 receives feature point group data together with an update request requesting to update the registered feature point group data from, for example, a plurality of information terminals 1, it determines that the feature point group data based on captured image data generated at a later point in time is the latest feature point group data by, for example, referring to the timestamp included in the feature point group data.

[0096] When the device reception processing unit receives multiple pieces of feature point cloud data from multiple information terminals 1 that have transmitted detailed position data indicating the same position and orientation, the spatial data management unit 237 may store the feature point cloud data with a larger amount of data as registered feature point cloud data in the storage device 4. The amount of data is, for example, the number of point clouds included in the feature point cloud data. It is considered that the larger the number of point clouds is, the more detailed features of the subject belonging to the space are expressed in the feature point cloud data, so by the spatial data management unit 237 registering such feature point cloud data, the accuracy of the information terminal 1 identifying the position and orientation using the registered feature point cloud data is improved.

[0097] The spatial data management unit 237 may store the feature point group data received from the information terminal 1 as registered feature point group data in the storage device 4 when the feature point group data associated with the position and orientation indicated by the detailed position data transmitted by the information terminal 1 together with the feature point group data is not stored in the storage device 4. The storage unit 22 stores latitude, longitude, altitude, and orientation corresponding to each of the multiple registered feature point group data stored in the storage device 4, and the spatial data management unit 237 stores the feature point group data in the storage device 4 when the latitude, longitude, altitude, and orientation indicated by the detailed position data are not stored in the storage device 22.

[0098] When registered feature point group data corresponding to the position and orientation indicated by the position data is stored in the storage device 4, the spatial data management unit 237 may store the feature point group data received from the information terminal 1 in the storage device 4 as registered feature point group data, provided that the amount of data of the feature point group data received from the information terminal 1 is greater than the amount of data of the stored registered feature point group data.

[0099] The memory unit 22 stores the data amount of each of the multiple registered feature point cloud data stored in the storage device 4 together with the latitude, longitude, altitude and direction, and the spatial data management unit 237 compares the data amount of the feature point cloud data transmitted from the information terminal 1 with the data amount of the registered feature point cloud data stored in the memory unit 22 in association with the position and orientation indicated by the detailed position data transmitted from the information terminal 1. If, as a result of the comparison, the spatial data management unit 237 finds that the data amount of the feature point cloud data received from the information terminal 1 is greater, it stores the feature point cloud data received from the information terminal 1 in the storage device 4. By the spatial data management unit 237 operating in this manner, it is possible to prevent the registered feature point cloud data from deteriorating by updating the registered feature point cloud data.

[0100] The spatial data management unit 237 may store the feature point group data in the storage device 4 on the condition that the feature point group data received from the information terminal 1 includes data indicating an object having a pre-registered shape. The object having a pre-registered shape is, for example, a newly constructed building. By operating in this manner, the spatial data management unit 237 can quickly register the registered feature point group data when it is known in advance that no registered feature point group data exists.

[0101] The number management unit 238 determines the number of the multiple computers 30 that execute the matching process for identifying the space to which the information terminal 1 belongs, based on preset conditions. The number management unit 238 changes the number of the multiple computers 30, for example, based on the amount of registered feature point cloud data stored in the storage device 4. The feature point cloud transmission unit 234 transmits the feature point cloud data received from the multiple information terminals 1 to the number of computers 30 determined by the number management unit 238. By the number management unit 238 operating in this manner, it is possible to suppress an increase in the time required for the matching process even if the amount of registered feature point cloud data increases over time.

[0102] The number management unit 238 may change the number of the multiple computers 30 based on the time required from when the feature point group transmission unit 234 transmits the feature point group data to the multiple computers 30 until it acquires the predetermined information indicating that the space to which the information terminal 1 belongs has been identified (i.e., the time required for the matching process). If the time required from when the feature point group transmission unit 234 transmits the feature point group data to the multiple computers 30 until it acquires the predetermined information is long, it is considered that the number of computers 30 is insufficient. Therefore, when the time becomes equal to or greater than a threshold, the number management unit 238 increases the number of computers 30 that execute the matching process, thereby preventing the time required for the matching process from becoming too long.

[0103] Incidentally, when the number of information terminals 1 from which the information processing device 2 receives feature point cloud data increases, the amount of feature point cloud data received by the terminal data receiving unit 233 increases. When the amount of feature point cloud data increases, the time required for matching processing increases. Therefore, the number management unit 238 may change the number of the multiple computers 30 based on the amount of feature point cloud data received by the terminal data receiving unit 233. The number management unit 238 may change the number of the multiple computers based on the number of information terminals 1 that transmit the feature point cloud data.

[0104] When the amount of feature point cloud data received by the terminal data receiving unit 233 within a predetermined time period becomes equal to or greater than a threshold value, or when the number of information terminals 1 receiving feature point cloud data within a predetermined time period increases, the number management unit 238 increases the number of computers 30 that execute the matching process, thereby preventing the time required for the matching process from becoming too long. The number management unit 238 may increase the number of computers 30 that execute the matching process when the number of information terminals 1 from which the terminal data receiving unit 233 receives feature point cloud data becomes equal to or greater than a threshold value.

[0105] When the information terminal 1 is moving, it is expected that the information terminal 1 will move to a different space, and the frequency of matching processing for identifying the space to which the information terminal 1 belongs will increase. Therefore, the number management unit 238 may change the number of the multiple computers 30 based on whether or not the approximate position data received from the information terminal 1 by the terminal data receiving unit 233 indicates that the information terminal 1 is moving.

[0106] When the number of moving information terminals 1 is determined to be equal to or greater than a predetermined number, the number of computers 30 that execute the matching process is increased, thereby preventing the time required for the matching process from becoming too long. Conversely, when the number of computers 238 is determined to be less than a predetermined number, the number of computers 30 that execute the matching process is decreased, thereby reducing the cost of using the computers 30.

[0107] The number management unit 238 may specify the moving speed of the information terminal 1 based on a plurality of approximate position data transmitted at a plurality of times, and when the number of information terminals 1 whose specified moving speed is equal to or greater than a threshold is equal to or greater than a predetermined number, may increase the number of computers 30. By operating the number management unit 238 in this manner, the number of computers 30 can be changed based on the number of information terminals 1 that are highly likely to switch spaces, and therefore the number of computers 30 that execute the matching process can be made a more appropriate number.

[0108] (Computer Group 3 Configuration) 9 is a diagram showing a configuration of a computer 30 included in the computer group 3. The communication unit 31 includes a communication unit 31, a storage unit 32, and a control unit 33. The control unit 33 includes a feature point group acquisition unit 331, a registered data acquisition unit 332, and a data analysis unit 333.

[0109] The communication unit 31 has a communication interface for connecting to a communication network. The communication unit 31 transmits and receives various data between the information processing device 2 and the storage device 4.

[0110] The storage unit 32 includes a ROM, a RAM, and an SSD. The storage unit 32 stores a program executed by the control unit 33.

[0111] The control unit 33 includes, for example, a CPU. The control unit 33 executes the programs stored in the storage unit 32 to function as a feature point group acquisition unit 331, a registered data acquisition unit 332, and a data analysis unit 333.

[0112] The feature point group acquiring unit 331 acquires feature point group data from the information processing device 2. The feature point group acquiring unit 331 inputs the acquired feature point group data to the data analyzing unit 333.

[0113] The registered data acquisition unit 332 acquires a plurality of registered feature point cloud data by reading the plurality of registered feature point cloud data corresponding to a space assigned to the computer 30 from the storage device 4 in which the registered feature point cloud data corresponding to a plurality of spaces is stored. The registered data acquisition unit 332 inputs the acquired plurality of registered feature point cloud data to the data analysis unit 333. The registered data acquisition unit 332 may store the acquired plurality of registered feature point cloud data in the storage unit 32.

[0114] In addition, the spatial identification information corresponding to the space assigned to the computer 30 is stored, for example, in the memory unit 32, and the registered data acquisition unit 332 acquires from the memory unit 4 a plurality of registered feature point group data stored in the memory unit 4 in association with the spatial identification information.

[0115] The data analysis unit 333 transmits predetermined information to the information processing device 2 when the difference between the feature point group data acquired by the feature point group acquisition unit 331 and any of the multiple registered feature point group data acquired by the registered data acquisition unit 332 corresponding to the space assigned to each computer 30 is equal to or smaller than a threshold. As described above, the predetermined information is information indicating that the feature point group data received by the registered data acquisition unit 332 is composed of a feature point group of a subject included in the space for which the computer 30 is responsible for matching processing.

[0116] [Processing flow in information processing system S] (Process to obtain spatial data) Fig. 10 is a sequence diagram showing a flow until multiple information terminals 1 belonging to the same space acquire spatial data. The sequence diagram shown in Fig. 10 starts from the point when it becomes necessary for the information terminal 1 to acquire spatial data. The point when it becomes necessary to acquire spatial data is, for example, the point when the user U starts a predetermined application software on the information terminal 1, logs into a predetermined site, moves a predetermined distance or more from the point where the previous spatial data was acquired, or a predetermined time or more has passed since the previous spatial data was acquired.

[0117] The first information terminal and the second information terminal in Fig. 10 belong to the same first space, and are, for example, the information terminal 1a-1 and the information terminal 1a-2 shown in Fig. 1. The first information terminal transmits first approximate position data indicating the position of the first information terminal and first feature point cloud data generated based on the captured image data to the information processing device 2. The information processing device 2 transmits the first feature point cloud data to a first computer that executes a matching process using registered feature point cloud data of the first space including the approximate position indicated by the received first approximate position data. Here, the information processing device 2 may transmit the first feature point cloud data to multiple first computers.

[0118] When the first computer determines that the first feature point cloud data matches any of the multiple registered feature point cloud data, it determines that the first information terminal is in the first space corresponding to the matching registered feature point cloud data, and transmits first spatial data including the multiple registered feature point cloud data corresponding to the first space to the information processing device 2. The information processing device 2 transmits the received first spatial data to the first information terminal. After that, the first information terminal identifies its own detailed location by comparing the feature point cloud data it generated with the multiple registered feature point cloud data it received.

[0119] The second information terminal operates in the same manner as the first information terminal, and transmits second approximate position data indicating the position of the second information terminal and second feature point cloud data generated based on the captured image data to the information processing device 2. Since the second information terminal also belongs to the same first space as the first information terminal, the information processing device 2 transmits the second feature point cloud data to the first computer and receives the first space data from the first computer. The information processing device 2 transmits the first space data to the second information terminal, and the second information terminal identifies its own detailed position by comparing the feature point cloud data it generated with a plurality of registered feature point cloud data included in the received first space data.

[0120] Fig. 11 is a sequence diagram showing a flow until multiple information terminals 1 belonging to different spaces acquire spatial data. The first information terminal and the third information terminal in Fig. 11 belong to different spaces, and are, for example, the information terminal 1a-1 and the information terminal 1b-1 shown in Fig. 1. The operation of the first information terminal is the same as the operation shown in Fig. 10.

[0121] The third information terminal transmits third approximate position data indicating the position of the third information terminal and third feature point cloud data generated based on the captured image data to the information processing device 2. Since the third information terminal belongs to a second space different from the first information terminal, the information processing device 2 transmits the third feature point cloud data to a second computer in charge of the second space corresponding to the third approximate position data, and receives the second space data from the third computer. The information processing device 2 transmits the second space data to the second information terminal, and the second information terminal identifies its own detailed position by comparing the feature point cloud data generated by itself with a plurality of registered feature point cloud data included in the received second space data.

[0122] (Processing for displaying AR images) Fig. 12 is a sequence diagram showing a process flow in which multiple information terminals 1 belonging to the same space display an AR image. The sequence diagram shown in Fig. 12 starts from the point in time when a user U1 using a first information terminal performs an operation to display an AR image. The first information terminal identifies the content of the operation performed by the user U1 (S11) and identifies the operation position coordinates indicating the position where the operation was performed (S12). The first information terminal transmits operation content data indicating the content of the operation and coordinate data indicating the operation position to the information processing device 2.

[0123] The information processing device 2 selects AR image data to be displayed on the first information terminal based on the received operation content data, and transmits the selected AR image data to the first information terminal. Note that, if the first information terminal stores the AR image data, the information processing device 2 does not need to transmit the AR image data to the first information terminal.

[0124] The information processing device 2 transmits display coordinate data corresponding to the coordinate data received from the first information terminal to a second information terminal belonging to the same space as the first information terminal. The information processing device 2 also transmits the AR image data transmitted to the first information terminal to the second information terminal. The first information terminal and the second information terminal display an AR image based on the received AR image data at a position and orientation corresponding to the display coordinate data (S13, S14). By the first information terminal and the second information terminal operating in this manner, multiple information terminals 1 belonging to the same space can display the same AR image at a position and orientation according to their respective positions and orientations.

[0125] (Spatial data registration) Fig. 13 is a sequence diagram showing the flow of processing in which the information terminal 1 registers feature point group data as spatial data. The sequence diagram shown in Fig. 13 starts from the point in time when the first information terminal compares the feature point group data it has created with the registered feature point group data.

[0126] When the first information terminal determines that there is no spatial data including registered feature point cloud data that matches the first feature point cloud data created by itself (YES in S21), it determines that the spatial data needs to be updated (S22) and transmits the first feature point cloud data together with an update request to the information processing device 2. Similarly, when the second information terminal determines that there is no spatial data including registered feature point cloud data that matches the second feature point cloud data created by itself (YES in S23), it transmits the second feature point cloud data together with an update request to update the spatial data.

[0127] When the information processing device 2 receives the first feature point cloud data and the second feature point cloud data corresponding to the same position and orientation, the information processing device 2 determines which feature point cloud data should be registered based on one of the above-mentioned predetermined conditions, and selects one of the feature point cloud data (S24). The information processing device 2 transmits the selected feature point cloud data to the storage device 4, and the storage device 4 registers the received feature point cloud data as spatial data (S25).

[0128] [Variations] In the above explanation, the information processing device 2 relays the feature point group data and the registered feature point group data between the information terminal 1 and the computer group 3 , but the computer group 3 may have the functions of the information processing device 2 .

[0129] In the above description, the information processing device 2 provides an AR image to the information terminal 1 in response to the information terminal 1 transmitting display coordinate data to the information processing device 2, but the information terminal 1 may store a plurality of AR images in advance. In this case, the information terminal 1 selects an AR image to be displayed from the plurality of stored AR images based on an instruction from the information processing device 2, and displays the selected AR image at a position and orientation based on the received display coordinate data.

[0130] In addition, in the above explanation, an example has been given of the case where the information processing device 2 has an in-memory database and various display coordinate data and feature point group data are exchanged via the in-memory database, but the information processing device 2 may not have an in-memory database, and multiple information terminals 1 may exchange these data via other storage media.

[0131] [Effects of Information Processing System S] The information processing device 2 according to this embodiment receives the display coordinate data transmitted by the first information terminal, registers the received display coordinate data in an in-memory database, and transmits the display coordinate data to a second information terminal belonging to the same space as the first information terminal in response to the display coordinate data being registered in the in-memory database. The information processing device 2 transmits and receives the display coordinate data to and from multiple information terminals 1 via the in-memory database, thereby making it possible to reduce the time required for the information terminal 1 to display an AR image.

[0132] Furthermore, the computer group 3 acquires feature point cloud data based on the captured image data generated by the information terminal 1, and the multiple computers 30, each responsible for a different space, compare the multiple registered feature point cloud data with the acquired feature point cloud data to identify the space to which the information terminal 1 belongs. The computer group 3 then provides the multiple registered feature point cloud data corresponding to the identified space to the information terminal 1. By the computer group 3 operating in this manner, the information terminal 1 can limit the multiple registered feature point cloud data to be compared with the feature point cloud data it has created to the multiple registered feature point cloud data corresponding to the space to which it belongs, and can therefore identify its own accurate position and orientation in a short time.

[0133] Furthermore, the information processing device 2 changes the number of computers 30 that execute the matching process with the feature point cloud data generated by the information terminal 1, based on preset conditions. By operating the information processing device 2 in this manner, even if the number of information terminals 1 increases or decreases, or the amount of feature point cloud data transmitted by the information terminal 1 increases or decreases, the time required to transmit appropriate registered feature point cloud data to the information terminal 1 can be maintained within a certain range.

[0134] Furthermore, when there is no registered feature point cloud data corresponding to the feature point cloud data generated by the information terminal 1, the information terminal 1 can register the feature point cloud data generated by itself in the storage device 4. By registering a large amount of feature point cloud data generated by the information terminals 1 in the storage device 4, the registered feature point cloud data stored in the storage device 4 can be kept up to date. Furthermore, when the information processing device 2 receives update requests for feature point cloud data corresponding to the same position and orientation from multiple information terminals 1, it registers more reliable feature point cloud data in the storage device 4. By the information processing device 2 operating in this manner, the reliability of the registered feature point cloud data stored in the storage device 4 is improved.

[0135] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by distributing or integrating functionally or physically in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment resulting from the combination combines the effect of the original embodiment. [Explanation of symbols]

[0136] 1 Information terminal 2. Information processing equipment 3. Computer Group 4 Storage device 11 Communications Department 12 Control section 13 Position detection unit 14 Imaging unit 15 Display section 16 Memory section 17 Control Unit 21 Communications Department 22 Memory section 23 Control Unit 30 Computers 31 Communications Department 32 Storage section 33 Control Unit 171 Transmission processing unit 172 Receiving processing unit 173 Operation reception section 174 Rough position identification part 175 Image Processing Unit 176 Detailed location identification part 177 Display Processing Unit 178 Spatial Data Acquisition Section 221 First spatial database 222 Second spatial database 231 Coordinate receiver 232 Coordinate transmission unit 233 Terminal data receiver 234 Feature point group transmission unit 235 Information Receiving Department 236 Spatial Data Transmission Unit 237 Spatial Data Management Department 238 Unit Management Department 331 Feature Point Cloud Acquisition Unit 332 Registration Data Acquisition Department 333 Data Analysis Department

Claims

1. An information terminal, an information processing device capable of communicating with the information terminal, and a computer group including a plurality of computers, The information terminal includes: an imaging unit that generates captured image data by capturing an image of at least a part of a space in which the information terminal resides; a transmission processing unit that transmits feature point group data composed of a plurality of feature points of a subject included in the captured image data to the information processing device; having The information processing device includes: a terminal data receiving unit for receiving the feature point group data from the information terminal; a feature point group transmission unit that transmits the feature point group data received from the information terminal to the group of computers; having The plurality of computers include a feature point group acquisition unit that acquires the feature point group data from the information processing device; a data analysis unit that transmits predetermined information to the information processing device when a difference between the feature point group data acquired by the feature point group acquisition unit and any one of a plurality of registered feature point group data corresponding to a space assigned to each computer is equal to or smaller than a threshold value; having The information processing device includes: An information receiving unit that receives the predetermined information; a spatial data transmission unit configured to transmit to the information terminal the plurality of registered feature point group data corresponding to the computer that has transmitted the predetermined information; Information processing system.

2. the information terminal further includes a detailed position specifying unit that specifies a position and an orientation corresponding to registered feature point cloud data that is most similar to the feature point cloud data among the plurality of registered feature point cloud data received from the information processing device. The information processing system according to claim 1 .

3. the information processing device has a plurality of in-memory databases that store the feature point group data corresponding to each of the plurality of spaces; the feature point group transmission unit transmits the feature point group data to the computer associated with a space corresponding to an in-memory database in which the feature point group data is written, among the plurality of in-memory databases; 3. The information processing system according to claim 1 or 2.

4. the information terminal further includes an approximate location identification unit that generates approximate location data indicating an approximate location of the information terminal based on a received radio wave; the transmission processing unit transmits a predetermined message together with the feature point cloud data to an in-memory database among the plurality of in-memory databases, the in-memory database corresponding to the space to which the approximate location belongs, thereby writing the feature point cloud data to the in-memory database corresponding to the space to which the approximate location belongs; The information processing system according to claim 3 .

5. the feature point group transmission unit transmits the feature point group data to the computer associated with a space corresponding to data to which the approximate position indicated by the approximate position data transmitted by the information terminal together with the feature point group data belongs; 5. The information processing system according to claim 4.

6. the feature point group acquisition unit acquires the plurality of registered feature point group data by reading the plurality of registered feature point group data corresponding to the space assigned to its own computer from a storage device storing the plurality of registered feature point group data corresponding to the spaces; The information processing system according to claim 1 .

7. An information terminal generating captured image data by capturing an image of at least a part of a space to which the information terminal belongs; a step of transmitting, by the information terminal, feature point group data configured of a group of multiple feature points of a subject included in the captured image data; A step of acquiring the feature point cloud data by a plurality of computers; providing, to the information terminal, the plurality of registered feature point cloud data corresponding to a space allocated to the computer, when a difference between the acquired feature point cloud data and any one of the plurality of registered feature point cloud data corresponding to the computer is equal to or smaller than a threshold value; An information processing method comprising the steps of:

Images